When using a new library such as requests, you can improve you developer effectiveness by being familiar with the most common questions that come up when using the python requests library. Using the Stack Overflow Data Explorer tool, we’ve determined the top 10 most popular requests questions & answers by daily views on Stack Overflow to to be familiar with. Check out the top 10 requests questions & answers below:
Looking to get a head start on your next software interview? Pickup a copy of the best book to prepare: Cracking The Coding Interview!
1. Importerror: no module named requests?
Requests is not a built in module (does not come with the default python installation), so you will have to install it:
OSX/Linux
Use $ pip install requests
(or pip3 install requests
for python3) if you have pip
installed. If pip is installed but not in your path you can use python -m pip install requests
(or python3 -m pip install requests
for python3)
Alternatively you can also use sudo easy_install -U requests
if you have easy_install
installed.
Alternatively you can use your systems package manager:
For centos: yum install python-requests
For Ubuntu: apt-get install python-requests
Windows
Use pip install requests
(or pip3 install requests
for python3) if you have pip
installed and Pip.exe added to the Path Environment Variable. If pip is installed but not in your path you can use python -m pip install requests
(or python3 -m pip install requests
for python3)
Alternatively from a cmd prompt, use > Path\easy_install.exe requests
, where Path
is your Python*\Scripts
folder, if it was installed. (For example: C:\Python32\Scripts
)
If you manually want to add a library to a windows machine, you can download the compressed library, uncompress it, and then place it into the Lib\site-packages
folder of your python path. (For example: C:\Python27\Lib\site-packages
)
From Source (Universal)
For any missing library, the source is usually available at https://pypi.python.org/pypi/. You can download requests here: https://pypi.python.org/pypi/requests
On mac osx and windows, after downloading the source zip, uncompress it and from the termiminal/cmd run python setup.py install
from the uncompressed dir.
(source)
2. How to post json data with python requests?
Starting with Requests version 2.4.2, you can use the json=
parameter (which takes a dictionary) instead of data=
(which takes a string) in the call:
>>> import requests
>>> r = requests.post('http://httpbin.org/post', json={"key": "value"})
>>> r.status_code
200
>>> r.json()
{'args': {},
'data': '{"key": "value"}',
'files': {},
'form': {},
'headers': {'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate',
'Connection': 'close',
'Content-Length': '16',
'Content-Type': 'application/json',
'Host': 'httpbin.org',
'User-Agent': 'python-requests/2.4.3 CPython/3.4.0',
'X-Request-Id': 'xx-xx-xx'},
'json': {'key': 'value'},
'origin': 'x.x.x.x',
'url': 'http://httpbin.org/post'}
3. Why requests raise this exception “check_hostname requires server_hostname”?
as a work around:
pip install urllib3==1.25.11
4. Python requests throwing sslerror?
The problem you are having is caused by an untrusted SSL certificate.
Like @dirk mentioned in a previous comment, the quickest fix is setting verify=False
:
requests.get('https://example.com', verify=False)
Please note that this will cause the certificate not to be verified. This will expose your application to security risks, such as man-in-the-middle attacks.
Of course, apply judgment. As mentioned in the comments, this may be acceptable for quick/throwaway applications/scripts, but really should not go to production software.
If just skipping the certificate check is not acceptable in your particular context, consider the following options, your best option is to set the verify
parameter to a string that is the path of the .pem
file of the certificate (which you should obtain by some sort of secure means).
So, as of version 2.0, the verify
parameter accepts the following values, with their respective semantics:
True
: causes the certificate to validated against the library’s own trusted certificate authorities (Note: you can see which Root Certificates Requests uses via the Certifi library, a trust database of RCs extracted from Requests: Certifi – Trust Database for Humans).False
: bypasses certificate validation completely.- Path to a CA_BUNDLE file for Requests to use to validate the certificates.
Source: Requests – SSL Cert Verification
Also take a look at the cert
parameter on the same link.
5. Max retries exceeded with url in requests?
What happened here is that itunes server refuses your connection (you’re sending too many requests from same ip address in short period of time)
Max retries exceeded with url: /in/app/adobe-reader/id469337564?mt=8
error trace is misleading it should be something like “No connection could be made because the target machine actively refused it”.
There is an issue at about python.requests lib at Github, check it out here
To overcome this issue (not so much an issue as it is misleading debug trace) you should catch connection related exceptions like so:
try:
page1 = requests.get(ap)
except requests.exceptions.ConnectionError:
r.status_code = "Connection refused"
Another way to overcome this problem is if you use enough time gap to send requests to server this can be achieved by sleep(timeinsec)
function in python (don’t forget to import sleep)
from time import sleep
All in all requests is awesome python lib, hope that solves your problem.
6. What’s the best way to parse a json response from the requests library?
Since you’re using requests
, you should use the response’s json
method.
import requests
response = requests.get(...)
data = response.json()
7. Correct way to try/except using python requests module?
Have a look at the Requests exception docs. In short:
In the event of a network problem (e.g. DNS failure, refused connection, etc), Requests will raise a
ConnectionError
exception.In the event of the rare invalid HTTP response, Requests will raise an
HTTPError
exception.If a request times out, a
Timeout
exception is raised.If a request exceeds the configured number of maximum redirections, a
TooManyRedirects
exception is raised.All exceptions that Requests explicitly raises inherit from
requests.exceptions.RequestException
.
To answer your question, what you show will not cover all of your bases. You’ll only catch connection-related errors, not ones that time out.
What to do when you catch the exception is really up to the design of your script/program. Is it acceptable to exit? Can you go on and try again? If the error is catastrophic and you can’t go on, then yes, you may abort your program by raising SystemExit (a nice way to both print an error and call sys.exit
).
You can either catch the base-class exception, which will handle all cases:
try:
r = requests.get(url, params={'s': thing})
except requests.exceptions.RequestException as e: # This is the correct syntax
raise SystemExit(e)
Or you can catch them separately and do different things.
try:
r = requests.get(url, params={'s': thing})
except requests.exceptions.Timeout:
# Maybe set up for a retry, or continue in a retry loop
except requests.exceptions.TooManyRedirects:
# Tell the user their URL was bad and try a different one
except requests.exceptions.RequestException as e:
# catastrophic error. bail.
raise SystemExit(e)
As Christian pointed out:
If you want http errors (e.g. 401 Unauthorized) to raise exceptions, you can call
Response.raise_for_status
. That will raise anHTTPError
, if the response was an http error.
An example:
try:
r = requests.get('http://www.google.com/nothere')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
raise SystemExit(err)
Will print:
404 Client Error: Not Found for url: http://www.google.com/nothere
8. How do i disable the security certificate check in python requests?
From the documentation:
requests
can also ignore verifying the SSL certificate if you set
verify
to False.>>> requests.get('https://kennethreitz.com', verify=False) <Response [200]>
If you’re using a third-party module and want to disable the checks, here’s a context manager that monkey patches requests
and changes it so that verify=False
is the default and suppresses the warning.
import warnings
import contextlib
import requests
from urllib3.exceptions import InsecureRequestWarning
old_merge_environment_settings = requests.Session.merge_environment_settings
@contextlib.contextmanager
def no_ssl_verification():
opened_adapters = set()
def merge_environment_settings(self, url, proxies, stream, verify, cert):
# Verification happens only once per connection so we need to close
# all the opened adapters once we're done. Otherwise, the effects of
# verify=False persist beyond the end of this context manager.
opened_adapters.add(self.get_adapter(url))
settings = old_merge_environment_settings(self, url, proxies, stream, verify, cert)
settings['verify'] = False
return settings
requests.Session.merge_environment_settings = merge_environment_settings
try:
with warnings.catch_warnings():
warnings.simplefilter('ignore', InsecureRequestWarning)
yield
finally:
requests.Session.merge_environment_settings = old_merge_environment_settings
for adapter in opened_adapters:
try:
adapter.close()
except:
pass
Here’s how you use it:
with no_ssl_verification():
requests.get('https://wrong.host.badssl.com/')
print('It works')
requests.get('https://wrong.host.badssl.com/', verify=True)
print('Even if you try to force it to')
requests.get('https://wrong.host.badssl.com/', verify=False)
print('It resets back')
session = requests.Session()
session.verify = True
with no_ssl_verification():
session.get('https://wrong.host.badssl.com/', verify=True)
print('Works even here')
try:
requests.get('https://wrong.host.badssl.com/')
except requests.exceptions.SSLError:
print('It breaks')
try:
session.get('https://wrong.host.badssl.com/')
except requests.exceptions.SSLError:
print('It breaks here again')
Note that this code closes all open adapters that handled a patched request once you leave the context manager. This is because requests maintains a per-session connection pool and certificate validation happens only once per connection so unexpected things like this will happen:
>>> import requests
>>> session = requests.Session()
>>> session.get('https://wrong.host.badssl.com/', verify=False)
/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
<Response [200]>
>>> session.get('https://wrong.host.badssl.com/', verify=True)
/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
InsecureRequestWarning)
<Response [200]>
9. Requests (caused by sslerror(“can’t connect to https url because the ssl module is not available.”) error in pycharm requesting website?
Don’t know if this has been solved yet but I was getting similar problems with Anaconda python 3.7.3 and Idle on Windows 10. Fixed it by adding:
<path>\Anaconda3
<path>\Anaconda3\scripts
<path>\Anaconda3\Library\bin
to the PATH variable.
10. How to download image using requests?
You can either use the response.raw
file object, or iterate over the response.
To use the response.raw
file-like object will not, by default, decode compressed responses (with GZIP or deflate). You can force it to decompress for you anyway by setting the decode_content
attribute to True
(requests
sets it to False
to control decoding itself). You can then use shutil.copyfileobj()
to have Python stream the data to a file object:
import requests
import shutil
r = requests.get(settings.STATICMAP_URL.format(**data), stream=True)
if r.status_code == 200:
with open(path, 'wb') as f:
r.raw.decode_content = True
shutil.copyfileobj(r.raw, f)
To iterate over the response use a loop; iterating like this ensures that data is decompressed by this stage:
r = requests.get(settings.STATICMAP_URL.format(**data), stream=True)
if r.status_code == 200:
with open(path, 'wb') as f:
for chunk in r:
f.write(chunk)
This’ll read the data in 128 byte chunks; if you feel another chunk size works better, use the Response.iter_content()
method with a custom chunk size:
r = requests.get(settings.STATICMAP_URL.format(**data), stream=True)
if r.status_code == 200:
with open(path, 'wb') as f:
for chunk in r.iter_content(1024):
f.write(chunk)
Note that you need to open the destination file in binary mode to ensure python doesn’t try and translate newlines for you. We also set stream=True
so that requests
doesn’t download the whole image into memory first.
Elevate your software skills
Ergonomic Mouse |
Custom Keyboard |
SW Architecture |
Clean Code |