SSLError: max retries exceeded with url error? How to fix this?

Question

I am attempting to scrape text off of websites, and I am using the requests module to do so.

With the given code (Facebook as an example here)

requests.get('http://facebook.com')

I receive back the following error:

SSLError: HTTPSConnectionPool(host='facebook.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

I have tried the following with no luck:

pip install certifi
pip install certifi_win32

Any help would be greatly appreciated! Thank you!

Answer 1

I had the same problem, but with a tiny difference.

I was trying to reach the website by IP

requests.get('http://10.20.30.40', verify=False)

And all the solutions provided was failing, until I used the domain name instead of the IP.

Answer 2

You can try this way

import requests
from urllib3.exceptions import InsecureRequestWarning
from urllib3 import disable_warnings

disable_warnings(InsecureRequestWarning)

page = requests.get('http://facebook.com', verify=False)

print(page.content)

Answer 3

I had the same problem while I scraping a website. I tried setting up False to verify and I used CA certificate, both didn't work. Reading through the docs I find Session Objects

I was making several requests to the same host for that reason I was getting "Max retries exceeded with url"

In your case, you can try this:

s = requests.Session()
response = s.get('http://facebook.com')

Answer 4

Use this code and replace url variable with your website link

import urllib3

urllib3.disable_warnings()

s = requests.Session()

response = s.get(url, verify=False)

response.text

Answer 5

The problem probably stems from an overly aggressive security measure that you might be able to fix in two steps:

1. Download the raw CA Bundle from https://certifiio.readthedocs.io/en/latest/
2. In requests.get use verify=[path to raw CA Bundle]