obtain the urllib.requests certificate bundle used with urlopen

Question

I am on windows, and I am seeing a difference between urllib.requests.urlopen and requests package when making calls to the same site.

When I perform the following:

import urllib.request
f = urllib.request.urlopen('https://<domain certificated server> ')

I can reach my server no problems.

When I do:

import requests
f = requests.get('https://<domain certificated server> ').text

I am getting an SSL Certificate error. I know this is caused by certifi. So my question is this, how can I leverage whatever Python code is doing and use that over certifi in requests?

Answer 1

The simple way is to tell requests to use the system bundle, if you know where it is. On Linux it's usually in /etc/ssl/certs/ca-certificates.crt.

requests.get('<server.internal>', verify='/etc/ssl/certs/ca-certificates.crt')

You can also define your own adapter if you are on Windows, don't know where the system bundle is located, or just want to support all cases. The default adapter tells urllib3 to load certificates from DEFAULT_CA_BUNDLE_PATH which comes from calling certifi.where(). If you don't tell urllib3 where to get certificates, it will use the system default.

import requests
import requests.adapters

class SysCertsHTTPAdapter(requests.adapters.HTTPAdapter):
  def cert_verify(self, conn, url, verify, cert):
    # require verification, but use system defaults
    # default requests implementation sets conn.ca_certs and/or conn.ca_cert_dir
    conn.cert_reqs = 'CERT_REQUIRED'


session = requests.Session()
session.mount('https://', SysCertsHTTPAdapter())
session.get("https://google.com")

Answer 2

Do you have any option to obtain a .pem file with the server certificates?.

urlopen is using ssl.enum_certificates on windows to load the certificates. So the certs are obtained from a windows service.

Once you get the certificate file, use it on your call requests.get('<server.internal>', verify='/path/to/certfile')

Answer 3

I found here that you can run python3 -c "import ssl; print(ssl.get_default_verify_paths())" to get the default paths used for loading certs.

On Ubuntu, I got this:

DefaultVerifyPaths(cafile=None, capath='/usr/lib/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/usr/lib/ssl/certs')

I noticed that /usr/lib/ssl/cert.pem didn't exist on my system, and so created a symbolic link to it.

sudo ln -s /usr/lib/ssl/certs/ca-certificates.crt  /usr/lib/ssl/cert.pem