pandasdmx: sdmx.Request leading to SSL Error - bad handshake

Question

I am using the pandasdmx library and would like to access the IMF datasource. The following code produces a SSLError.

from pathlib import Path

import pandasdmx as sdmx

http_proxy = "my/proxy_server/address"
proxies = {
    "http": http_proxy,
    "https": http_proxy,
}
ssl_cert = Path("path/to/my/ssl_certificate")  # .pem file
data_source = "IMF"

src = sdmx.Request(
    data_source,
    proxies=proxies,
    verify=ssl_cert,
    backend="sqlite",
    fast_save=True,
    expire_after=600,
)

flow_msg = src.dataflow()


Traceback (most recent call last):
  File "C:\Users\D292498\AppData\Local\conda\conda\envs\py38\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-f5bd59ed1595>", line 1, in <module>
    runfile('C:/Users/D292498/.PyCharmCE2019.2/config/scratches/scratch_1.py', wdir='C:/Users/D292498/.PyCharmCE2019.2/config/scratches')
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.2\helpers\pydev\_pydev_bundle\pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2019.2\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/D292498/.PyCharmCE2019.2/config/scratches/scratch_1.py", line 24, in <module>
    flow_msg = src.dataflow()
  File "C:\Users\D292498\AppData\Local\conda\conda\envs\py38\lib\site-packages\pandasdmx\api.py", line 392, in get
    raise e from None
  File "C:\Users\D292498\AppData\Local\conda\conda\envs\py38\lib\site-packages\pandasdmx\api.py", line 389, in get
    response = self.session.send(req)
  File "C:\Users\D292498\AppData\Local\conda\conda\envs\py38\lib\site-packages\requests_cache\core.py", line 109, in send
    return send_request_and_cache_response()
  File "C:\Users\D292498\AppData\Local\conda\conda\envs\py38\lib\site-packages\requests_cache\core.py", line 97, in send_request_and_cache_response
    response = super(CachedSession, self).send(request, **kwargs)
  File "C:\Users\D292498\AppData\Local\conda\conda\envs\py38\lib\site-packages\requests\sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\D292498\AppData\Local\conda\conda\envs\py38\lib\site-packages\requests\adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='sdmxcentral.imf.org', port=443): Max retries exceeded with url: /ws/public/sdmxapi/rest/dataflow/IMF/latest (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

However, I can easily load the website/xml document https://sdmxcentral.imf.org/ws/public/sdmxapi/rest/dataflow/IMF/latest in my browser. Also, using the regular request library seems to be working properly. The following code produces status code 200.

from pathlib import Path

import requests

proxy_url = "my/proxy_server/address"
proxies = {"http": proxy_url, "https": proxy_url}
ssl_cert = Path("path/to/my/ssl_certificate")  # .pem file
web_address = "https://sdmxcentral.imf.org/ws/public/sdmxapi/rest/dataflow/IMF/latest"

r = requests.get(web_address, proxies=proxies, verify=ssl_cert)
print(r.status_code)  # 200 OK

Answer 1

Apparently, sdmx.Request() is not correctly forwarding the verify argument.

As a result, the call to the underlying request library will use its standard CA certificate. We therefore need to add our cert information into that particular file.

The following lines of code should do the trick:

from pathlib import Path

import certifi

ssl_cert = Path("path/to/my/ssl_certificate")  # .pem file

cafile = certifi.where()

with open(ssl_cert, 'rb') as infile:
    customca = infile.read()

with open(cafile, 'ab') as outfile:
    outfile.write(customca)

Please find more details at https://incognitjoe.github.io/adding-certs-to-requests.html.

Answer 2

Apparently, sdmx.Request() is not correctly forwarding the verify argument.

In sdmx1—a more actively maintained fork of pandaSDMX—this bug was noted in an issue on 2020-11-20 and fixed in v1.6.0 on 2020-12-16. verify is now honoured, so the manual addition of a certificate in the other answer is no longer necessary.