TL;DR: I use the python requests module in an application. I wish to compile this application into a distributable binary using pyinstaller, and also to make the process run in a daemon using python-daemon. Either using pyinstaller or python-daemon on their own has no issues, however, trying to both use pyinstaller AND python-daemon, the requests module suddenly decides that its cacert.pem is in /tmp/[bunch of letters and numbers]/certifi/cacert.pem and not anywhere else. Trying to bundle or load a copy of any cacert.pem results in another error, where requests is unable to verify the local issuer.
Experiments
Given a simple python requests script (taken from w3 schools):
import requests
x = requests.get('https://w3schools.com/python/demopage.htm')
print(x.text)
It outputs exactly as expected:
<!DOCTYPE html>
<html>
<body>
<h1>This is a Test Page</h1>
</body>
</html>
I then try compiling using pyinstaller (to a linux executable) using the following .spec file:
# -*- mode: python ; coding: utf-8 -*-
block_cipher = None
a = Analysis(['test.py'],
pathex=['/workspaces/tra-data-analysis/src'],
binaries=[],
datas=[],
hiddenimports=[
"dnspython",
"requests",
],
hookspath=[],
runtime_hooks=[],
excludes=[],
win_no_prefer_redirects=False,
win_private_assemblies=False,
cipher=block_cipher,
noarchive=False)
pyz = PYZ(a.pure, a.zipped_data,
cipher=block_cipher)
exe = EXE(pyz,
a.scripts,
a.binaries,
a.zipfiles,
a.datas,
[('W ignore', None, 'OPTION')],
name='test',
debug=False,
bootloader_ignore_signals=False,
strip=False,
upx=True,
upx_exclude=[],
runtime_tmpdir=None,
console=True )
And the compiled binary still behaves as expected.
Wrapping the request snippet as a daemon process using python-daemon:
import requests
import daemon
from daemon import pidfile
import os
e = open("errorlog", "w")
with daemon.DaemonContext(
working_directory = os.getcwd(),
pidfile = pidfile.TimeoutPIDLockFile("pidfile"),
stderr=e
):
x = requests.get('https://w3schools.com/python/demopage.htm')
f = open("output", "w")
f.write(x.text)
f.close()
Still behaves as expected.
However, applying both the daemonization and the pyinstaller compilation, yields the following error:
Traceback (most recent call last):
File "test.py", line 13, in <module>
File "requests/api.py", line 75, in get
File "requests/api.py", line 61, in request
File "requests/sessions.py", line 529, in request
File "requests/sessions.py", line 645, in send
File "requests/adapters.py", line 417, in send
File "requests/adapters.py", line 228, in cert_verify
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /tmp/_MEIKPqY64/certifi/cacert.pem
[23066] Failed to execute script 'test' due to unhandled exception!
Neither the compilation nor the daemonization by themselves cause this issue, yet when both are used together, they cause some conflict where the requests looks for cacert.pem in a temporary location that does not exist.
Solutions already attempted
- making a copy of certifi module's cacert.pem in the local directory, then using the parameter
verify="."., Doing this yields the error:
Traceback (most recent call last):
File "urllib3/connectionpool.py", line 703, in urlopen
File "urllib3/connectionpool.py", line 386, in _make_request
File "urllib3/connectionpool.py", line 1040, in _validate_conn
File "urllib3/connection.py", line 416, in connect
File "urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
File "urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
File "ssl.py", line 512, in wrap_socket
File "ssl.py", line 1070, in _create
File "ssl.py", line 1341, in do_handshake
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "requests/adapters.py", line 440, in send
File "urllib3/connectionpool.py", line 785, in urlopen
File "urllib3/util/retry.py", line 592, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='w3schools.com', port=443): Max retries exceeded with url: /python/demopage.htm (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "test.py", line 13, in <module>
File "requests/api.py", line 75, in get
File "requests/api.py", line 61, in request
File "requests/sessions.py", line 529, in request
File "requests/sessions.py", line 645, in send
File "requests/adapters.py", line 517, in send
requests.exceptions.SSLError: HTTPSConnectionPool(host='w3schools.com', port=443): Max retries exceeded with url: /python/demopage.htm (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
[29400] Failed to execute script 'test' due to unhandled exception!
Bundling certifi's cacert in pyinstaller as data. Doing so yields the same results as 1
Using parameter verify=False. This one actually works, but is rather unsafe for the data I am actually pulling. I would like to avoid using this solution if possible
Compiling pyinstaller in none directory mode instead of one file mode by adding:
coll = COLLECT(exe,
a.binaries,
a.zipfiles,
a.datas,
strip=False,
upx=True,
upx_exclude=[],
name='lib'
)
to the .spec file. Results are the same as building in one file mode.
Summary of results
Using a simple requests.get() process with:
- No additional modifications: No problems
- Pyinstaller ONLY: No problems
- python-daemon ONLY: No problems
- both Pyinstaller and python-daemon: looks for cacert.pem in non existent folder
- both Pyinstaller and python-daemon with local copy of cacert.pem: SSL error
- both Pyinstaller and python-daemon with bundled copy of cacert.pem: SSL error
- both PyInstaller and python-daemon with one-dir mode: looks for cacert.pem in non existent folder
Conclusion
Either I am make a very simple oversight, or there is a deeper underlying problem with how pyinstaller interacts with python-daemon and requests. Any insights would be helpful, I've been chasing this problem for a few weeks now.
Pyinstaller unpacks and runs from /tmp (though this can be configured to be a different directory).
I found the below code which I use to do something similar to source and load a config file. You have to determine whether you are running as a packaged binary or a script and then make the file location relative to where it is running from. Alternatively, you can use an absolute path. You could also make a runtime argument (arg.parse) and specify the full path at runtime.
The below code sets the file path as being the same directory from where the program is being run: