We have a virtualenv in our deploy environment where the .pyc
files are sym links - particularly those in <venv_root>/lib/python2.7/
. Under what conditions does this happen?
Our environment:
- CentOS 7
- Python 2.7.5
- virtualenv 16.0.0
- some (but not all) python processes running on other servers with the virtualenv available via NFS (with write permissions)
- virtualenv created on a different filesystem to the root filesystem
I'm used to seeing the .py
files in <venv_root>/lib/python2.7/
sometimes being symlinks. I know I can use the --always-copy
flag to stop the .py
files being symlinks. But I'm used to the .pyc
files always being plain files.
Experiments I've tried (on the server in question):
- create fresh virtualenv on root filesystem
- pyc files created as files
- run scripts using the virtualenv on the NFS host and the NFS client
- no change to pyc files
- touch
lib/python2.7/os.py
and re-run scripts- no change to pyc files
- remove
lib/python2.7/os.pyc
and re-run scripts- runs on the NFS host, the pyc file is created as a normal file
- runs on the NFS client, the pyc file is created as a normal file
To repeat the key question - under what conditions will I end up with .pyc
files being sym links?
Here is a full listing of the virtualenv directory in question:
[root@appname-master]# ls -l /export/appname/common/venv/5_4_1/lib/python2.7/
total 68
lrwxrwxrwx 1 appname appname 31 Nov 8 04:17 _abcoll.py -> /usr/lib64/python2.7/_abcoll.py
lrwxrwxrwx 1 appname appname 62 Nov 8 04:17 _abcoll.pyc -> /usr/local/company/appname/venv/lib/python2.7/_abcoll.pyc
lrwxrwxrwx 1 appname appname 27 Nov 8 04:17 abc.py -> /usr/lib64/python2.7/abc.py
lrwxrwxrwx 1 appname appname 58 Nov 8 04:17 abc.pyc -> /usr/local/company/appname/venv/lib/python2.7/abc.pyc
lrwxrwxrwx 1 appname appname 30 Nov 8 04:17 codecs.py -> /usr/lib64/python2.7/codecs.py
lrwxrwxrwx 1 appname appname 61 Nov 8 04:17 codecs.pyc -> /usr/local/company/appname/venv/lib/python2.7/codecs.pyc
lrwxrwxrwx 1 appname appname 27 Nov 8 04:17 config -> /usr/lib64/python2.7/config
lrwxrwxrwx 1 appname appname 32 Nov 8 04:17 copy_reg.py -> /usr/lib64/python2.7/copy_reg.py
lrwxrwxrwx 1 appname appname 63 Nov 8 04:17 copy_reg.pyc -> /usr/local/company/appname/venv/lib/python2.7/copy_reg.pyc
lrwxrwxrwx 1 appname appname 60 Nov 8 04:17 distutils -> /usr/local/company/appname/venv/lib/python2.7/distutils
lrwxrwxrwx 1 appname appname 30 Nov 8 04:17 encodings -> /usr/lib64/python2.7/encodings
lrwxrwxrwx 1 appname appname 31 Nov 8 04:17 fnmatch.py -> /usr/lib64/python2.7/fnmatch.py
lrwxrwxrwx 1 appname appname 62 Nov 8 04:17 fnmatch.pyc -> /usr/local/company/appname/venv/lib/python2.7/fnmatch.pyc
lrwxrwxrwx 1 appname appname 35 Nov 8 04:17 genericpath.py -> /usr/lib64/python2.7/genericpath.py
lrwxrwxrwx 1 appname appname 66 Nov 8 04:17 genericpath.pyc -> /usr/local/company/appname/venv/lib/python2.7/genericpath.pyc
lrwxrwxrwx 1 appname appname 32 Nov 8 04:17 lib-dynload -> /usr/lib64/python2.7/lib-dynload
lrwxrwxrwx 1 appname appname 33 Nov 8 04:17 linecache.py -> /usr/lib64/python2.7/linecache.py
lrwxrwxrwx 1 appname appname 64 Nov 8 04:17 linecache.pyc -> /usr/local/company/appname/venv/lib/python2.7/linecache.pyc
lrwxrwxrwx 1 appname appname 30 Nov 8 04:17 locale.py -> /usr/lib64/python2.7/locale.py
lrwxrwxrwx 1 appname appname 61 Nov 8 04:17 locale.pyc -> /usr/local/company/appname/venv/lib/python2.7/locale.pyc
-rw-rw-r--+ 1 appname appname 0 Nov 8 04:17 no-global-site-packages.txt
lrwxrwxrwx 1 appname appname 30 Nov 8 04:17 ntpath.py -> /usr/lib64/python2.7/ntpath.py
-rw-rw-r--+ 1 appname appname 4 Nov 8 04:17 orig-prefix.txt
lrwxrwxrwx 1 appname appname 26 Nov 8 04:17 os.py -> /usr/lib64/python2.7/os.py
lrwxrwxrwx 1 appname appname 57 Nov 8 04:17 os.pyc -> /usr/local/company/appname/venv/lib/python2.7/os.pyc
lrwxrwxrwx 1 appname appname 33 Nov 8 04:17 posixpath.py -> /usr/lib64/python2.7/posixpath.py
lrwxrwxrwx 1 appname appname 64 Nov 8 04:17 posixpath.pyc -> /usr/local/company/appname/venv/lib/python2.7/posixpath.pyc
lrwxrwxrwx 1 appname appname 26 Nov 8 04:17 re.py -> /usr/lib64/python2.7/re.py
lrwxrwxrwx 1 appname appname 57 Nov 8 04:17 re.pyc -> /usr/local/company/appname/venv/lib/python2.7/re.pyc
drwxrwxr-x+ 40 appname appname 4096 Nov 8 04:17 site-packages
-rw-rw-r--+ 1 appname appname 29021 Nov 8 04:17 site.py
-rw-rw-r--+ 1 appname appname 26716 Nov 8 04:17 site.pyc
lrwxrwxrwx 1 appname appname 35 Nov 8 04:17 sre_compile.py -> /usr/lib64/python2.7/sre_compile.py
lrwxrwxrwx 1 appname appname 66 Nov 8 04:17 sre_compile.pyc -> /usr/local/company/appname/venv/lib/python2.7/sre_compile.pyc
lrwxrwxrwx 1 appname appname 37 Nov 8 04:17 sre_constants.py -> /usr/lib64/python2.7/sre_constants.py
lrwxrwxrwx 1 appname appname 68 Nov 8 04:17 sre_constants.pyc -> /usr/local/company/appname/venv/lib/python2.7/sre_constants.pyc
lrwxrwxrwx 1 appname appname 33 Nov 8 04:17 sre_parse.py -> /usr/lib64/python2.7/sre_parse.py
lrwxrwxrwx 1 appname appname 64 Nov 8 04:17 sre_parse.pyc -> /usr/local/company/appname/venv/lib/python2.7/sre_parse.pyc
lrwxrwxrwx 1 appname appname 27 Nov 8 04:17 sre.py -> /usr/lib64/python2.7/sre.py
lrwxrwxrwx 1 appname appname 28 Nov 8 04:17 stat.py -> /usr/lib64/python2.7/stat.py
lrwxrwxrwx 1 appname appname 59 Nov 8 04:17 stat.pyc -> /usr/local/company/appname/venv/lib/python2.7/stat.pyc
lrwxrwxrwx 1 appname appname 29 Nov 8 04:17 types.py -> /usr/lib64/python2.7/types.py
lrwxrwxrwx 1 appname appname 60 Nov 8 04:17 types.pyc -> /usr/local/company/appname/venv/lib/python2.7/types.pyc
lrwxrwxrwx 1 appname appname 32 Nov 8 04:17 UserDict.py -> /usr/lib64/python2.7/UserDict.py
lrwxrwxrwx 1 appname appname 63 Nov 8 04:17 UserDict.pyc -> /usr/local/company/appname/venv/lib/python2.7/UserDict.pyc
lrwxrwxrwx 1 appname appname 32 Nov 8 04:17 warnings.py -> /usr/lib64/python2.7/warnings.py
lrwxrwxrwx 1 appname appname 63 Nov 8 04:17 warnings.pyc -> /usr/local/company/appname/venv/lib/python2.7/warnings.pyc
lrwxrwxrwx 1 appname appname 35 Nov 8 04:17 _weakrefset.py -> /usr/lib64/python2.7/_weakrefset.py
lrwxrwxrwx 1 appname appname 66 Nov 8 04:17 _weakrefset.pyc -> /usr/local/company/appname/venv/lib/python2.7/_weakrefset.pyc
(I'm also interested as to why the .pyc
sym links are pointing at another virtualenv when the .py
links point at the system files. But I thought understanding how .pyc
files can ever become sym links would be a good start.)
tox appears to be the culprit (or possibly one of its dependencies). The below is run on Ubuntu 18.04.