Can I have my pip user-installed package be preferred over system?

Question

I would like to figure out a "fool-proof" installation instruction to put in the README of a Python project, call it footools, such that other people in our group can install the newest SVN version of it on their laptops and their server accounts.

The problem is getting the user-installed libs to be used by Python when they call the scripts installed by pip. E.g., we're using a server that has an old version of footools in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/.

If I do python2.7 setup.py install --user and run the main entry script, it uses the files in /Users/unhammer/Library/Python/2.7/lib/python/site-packages/. This is what I want, but setup.py alone doesn't install dependencies.

If I (revert the installation and) instead do pip-2.7 install --user . and run the main entry script, it uses the old files in /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/ – that's not what I want.

If I (revert the installation and) instead do pip-2.7 install --user -e . and run the main entry script, it uses the files in . – that's not what I want, the user should be able to remove the source dir (and be able to svn up without that affecting their install).

---

I could use (and recommend other people to use) python2.7 setup.py install --user – but then they have to first do

pip-2.7 install -U --user -r requirements.txt -e .
pip-2.7 uninstall -y footools

in order to get the dependencies installed (since pip has no install --only-deps option). That's rather verbose though.

What is setup.py doing that pip is not doing here?

(Edited to make it clear I'm looking for simpler+safer installation instructions.)

Answer 1

Install virtualenvwrapper. I allows setting up separate python environments to alleviate any conflicts you might be having. Here is a tutorial for installing and using virtualenv.

Related:

• https://virtualenvwrapper.readthedocs.org/en/latest/

Answer 2

Console scripts generated by pip in the process of installation should use user installed versions of libraries as according to PEP 370:

The user site directory is added before the system site directories but after Python's search paths and PYTHONPATH. This setup allows the user to install a different version of a package than the system administrator but it prevents the user from accidently overwriting a stdlib module. Stdlib modules can still be overwritten with PYTHONPATH.

Sidenote

---

Setuptools use hack by inserting code in easy_install.pth file which is placed in site-packages directory. This code makes packages installed with setuptools come before other packages in sys.path so they shadow other packages with the same name. This is referred to as sys.path modification in the table comparing setuptools and pip. This is the reason console scripts use user installed libraries when you install with setup.py install instead of using pip.

---

Taking all of the above into account the reason for what you observe might be caused by:

• PYTHONPATH pointing to directories with system-wide installed libraries
• Having system-wide libraries installed using sudo python.py install (...)
• Having OS influence sys.path construction in some way

In the first case either clearing PYTHONPATH or adding path to user installed library to the beginning of PYTHONPATH should help.
In the second case uninstalling system-wide libraries and installing them with distro package manager instead might help (please note that you never should use sudo with pip or setup.py to install Python packages).
In the third case it's necessary to find out how does OS influence sys.path construction and if there's some way of placing user installed libraries before system ones.

You might be interested in reading issue pip list reports wrong version of package installed both in system site and user site where I asked basically the same question as you:

Does it mean that having system wide Python packages installed with easy_install thus having them use sys.path manipulation breaks scripts from user bin directory? If so is there any workaround?

Last resort solution would be to manually place directory/directories with user installed libraries in the beginning of sys.path from your scripts before importing these libraries.

Having said that if your users do not need direct access to source code I would propose packaging your app together with all dependencies using tool like pex or Platter into self-contained bundle.