Maintaining a professional Python system

Question

Ok, I confess that this question is probably a bit too meta for this forum but let me describe the situation. I had a huge Python project with like 10 packages. I am using git. A commit triggers a test on my remote server (teamcity) and if I feel like it I install it on the remote server (by creating a whole new conda environment). This kind of worked well but doesn't scale nicely and I have read about setuptools.

In the Java world I have worked with Nexus before and in combination with Maven that was just paradise. I was hoping setuptools brings me a few steps closer to such a situation.

So I split my project into a few projects (let's say mother and one child) and hence had to install the underlying project locally (to make sure the child project can access it). Let's say my base project (mother) is pytools:

from setuptools import setup

setup(
    name='pytools',
    version='0.4',
    packages=['pytools'],
    description='', requires=['pandas'],
    test_suite="test.suite.suite"
)

I then use python setup.py install to install it locally to make sure the child can use it. I guess this is the fundamental mistake.

Once installed I have the egg at a very prominent position in my sys.path and I would like to avoid messing with sys.path. It's so high up that working with pytools within an IDE becomes hard.

I assume I should develop pytools in an environment only providing pandas and never install it in there? How is a second project accessing pytools then? One route would be to copy eggs over from the dist folder (do you commit the dist folder in git?) into any other project? Or tell setuptools the position of the repository to make sure the child project can install pytools?

How do you test then? A test on the server builds a whole new environment before it runs?

Before you go and point me to the documentation of setuptools (many thanks) you should assume I have read it. But as often it opens a whole new set of questions and the amazing power and flexibility it provides offers the possibility to make a few very poor decisions. I therefore ask the experts for helpful comments and maybe you are even willing to share how you run Python in a multi-project, multi-user environment.

Answer 1

Setuptools is not really the way to install things for production; it's for distributing your packages. The best way to set up a production environment is to use a virtualenv along with pip; install your packages locally inside the virtualenv, use pip freeze to create a requirements.txt file, then on production do pip install -r requirements.txt to rebuild your environment from that file.