Faster pip install from git URI

Question

I have a Django web app which installs some requirements via Pip from a requirements.txt file.

I originally wrote my deployment scripts to run pip install -r requirements.txt every time I deployed, on the off chance that I'd changed the requirements.txt file in a commit I was deploying. This didn't have too much of a performance hit (in terms of time it took to deploy) because the packages were all on PyPI, and Pip does some sort of intelligent caching and won't download packages I already have every time.

Recently, though, I switched to using some packages that aren't yet on PyPI - just on Github. I can install them by adding lines like the following to my requirements.txt:

-e git+git://github.com/BowdoinOrient/django-storages-py3.git#egg=django-storages-py3
-e git+git://github.com/BowdoinOrient/topia.termextract.git#egg=topia.termextract

but this is quite slow - Pip seems to check out the git repo every time, or at least do some kind of network activity, regardless of whether that Github repo has been updated recently.

I thought I might be able to keep Pip from doing this by specifying exact commits I want to install, like so:

-e git+git://github.com/BowdoinOrient/django-storages-py3.git@83f18f5ccf39b5be230c6fc24d3b0b35c98277db#egg=django-storages-py3
-e git+git://github.com/BowdoinOrient/topia.termextract.git@2effd5f7274fb962292503d6d16938e68497059e#egg=topia.termextract

but Pip will always slow down on these lines and check Github for some kind of additional information, even though I already have those modules installed at those commits.

Is there any way to speed Pip up over these steps? Or something other than Pip that's faster that I should be using? I looked at Curdling but I don't think it's what I need.

Thanks!

Answer 1

Fixed my own problem: removing the -e (--editable) flag from VCS lines in the requirements.txt file keeps pip from cloning them if they already exist at the correct commit hash.

Documentation on pip's -e flag.

I also had the wrong egg name of one of the repositories as well, so watch out for that if you run across this problem too.