What does pip-compile do? What is its use? (how do I maintain the contents of my requirements.txt file?)

Question

I read pip-compiles definition in pip-tools documentation but I don't understand how it works.
Q1: What is pip-compile's use?
Q2: What does compiling requirements.in to produce requirements.txt mean?
Q3: How do I maintain the contents of the requirements.txt file?

Answer 1

You want to be able to lock down the versions of all of the packages that your Python code depends on in your requirements.txt file. You want this file to include versions for not just the direct dependencies that your code imports directly, but also versions for all of the transitive dependencies as well, that is, the versions of modules that your directly dependent modules themselves depend on.

How do you maintain the contents of requirements.txt? You can use pip freeze > requirements.txt, but this is messy. It depends not on a clear list of what the direct and indirect dependencies of your app are, but rather on what happens to be in your environment at the time of creation. What you really want is to have a file in which you list the direct dependencies of your app, along with versions for each of them, and then somehow produce the appropriate requirements.txt file from that list such that it contains exactly versions for those direct dependencies as well as versions for the transitive dependencies needed by those direct dependencies.

The requirements.in file and pip-compile together give you this desired behavior. In requirements.in, you list just the direct dependencies of your app. Then you run pip-compile on that file to produce requirements.txt. The compile process will produce what you want -- a file that contains both the modules listed in requirements.in and the transitive dependencies of those modules.

UPDATE: Someone asked why you should go through this exercise to lock down all of the versions of the packages upon which your application relies. The reason for this is that if you don't do this, then whenever you rebuild your application, you will get a build that uses the latest (ie: different) versions of some or all of the packages that it uses. So what if a change is made to one of those packages that causes your app's behavior to change? Maybe it causes an exception to be thrown, killing your app. Or worse, it might cause a subtle change in behavior that is quite difficult to track down. You want to prevent both of these possibilities. Going through the process discussed by this question/answer locks down all of the versions of the packages that your application uses, preventing changes made to later versions of those packages from affecting the behavior of your application.