How to read top-level requirements (as from requirements.in) from setup.py and write back pinned requirements (as in requirements.txt)?

Question

I have a package with a setup.py file and want to use pip-tools to pin my dependencies for production.

Let's say my setup.py looks as follows:

#!/usr/bin/env python
import pathlib
from setuptools import setup, find_packages

setup(
    author="Foo",
    description="My package",
    install_requires=["package1==1.0", "package2==2.0"],
    extras_require={
        "top_level": ["package1", "package2"],
    },
    version="0.1.0",
)

How could I here track my top level requirements within a setup.py and write them back to the same file within the section install_requires? Would I just pip-compile from setup.py into a requirements.txt and read the contents from this file into install_requires?

Answer 1

You can track top-level deps in requirements.in and use pip-compile to lock deps in requirements.txt. Then you can make a setup.py and use both requirements files accordingly. For example:

def get_requirements(filename) -> list[str]:
    ...

setup(
    author="Foo",
    description="My package",
    install_requires=get_requirements('requirements.txt'),
    extras_require={"top_level": get_requirements('requirements.in')},
    version="0.1.0",
)

See an example of Sentry's setup.py.

Answer 2

You'd be better off not using a setup.py file at all, since as you've evidently found out, it'd be a bit fiddly to put your locked requirements back in there, not to mention there are more modern systems these days than running arbitrary code to set up a package.

Instead, I'd recommend (at the time of writing) to use Hatch and hatch-requirements-txt (if you really want a requirements.txt; you could use tomli_w to just modify pyproject.toml too...). (You can convert a setup.py project to Hatch with hatch new --init.)

You can then do something like (untested, just an idea)

python -c 'import tomllib, pathlib; print(*tomllib.loads(pathlib.Path("pyproject.toml").read_text())["project"]["optional-dependencies"]["toplevel"], sep="\n")' | pip-compile -

to generate requirements.txt from the optional dependencies in the TOML file.