TLDR:
Is there a way to ask pip or poetry, which packages are not [package.dependencies] of other packages in project.toml, or do I manually need to parse parse the poetry.lock file and get the info that way?
And I'm asking that question because I want to:
Go from a python environment which has all versions pined to a project.toml which only lists the packages which were installed explicitly by the original creator of the environment via pip install ..., so I can more easily keep the project up to date.
Context
In this project we had just a system python running the project, in order to make the project stable, we used pip freeze > requirements.txt to get the list of all python packages, so the environment can be re-created. And later still migrated from that to poetry, by just adding all the pined versions into pyproject.toml.
This works however it can be hard to update anything, as all the versions are pined exactly.
Ideally I would want to remove all packages except for one which are "top level" dependencies.
What I mean by "top level"; we have pymongo==4.6.1 which has the package dependencies of dnspython = ">=1.16.0,<3.0.0", but we also have dnspython=="2.4.2" if I were to remove dnspython from the toml and re-lock, pymongo would still work as poetry would use pymongo's requirements to manage it. Which is what I want.
I understand there are still risks involved with removing all packages from the toml which are requirements of other packages on the list, but there are tests and it would give a better overview of what is required for the project to run, and allow packages to updated via relocking.
My idea is to:
- Remove all deps which are a requirement of some other dep
- Add ^ to remaining deps, to allow for minor and patch updates
- Optionally, rather then removing the packages from the toml, just move them into a group, such as [tool.poetry.group.sub_deps], and add * or => to the version, so we include them, even if the parent package drops them for some reason and they are required for the project.
I see there is also this question, where they suggest: pipreqs, but that seems to works based on scanning imports, which isn't reliable, as some packages import name is not the same as the package name, nor does it take into account packaged like psycopg2-binary which are used by the system. I'm less interested in removing unused packages, and more just having the toml require less manual version management.