How to create a nested package space for python packages?

Question

I'm restructuring the python libraries our company is developing into various groups. To avoid polluting the top level module namespace I'd like to group everything under a top level 'companyname' package, so we'll have 'coname.utils', 'coname.qa', 'coname.api', and so on. Several of these are expected to be distributed with our product, or publicly installable. 'qa' will be purely internal.

Google does something similar. E.g., the protobuf library is available as "google.protobuf" in the module path. However, it's not particularly clean: the .pth file installed by the protobuf package looks like this:

import sys, types, os;p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('google',));ie = os.path.exists(os.path.join(p,'__init__.py'));m = not ie and sys.modules.setdefault('google', types.ModuleType('google'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)

I think it does that to fool the import engine because there's no __init__.py in the "google" directory. But... not elegant.

Is there a well established way of achieving this? I don't mind making any of the "coname-*" packages require a "coname-top" package just to get the __init__.py in there. I'm as-yet unsure how to convince setuptools to treat the package as not at the top of the module tree, nor if it's possible to create sub-packages from one tree.

To clarify, I'm specifically asking how to set up the above so that coname-qa can be distributed and installed separately from coname-api, for example. It is reasonable that both would depend on coname-tools.

Answer 1

1. The directory a needs to be a package. Add an __init__.py file to make it a package, which is a step up from being a simple directory.

2. The directory b also needs to be a subpackage of a. Add an __init__.pyfile.

3. The directory test should probably also be a package. Hard to say if this is necessary or not. It's usually a good idea for every directory of Python modules to be a formal package.

4. In order to import, the package needs to be on sys.path; this is built from the PYTHONPATH environment variable. By default the installed site-packages and the current working directory are (effectively) the only two places where a package can be found.

That means that a must either be installed, or, your current working directory must also be a package one level above a.

OR, you need to set your PYTHONPATH environment variable to include a.

Answer 2

What you are trying to do is create a namespace package. You can use setuptools to accomplish this.

See https://packaging.python.org/guides/packaging-namespace-packages/#pkgutil-style-namespace-packages

Set up a directory structured like this:

coname-utils/
    setup.py
    coname/
        utils/
            __init__.py
            ...
        ...

coname-qa/
    setup.py
    coname/
        qa/
            __init__.py
            ...
        ...

coname-api/
    setup.py
    coname/
        api/
            __init__.py
            ...
        ...

There are a few variations, depending on what tools you intend to use for deployment. You can look at the link above for details on that.

This directory structure segregates the portions of your namespace that you want to be separately installable. You can also version the different subpackages separately, which can be nice for maintenance. Presumably you might have dependencies between some of your subpackages. You might add code to one subpackage that relies on api members in another subpackage that only exist in some versions of that subpackage. In the setup.py files you can define what versions of dependencies are required.

See setuptools documentation for details on how to write the setup.py

Once you have this set up, you should be able to build your project with setuptools and you can separately install utils, qa, api, etc. Once installed in an environment, you can import them with "import coname.utils", "import coname.qa", etc.