cython: relative cimport beyond main package is not allowed

Question

I am trying to use explicit relative imports in cython. From the release notes it seems like relative imports should work after cython 0.23, and I'm using 0.23.4 with python 3.5. But I get this strange error that I cannot find many references to. The error is only from the cimport:

driver.pyx:4:0: relative cimport beyond main package is not allowed

The directory structure is:

    myProject/
        setup.py
        __init__.py
        test/
            driver.pyx
            other.pyx
            other.pxd

It seems like I'm probably messing up in setup.py so I included all the files below.

setup.py

from setuptools import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [
    Extension('other', ['test/other.pyx'],),
    Extension('driver', ['test/driver.pyx'],),
]

setup(
    name='Test',
    ext_modules=ext_modules,
    include_dirs=["test/"],
    cmdclass={'build_ext': build_ext},
)

driver.pyx

#!/usr/bin/env python
from . import other
from . cimport other

other.pyx

#!/usr/bin/env python

HI = "Hello"

cdef class Other:
    def __init__(self):
        self.name = "Test"

    cdef get_name(self):
        return self.name

other.pxd

cdef class Other:
    cdef get_name(self)

I've tried moving __init__.py into test/. I've tried running setup.py in the test directory (adjusting the include_dirs appropriately). They both give the same error.

If I do cimport other and remove the . it works but this is a toy example and I need relative imports so other folders can import properly. This is the only example I can find for this error and I'm pretty confident my issue is different.

Answer 1

The only other example I could find of this error was in hal.pyx of the machinekit project. I was pretty confident that this was a different error but today I realized that after that error was solved machinekit was working which means explicit relative imports must work. Their setup.py file refers to linuxcnc which isn't in the directory tree but I guess is created at some point during compile time. The important thing is that the include_dirs includes the parent directory instead of the child directory.

Translated to my project structure it would mean I put myProject in include_dirs instead of test/. After read this guide for the second time I finally started to understand a little of how python thinks of packages. The problem was that the include_dirs was the child directory. It seems like this effectively made cython view it as a single flat directory in which case no relative imports would be allowed? An error like this might have made it more clear:

ValueError: Attempted relative import in non-package

I still don't have a sufficiently deep understanding to know exactly what was going on but luckily the solution was relatively simple. I just changed the include_dirs to make cython recognize the nested file structure:

from setuptools import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [
    Extension('other', ['test/other.pyx'],),
    Extension('driver', ['test/driver.pyx'],),
]

setup(
    name='Test',
    ext_modules=ext_modules,
    include_dirs=["."],
    cmdclass={'build_ext': build_ext},
)

It all works now!

Answer 2

There are at least four solutions to the Cythonization error (these results are with cython == 0.29.24):

1. adding the file example_package/__init__.pxd and changing the names of the Extensions being built to be submodules of the module being built, i.e., example_package.other and example_package.driver (in the question these would be Test.other and Test.driver).
   
   This change is anyway necessary for importing the installed submodules driver and other, as described below. Note that the installed package is actually a namespace package in this case, due to absence of the keyword parameter and argument packages=['example_package'], as discussed below.

2. adding the file example_package/__init__.py and changing the names of the Extensions being built to be submodules of the module being built, i.e., example_package.other and example_package.driver. Even in this case, where an __init__.py is present, the installed package example_package will be a namespace package. Turning it into a regular package requires passing packages=['example_package'] to the function setuptools.setup.
   
   Likewise to adding an __init__.pxd, this change is necessary for importing the installed submodules.

3. adding the file example_package/__init__.pxd and changing the cimport statement to an absolute cimport inside the file example_package/driver.pyx (The package builds and installs with this alternative, but does not import, due to the need to also change the names of Extensions):

    from . import other
    from example_package cimport other
   

4. adding the file example_package/__init__.py and changing the cimport statement to an absolute cimport inside the file example_package/driver.pyx, as done in the previous item. The package builds and installs with this, but does not import.

The question is explicitly asking for relative imports, so in that sense, the first two alternatives are the answers to the question, because they do work with relative imports.

Either of the four changes listed above avoids the following error:

Error compiling Cython file:
------------------------------------------------------------
...
from . import other
from . cimport other
^
------------------------------------------------------------

example_package/driver.pyx:2:0: relative cimport beyond main package is not allowed

but as already noted above, and also discussed below, the change of Extension names of the first or second alternatives is necessary for importing the installed submodules (additionally passing the parameter and keyword argument packages=[PACKAGE_NAME] in the fourth alternative allows the Python package example_package to import, but not its submodules driver and other).

Modified setup.py

The file setup.py that I recommend, with all additional changes (not only those changes necessary for building and installing, listed above) is:

"""Installation script."""
import os
import setuptools


try:
    from Cython.Build import cythonize
    cy_ext = f'{os.extsep}pyx'
except ImportError:
    # this case is intended for use when installing from
    # a source distribution (produced with `sdist`),
    # which, as recommended by Cython documentation,
    # should include the generated `*.c` files,
    # in order to enable installation in absence of `cython`
    print('`import cython` failed')
    cy_ext = f'{os.extsep}c'


PACKAGE_NAME = 'example_package'


def run_setup():
    """Build and install package."""
    ext_modules = extensions()
    setuptools.setup(
        name=PACKAGE_NAME,
        ext_modules=ext_modules,
        packages=[PACKAGE_NAME],
        package_dir={PACKAGE_NAME: PACKAGE_NAME})


def extensions():
    """Return C extensions, cythonize as needed."""
    extensions = dict(
        other=setuptools.extension.Extension(
            f'{PACKAGE_NAME}.other',
            sources=[f'{PACKAGE_NAME}/other{cy_ext}'],),
        driver=setuptools.extension.Extension(
            f'{PACKAGE_NAME}.driver',
            sources=[f'{PACKAGE_NAME}/driver{cy_ext}'],))
    if cy_ext == f'{os.extsep}pyx':
        ext_modules = list()
        for k, v in extensions.items():
            c = cythonize(
                [v],
                # show_all_warnings=True  # this line requires `cython >= 3.0`
                )
            ext_modules.append(c[0])
    else:
        ext_modules = list(extensions.values())
    return ext_modules


if __name__ == '__main__':
    run_setup()

Other changes

The other changes in this answer are not necessary for successfully building and installing the package, but recommended for other reasons. For some of the other changes, I describe the motivation below.

Note that:

• only adding example_package/__init__.pxd or example_package/__init__.py is insufficient, and
• only changing the Extension names is insuffient, and
• only changing the cimport statement to from example_package cimport other is insufficient.

Two of these changes are needed together for building and installing, i.e., one of the four alternatives listed earlier.

For being able to import the extension modules built from the Cython sources driver.pyx and other.pyx, it is also necessary to change the names of the extensions to:

• Extension('example_package.other', ...)
• Extension('example_package.driver', ...)

Note that this makes import work because now example_package has become a namespace package (CPython glossary entry):

>>> 
<module 'example_package' (namespace)>
>>> import example_package.driver
>>> import example_package.other

(Also, I have omitted the parameter include_dirs of setuptools.setup in the setup.py file that I used, and which I include below.)

These changes are needed for building and installing the package, and for importing the extension modules. For importing the installed package from Python in case it did not include any extensions modules (and thus had not become a namespace package):

• a file __init__.py needs to be added in the directory example_package/ (which in the question is the directory Test/), and
• the keyword argument packages=[example_package], needs to be passed to the function setuptools.setup.

Otherwise, the statement import example_package will raise a ModuleNotFoundError. The addition of an __init__.py file is also necessary to make the package a regular package (CPython glossary entry), which is usually what is intended, instead of a namespace package.

Whether to use an __init__.pxd

A regular Python package includes an __init__.py file. An __init__.pxd file is only relevant in case other packages need the *.pxd headers. If this is not the case, it seems that the file example_package/__init__.py suffices, since the four solutions above are essentially two solutions, each with either __init__.py or __init__.pxd as alternatives.

So my recommendation for files and their arrangement is:

.
├── example_package
│   ├── __init__.py
│   ├── driver.pyx
│   ├── other.pxd
│   └── other.pyx
└── setup.py

Both changes are needed

Only adding the __init__.pxd file raises the cythonization error:

Error compiling Cython file:
------------------------------------------------------------
...
from . import other
from . cimport other
^
------------------------------------------------------------

example_package/driver.pyx:3:0: relative cimport beyond main package is not allowed

and only changing the cimport statement (without an __init__.pxd) raises the cythonization error:

Error compiling Cython file:
------------------------------------------------------------
...
#!/usr/bin/env python
from . import other
from example_package cimport other
^
------------------------------------------------------------

example_package/driver.pyx:3:0: 'example_package.pxd' not found

Error compiling Cython file:
------------------------------------------------------------
...
#!/usr/bin/env python
from . import other
from example_package cimport other
^
------------------------------------------------------------

example_package/driver.pyx:3:0: 'example_package/other.pxd' not found

Naming the package

Above I write example_package as the package's name, though I did build and install the example also with the name Test/ as it is named in the question, in order to ensure that this indeed works, and so that the minimal changes needed are the __init__.pxd file and the from example_package cimport other.

For uniformity, I actually also renamed the directory to Test/ when running setup.py with this name of the package, but I am not on a case-sensitive filesystem at the moment, so I do not know whether a directory named test/ together with the keyword argument name='Test', in setup.py, as in the question, would have caused issues on a case-sensitive filesystem.

So:

• using Test as package name and Test as directory name worked for me for building and installing, and
• using test as package name and test as directory name worked for me for building and installing.

I recommend using another package name. Also, for reasons described below:

• Importing when the package is named Test is done with the statement import Test. Writing import test will import another package (see below).
• using test as package name does not import the installed test package, for reasons explained below, even if an __init__.py file is added.

In any case, for reasons explained below, my recommendation is to change the package name, even if it is intended to be an auxiliary package that is intended to be used only as a test harness for the main package.

Also, lowercase package naming is mandated by PEP 8, thus leading to test, which may be understood to be a directory of tests, which is not the case if this actually is intended to be an example of a main package.

The error that happens after building and installing, when the package and directory are named test is (the dots ... are the result of editing the actual output):

>>> import test
>>> test
<module 'test' from '.../lib/python3.9/test/__init__.py'>

In other words, CPython includes a package called test:

The test package contains all regression tests for Python as well as the modules test.support and test.regrtest.

Therefore, the name test cannot be used for an example package that is intended to be imported after installation (though the package does get built and installed, and even uninstalled by pip uninstall -y test, fine).

Another detail is that from test cimport other is actually wrong, even though it compiles, because had the built test package actually been magically imported somehow (in the presence of CPython's test package), at runtime this cimport statement would have defaulted to CPython's test package. Nonetheless, Cython's translation may transform this cimport to some other form that would have actually imported from test.other of the built package. Since the import of the installed test package appears to be impossible in the presence of CPython's test package, it is unknown whether this cimport would have raised a runtime error.

Also, note that:

Note: The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python.

Between all experiments, I run rm -rf build dist *.egg-info test/*.c. So before changing the file arrangement to the one shown earlier, what I used is the same with the question.

Renaming the package to example_package

I changed the name of the package to example_package, assuming that test/ contains the actual package to be installed, based on the argument given to the parameter name= within the file setup.py of the question.

The motivation for this renaming is that "test" or "tests" is usually used to name the directory of tests that accompany a Python package. There are many arrangements for such directories, and for how the tests are used. In the next section, I discuss my suggestion for arranging tests.

Regarding possibilities, arrangements other than what I describe in the next section have been used in general, including placing the tests in directories within the package itself. Given that the question writes myProject/, and has a file myProject/__init__.py, I am not sure whether the question actually uses such an arrangement.

In that case, though, driver and other would actually be test modules. Though installing the tests as a separate package (called Test in the question), which is what the module myProject/setup.py does, suggests that driver and other are the main package's modules, and thus that the main package is called "Test".

If not, i.e. if driver and other are actually test modules, and setup.py is not the main package's setup script, but instead a setup script that builds and installs an "auxiliary" package that is intended for only testing the main package (which may in this case be named "myProject", with a setup.py present in the directory that contains the directory myProject/ of the question), then my renaming of Test to example_package/ would not correspond to this being the main package. (It is also interesting to have a test-harness package that includes Cython code and thus requires to be compiled--and possibly installed.)

In that case, perhaps Test could be renamed instead to tests_of_example_package. In other words, in that case it is relevant to include the word "test" in the package's name, though it seems that qualifying the package as auxiliary to example_package is explicit. Explicit is better than implicit (PEP 20).

(Tests sometimes are arranged as packages (using __init__.py), even when not installing this as an auxiliary Python package (intended as only a test harness of the main Python package it accompanies). The motivation is to enable importing common modules of the test suite that are used by multiple test modules, but are not themselves modules that are run directly by the test runner.)

If this is the main package, then I assume that "Test" was used for the purpose of writing an example in the example. If so, then my only reason for renaming (lowercase aside) is to distinguish the main package itself from its tests.

Lowercase names for Python packages are mandated by PEP 8:

Python packages should also have short, all-lowercase names, although the use of underscores is discouraged.

The underscore in example_package is only for the sake of example.

Arranging tests

Tests might be placed in a test/ directory that is in the same directory with the directory that contains the Python package, and is named after the package. I strongly recommend this approach, for example (this tree was created with the program tree):

.
├── example_package
│   └── __init__.py
├── setup.py
└── tests
    └── module_name_test.py

For testing without accidentally importing the package example_package from its source directory, but from where it is installed (usually under site-packages/), I recommend in all cases first cding to the directory tests/ before running any tests. This is the most reliable approach to testing, does not rely on how each testing framework works, how the testing framework's various configuration options work, how the options interact with each other, nor how bugs in the testing framework itself affect testing.

In this way, the package source can be placed inside the directory example_package, without any reason to use any other directory arrangement.

Shebangs in Python modules

The shebang inside the *.pyx files can be removed, because it has no effect. The shebang line is treated by Cython as a Python comment line that is moved to inside a C comment somewhere later inside the *.c files that Cython generates from the *.pyx files. So it has no effect. I am not aware of any use of shebang lines in C sources that are intended to be compiled by directly calling gcc (or another C compiler), as Cython does (whether Cython calls gcc, or another compiler depends on the system, environment path, environment variables, and other information).

Also, the shebang is only relevant to Python modules that might be executed as executables. This is not intended to be the case for modules inside a Python package, so shebang lines are almost never used there.

An exception might be a package module that might be infrequently be run directly during development, e.g., for experimentation or debugging purposes. Nonetheless, such a module would be expected to have a __main__ stanza.

So Python modules to which a shebang is relevant, also typically have a __main__ stanza.

For completeness, setup.py is intended to be run as __main__, and does have a __main__ stanza, but the way setup scripts are run (when not using pip--using pip is strongly recommended) is by python setup.py, so there is no need for a shebang in setup.py (no shebang appears there in the question--I just mention this for completeness).

Importing setuptools in setup.py, instead of distutils

The distutils module]() is deprecated as of Python 3.10, as specified in PEP 632, and will be removed in Python 3.12.

Switching when Cython is absent to the extension .c, instead of .pyx

This is in accord with Cython recommendations:

It is strongly recommended that you distribute the generated .c files as well as your Cython sources, so that users can install your module without needing to have Cython available.

Uppercase names for module-scope variables in setup.py that remain unchanged ("constants")

Module-scope Python variables that are intended to be used as constants, i.e., remain unchanged after the initial assignment, are mandated by PEP 8 to have identifiers that are uppercase with underscores:

Constants are usually defined on a module level and written in all capital letters with underscores separating words. Examples include MAX_OVERFLOW and TOTAL.

Hence the identifier PACKAGE_NAME.

Formatted strings

I used formatted string literals, which require Python >= 3.6.

Arranging code as functions within the module setup.py

This is good practice in general, enabling naming different sections of code via function names, executing the code only when run as __main__, by including a __main__ stanza, and thus enabling importing setup.py and using specific functionality that may be relevant to outside code (e.g., an installation framework) without necessarily running all code--e.g., without running the function setuptools.setup.

The question presents a minimal working example, so a small setup.py is relevant in the question. I am writing this section as recommendation for what to do in actual packages, not in questions.

The same observation applies to module and function docstrings inside setup.py.

Also, I recommend the top-down arrangement of functions: callers above callees, because this layout is more readable.

I used os.extsep for generality, though using a dot I think will still work, and is more readable.

Arrangement of package

As I noted earlier, the only change to the question's example that was needed to avoid the build error "relative cimport beyond main package is not allowed" was the addition of either an __init__.py or an __init__.pxd, and either an absolute cimport inside driver.pyx or the renaming of the Extensions.

Removing the file __init__.py

In the final version, I removed the file __init__.py that is in the same directory with setup.py. My understanding is that this file has no effect in this example. If the example is intended to have test/ as the main package's directory, then any __init__.py would appear inside test/.

If test/ is actually an auxiliary package of tests for the main package, then the __init__.py would be part of the main package, and unrelated to the test/ package. However, in that case, it seems that there would be a setup.py file above myProject/, which would be responsible for building both the main package, and the test-harness package.

Using absolute imports

The default language_level in cython < 3.0.0 is 2, even on Python 3:

language_level (2/3/3str) Globally set the Python language level to be used for module compilation. Default is compatibility with Python 2. To enable Python 3 source code semantics, set this to 3 (or 3str) at the start of a module or pass the "-3" or "--3str" command line options to the compiler.

The question uses Python 3.5 and cython == 0.23.4, so this is the case.

The default Cython semantics are changing in cython >= 3.0.0:

The default language level was changed to 3str, i.e. Python 3 semantics, ...

With both Python 2 and Python 3 semantics (passing compiler_directives=dict(language_level=3), or installing the pre-release cython == 3.0.0a8), the first two solutions (which use relative imports) do work.

Nonetheless, absolute imports are recommended by PEP 8:

Absolute imports are recommended, as they are usually more readable and tend to be better behaved (or at least give better error messages) if the import system is incorrectly configured ...

Absolute imports are also robust to refactoring a package's structure. They are explicit, and explicit is better than implicit (PEP 20).

The resulting module driver.pyx after this change would be:

from example_package import other
from example_package cimport other

The setup.py code in this answer is based on what I have written in the file download.py of the Python package dd.