Building Python package containing multiple Cython extensions

Question

I have the following directory structure:

testcython/
    setup.py
    testcython/
        __init__.py
        foo.pyx
        stuff.py
        bar/
            __init__.pxd
            __init__.py
            bar.pxd
            bar.pyx

where the file contents are as follows:

bar.pxd

# cython: language_level=3

cdef int square(int x)

bar.pyx

# cython: language_level=3

cdef int square(int x):
    return x * x

foo.pyx

# cython: language_level=3

import cython
cimport numpy as np
import numpy as np

from .Bar cimport square

def do_square(x):
    return square(x)

stuff.py

from __future__ import print_function

from .Foo import do_square

def do():
    print(do_square(2))

setup.py

import os, sys

from Cython.Build import build_ext, cythonize
from setuptools import setup, Extension, find_packages

def ext_modules():
    import numpy as np

    include_dirs = ['.', np.get_include()]
    root_dir = os.path.abspath(os.path.dirname(__file__))
    bar_ext = Extension(
        "Bar",
        sources=[root_dir + "/testcython/bar/bar.pyx"],
        include_dirs=include_dirs,
    )
    foo_ext = Extension(
        "Foo",
        sources=[root_dir + "/testcython/foo.pyx"],
        include_dirs=include_dirs
    )
    exts = [bar_ext, foo_ext]

    return cythonize(exts)

REQUIREMENTS = [
    "numpy",
    "cython"
]

setup(
    name="testcython",
    packages=find_packages(),
    ext_package="testcython",
    ext_modules=ext_modules(),
    cmdclass={"build_ext" : build_ext},
    zip_safe=False,
    install_requires=REQUIREMENTS
)

Question

The problem is that when I attempt to install this (with pip install -e . in the top testcython directory), I get the following errors from Cython:

Complete output from command python setup.py egg_info:

    Error compiling Cython file:
    ------------------------------------------------------------
    ...

    import cython
    cimport numpy as np
    import numpy as np

    from .Bar cimport square
    ^
    ------------------------------------------------------------

    testcython/foo.pyx:7:0: relative cimport beyond main package is not allowed

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
    import numpy as np

    from .Bar cimport square

    def do_square(x):
        return square(x)
              ^
    ------------------------------------------------------------

This answer (cython: relative cimport beyond main package is not allowed) implies that including the root dir ('.') in the include_dirs argument(s) of the Extension objects should resolve the issue.

Whilst this part of the Cython documentation mentions to use zip_safe=False in the args of setup when using the setuptools package.

As you can see from my setup.py file above, I have included both of these - yet I still receive the error above.

Note: If I change the name of the extensions (in Extension constructor) from Bar and Foo to testcython.Bar and testcython.Foo, respectively, then I get a different error:

Complete output from command python setup.py egg_info:

    Error compiling Cython file:
    ------------------------------------------------------------
    ...

    import cython
    cimport numpy as np
    import numpy as np

    from .Bar cimport square
    ^
    ------------------------------------------------------------

    testcython/foo.pyx:7:0: 'testcython/Bar/square.pxd' not found

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
    import numpy as np

    from .Bar cimport square

    def do_square(x):
        return square(x)
              ^
    ------------------------------------------------------------

Answer 1

I resolved this issue, with the help of a colleague, so I'll mention the solution here in case it helps people in the future.

The problem is related to how the Cython modules are imported, and more specifically - where the .so file is placed upon building the extension. Originally, the Bar.so file was generated in the testcython directory - such that when attempting import from the bar sub-module, it couldn't find the corresponding shared object file.

To solve this I needed to use the name "bar.bar" when creating this extension, this then results in the .so file being generated into the testcython/bar directory. Then, in foo.pyx, to use members from this bar module the import had to be changed to from testcython.bar.bar cimport <name>.

Note:

Additionally, the function square shown in the question cannot be used from another Cython module in this form as no __pyx_capi__ is generated for free cdef functions. Instead, this function has to be wrapped in some cdef class as a static method in order to use it from another Cython module, i.e.:

cdef class Square:
    @staticmethod
    cdef int square(int x)

Then this can be imported, in foo.pyx for example, with from testcython.bar.bar cimport Square. The class Square then essentially acts like a "namespace".