Set setuptools to create cimportable package with headers availible

Question

I try to implement the answer https://stackoverflow.com/a/57480599/7482208, but I am stuck on cimporting one package from another.

The code is here: https://github.com/iamishalkin/setuptools_cython_question

What I want is to have one independent package wrap from wrapper folder such that you can use it without cust package.

And I also want to be able to create custom functions by inheriting FuncWrapper class from wrap.

What I do:

• Firstly I run python setup.py bdist_wheel in wrapper folder (this is what I am doing wrong, I suppose, as it gives me only binary file)
• Next pip instal dist/(some_name).whl
• Next I add import wrap and include_dirs=wrap.get_include() to custom/setup.py like it is done in numpy
• I run python setup.py bdist_wheel in custom folder and this fails, on the first step no files except binary one were created

So the question is: how to add .pxd files to the final package.

I also tried sdist which does not compile cython code but just copies it.

Answer 1

As I said in a comment, the Cython documentation recommends putting .pxd files in package_data to install them. This necessitates a slightly different structure:

| setup.py
+ wrapper
   | wrap.pxd
   | wrap.pyx
   | __init__.py # just so it's recognised as a package
                 # may be unnecessary with recent Python versions

setup.py then creates a "package" called wrapper (this is modified from your version so it's possible it could be simplied further):

from setuptools import setup, Extension
from Cython.Build import cythonize
from Cython.Distutils import build_ext

NAME = "some_name"

ext_abc = Extension(name="wrapper.wrap",
                    sources=["wrapper/wrap.pyx"]
                    )

EXTENSIONS = [
    ext_abc
]

if __name__ == "__main__":
    setup(
        zip_safe=False,
        name=NAME,
        packages=["wrapper"],
        cmdclass={"build_ext": build_ext},
        ext_modules=cythonize(EXTENSIONS, language_level=3),
        package_data = {
            "wrapper": ["*.pxd"],
    },
        )

Note that I've changed the name of the extension to "wrapper.wrap" to ensure that it's installed as part of the package. The package_data is then able to recognised .pxd files as part of wrapper that you want to install. This doesn't work unless you put it in a "package".

You then install it. I just installed it with python3 setup.py install but I'm sure going through a wheel does largely the same thing.

---

For another module to use you file it's very simple:

from wrapper.wrap cimport FuncWrapper

The setup.py for that other module need have nothing special - you definitely don't need anything like include_dirs=wrap.get_include().

If you want to have an interface where you don't need submodules so can just do

from wrapper cimport FuncWrapper

then just use an __init__.py containing:

from .wrap import *

and an __init__.pxd containing:

from wrapper.wrap cimport * # relative import is a little broken in Cython I think

---

I'm sure there are other ways of doing this - I've only really used setuptools for compiling Cython stuff and never really worried about distributing too much so am not an expert - but this looks to be the standard approach.

Answer 2

In case anyone else runs into this problem, I was able to get this working using a MANIFEST.in file placed in the top-level directory. The contents are simply:

include MANIFEST.in
global-include *.pyx
global-include *.pxd
global-exclude *.pyc *.pyo *.pyd

I use this in addition to what the selected answer recommends.