How can I create a single Python package from a Cython files (.pyx) that includes multiple .pyx files using build package

Question

I'm trying to build a python package of a collection of Cython functions under one package name, using include statement in Cython, following the idea of this.

Here's the file organization (root directory has subdirectory mypkg):

• setup.py

from Cython.Distutils.build_ext import new_build_ext
from setuptools import setup
from setuptools.extension import Extension

ext_modules = [
Extension(
"mypkg.main",
sources=["mypkg/main.pyx"],
),
]

setup(
ext_modules=ext_modules,
cmdclass={"build_ext": new_build_ext},
requires=['cython'],
)

• pyproject.toml:

[build-system]
build-backend = "setuptools.build_meta"
requires = [
"setuptools>=64.0.0",
"cython>=3.0.0",
"tomli",
]

[project]
name = "mypkg"
version = "0.1.0"
description = "Simple Cython example with multiple included .pyx files"

• mypkg/main.pyx:

include "add.pyx"
include "prod.pyx"

• mypkg/add.pyx:

def add(float a, float b):
    return a + b

• mypkg/prod.pyx:

def prod(float a, float b):
    return a * b

• empty mypkg/__init__.py
• empty mypkg/__init__.pxd

---

With this file structure, when I try to build Python package, cython build_ext module fails to find the include files, add.pyx. What I'm missing here and how can I build package mypkg and be able to import and use the functions add() and prod() in Python?
I'm using:

• python 3.8.16.
• Cython 3.0.2
• setuptools 67.8.0
• build package from pip install build

Here's error message:

(...)
running build_ext
Traceback (most recent call last):
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
    main()
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/opt/conda/envs/py_3.8/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/build_meta.py", line 434, in build_wheel
    return self._build_with_temp_dir(
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/build_meta.py", line 419, in _build_with_temp_dir
    self.run_setup()
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/build_meta.py", line 341, in run_setup
    exec(code, locals())
  File "<string>", line 13, in <module>
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/__init__.py", line 103, in setup
    return distutils.core.setup(**attrs)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/dist.py", line 989, in run_command
    super().run_command(command)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 364, in run
    self.run_command("build")
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/dist.py", line 989, in run_command
    super().run_command(command)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/_distutils/command/build.py", line 131, in run
    self.run_command(cmd_name)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
    self.distribution.run_command(command)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/dist.py", line 989, in run_command
    super().run_command(command)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
    self.build_extensions()
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
    self._build_extensions_serial()
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
    self.build_extension(ext)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/Cython/Distutils/build_ext.py", line 122, in build_extension
    new_ext = cythonize(
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 990, in cythonize
    module_list, module_metadata = create_extension_list(
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 839, in create_extension_list
    kwds = deps.distutils_info(file, aliases, base).values
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 691, in distutils_info
    return (self.transitive_merge(filename, self.distutils_info0, DistutilsInfo.merge)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 700, in transitive_merge
    return self.transitive_merge_helper(
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 706, in transitive_merge_helper
    deps = extract(node)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 671, in distutils_info0
    cimports, externs, incdirs = self.cimports_externs_incdirs(filename)
  File "Cython/Utils.py", line 129, in Cython.Utils.cached_method.wrapper
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 558, in cimports_externs_incdirs
    for include in self.included_files(filename):
  File "Cython/Utils.py", line 129, in Cython.Utils.cached_method.wrapper
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/Cython/Build/Dependencies.py", line 540, in included_files
    include_path = self.context.find_include_file(include, source_file_path=filename)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/Cython/Compiler/Main.py", line 302, in find_include_file
    error(pos, "'%s' not found" % filename)
  File "/tmp/build-env-q2v7shn5/lib/python3.8/site-packages/Cython/Compiler/Errors.py", line 178, in error
    raise InternalError(message)
Cython.Compiler.Errors.InternalError: Internal compiler error: 'add.pyx' not found

ERROR Backend subprocess exited when trying to invoke build_wheel

---

I was expecting a built python pakcage mypkg and be able to import the package in Python as import mypkg.main and run two functions: mypkg.main.add(1.0, 2.0) and mypkg.main.prod(2.0, 3.0). What actually happened was the build process fails to find the included file add.pyx and build fails.

Answer 1

maybe try modifying the setup.py and main.pyx for the following changes.

$ diff setup.py setup.py.orig
8,9c8
< #sources=["mypkg/main.pyx"],
< sources=["mypkg/main.pyx", "mypkg/add.pyx", "mypkg/prod.pyx"],
---
> sources=["mypkg/main.pyx"],

$ diff mypkg/main.pyx.orig mypkg/main.pyx
1,2c1,2
< include "add.pyx"
< include "prod.pyx"
---
> from .add import add
> from .prod import prod

$ tree
.
├── mypkg
│   ├── add.c
│   ├── add.pyx
│   ├── __init__.py
│   ├── main.c
│   ├── main.pyx
|   ├── main.pyx.orig
│   ├── prod.c
│   └── prod.pyx
├── pyproject.toml
├── setup.py
└── setup.py.orig

Then it seems to build:

$ python setup.py build_ext --inplace
running build_ext
Warning: Multiple cython sources found for extension 'mypkg.main': ['mypkg/main.pyx', 'mypkg/add.pyx', 'mypkg/prod.pyx']
See https://cython.readthedocs.io/en/latest/src/userguide/sharing_declarations.html for sharing declarations among Cython files.
building 'mypkg.main' extension
gcc -pthread -B /home/user/miniconda3/envs/pytorch/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/user/miniconda3/envs/pytorch/include -fPIC -O2 -isystem /home/user/miniconda3/envs/pytorch/include -fPIC -I/home/user/miniconda3/envs/pytorch/include/python3.8 -c mypkg/add.c -o build/temp.linux-x86_64-cpython-38/mypkg/add.o
gcc -pthread -B /home/user/miniconda3/envs/pytorch/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/user/miniconda3/envs/pytorch/include -fPIC -O2 -isystem /home/user/miniconda3/envs/pytorch/include -fPIC -I/home/user/miniconda3/envs/pytorch/include/python3.8 -c mypkg/main.c -o build/temp.linux-x86_64-cpython-38/mypkg/main.o
gcc -pthread -B /home/user/miniconda3/envs/pytorch/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/user/miniconda3/envs/pytorch/include -fPIC -O2 -isystem /home/user/miniconda3/envs/pytorch/include -fPIC -I/home/user/miniconda3/envs/pytorch/include/python3.8 -c mypkg/prod.c -o build/temp.linux-x86_64-cpython-38/mypkg/prod.o
gcc -pthread -B /home/user/miniconda3/envs/pytorch/compiler_compat -shared -Wl,--allow-shlib-undefined -Wl,-rpath,/home/user/miniconda3/envs/pytorch/lib -Wl,-rpath-link,/home/user/miniconda3/envs/pytorch/lib -L/home/user/miniconda3/envs/pytorch/lib -Wl,--allow-shlib-undefined -Wl,-rpath,/home/user/miniconda3/envs/pytorch/lib -Wl,-rpath-link,/home/user/miniconda3/envs/pytorch/lib -L/home/user/miniconda3/envs/pytorch/lib build/temp.linux-x86_64-cpython-38/mypkg/add.o build/temp.linux-x86_64-cpython-38/mypkg/main.o build/temp.linux-x86_64-cpython-38/mypkg/prod.o -o /home/user/younseo/mypkg/main.cpython-38-x86_64-linux-gnu.so