NumPy C-API: Undefined reference error trying to support numpy.float16

78 Views Asked by At

I am trying to build one of the ufunc examples from NumPy's documentation (see here), but I am unable to get the support for half-precision floating point numbers to work.

This is my minimal example:

pyproject.toml

[build-system]
build-backend = "mesonpy"
requires = ["meson-python==0.15", "numpy==1.26.3"]

[project]
name = "mypackage"
version = "0.1.0"
dependencies = ["numpy==1.26.3"]

meson.build - There's an open issue on GitHub regarding adding NumPy as a build dependency in Meson - the custom lookup in the code below follows what seems to be the current recommendation from the SciPy project on how to add the NumPy headers.

project(
    'mypackage',
    'c',
    version: '0.1.0'
)

py = import('python').find_installation(pure: false)

incdir_numpy = meson.get_external_property('numpy-include-dir', 'not-given')
if incdir_numpy == 'not-given'
  incdir_numpy = run_command(
    py,
    ['-c', '''
import os
import numpy
try:
    incdir = os.path.relpath(numpy.get_include())
except Exception:
    incdir = numpy.get_include()
print(incdir)
'''
    ],
    check: true
  ).stdout().strip()
  _incdir_numpy_abs = run_command(
    py,
    ['-c', '''
import os
import numpy
print(numpy.get_include())
'''
    ],
    check: true
  ).stdout().strip()
else
  _incdir_numpy_abs = incdir_numpy
endif
inc_np = include_directories(incdir_numpy)
numpy_nodepr_api = ['-DNPY_NO_DEPRECATED_API=NPY_1_26_API_VERSION']
np_dep = declare_dependency(include_directories: inc_np, compile_args: numpy_nodepr_api)

py.extension_module(
    'npufunc',
    'npufunc.c',
    dependencies: np_dep,
    install: true
)

npufunc.c - This is just straightforward copy & paste from the example. I removed the methods for other data types (single, double, etc.), since they don't cause trouble.

#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include "numpy/ndarraytypes.h"
#include "numpy/ufuncobject.h"
#include "numpy/halffloat.h"
#include <math.h>

static PyMethodDef LogitMethods[] = {
    {NULL, NULL, 0, NULL}
};

static void half_float_logit(char **args, const npy_intp *dimensions,
                            const npy_intp *steps, void *data)
{
    npy_intp i;
    npy_intp n = dimensions[0];
    char *in = args[0], *out = args[1];
    npy_intp in_step = steps[0], out_step = steps[1];

    float tmp;

    for (i = 0; i < n; i++) {
        tmp = npy_half_to_float(*(npy_half *)in);
        tmp /= 1 - tmp;
        tmp = logf(tmp);
        *((npy_half *)out) = npy_float_to_half(tmp);

        in += in_step;
        out += out_step;
    }
}

PyUFuncGenericFunction funcs[1] = {&half_float_logit};

static char types[2] = {NPY_HALF, NPY_HALF};

static struct PyModuleDef moduledef = {
    PyModuleDef_HEAD_INIT,
    "npufunc",
    NULL,
    -1,
    LogitMethods,
    NULL,
    NULL,
    NULL,
    NULL
};

PyMODINIT_FUNC PyInit_npufunc(void)
{
    PyObject *m, *logit, *d;

    import_array();
    import_umath();

    m = PyModule_Create(&moduledef);
    if (!m) {
        return NULL;
    }

    logit = PyUFunc_FromFuncAndData(funcs, NULL, types, 1, 1, 1,
                                    PyUFunc_None, "logit",
                                    "logit_docstring", 0);

    d = PyModule_GetDict(m);

    PyDict_SetItemString(d, "logit", logit);
    Py_DECREF(logit);

    return m;
}

But I am getting an error from the linker when using GCC 13.2 (via MSYS2 - I'm working on Windows 10):

[2/2] Linking target npufunc.cp312-win_amd64.pyd
FAILED: npufunc.cp312-win_amd64.pyd
"cc"  -o npufunc.cp312-win_amd64.pyd npufunc.cp312-win_amd64.pyd.p/npufunc.c.obj "-Wl,--allow-shlib-undefined" "-Wl,-O1" "-shared" "-Wl,--start-group" "-Wl,--out-implib=npufunc.cp312-win_amd64.dll.a" "C:\Program Files\Python312\python312.dll" "l" "-lkernel32" "-luser32" "-lgdi32" "-lwinspool" "-lshell32" "-lole32" "-loleaut32" "-luuid" "-lcomdlg32" "-ladvapi32"-,-" "-Wl,--end-group"
C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: npufunc.cp312-win_amd_amd64.pyd.p/npufunc.c.obj:npufunc.c:(.text+0x45): undefined reference to `npy_half_to_float'
C:/msys64/ucrt64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: npufunc.cp312-win_amd_amd64.pyd.p/npufunc.c.obj:npufunc.c:(.text+0x5a): undefined reference to `npy_float_to_half'
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

Switching to MSVC 19.38 doesn't help either:

[2/2] Linking target npufunc.cp312-win_amd64.pyd
FAILED: npufunc.cp312-win_amd64.pyd
"link"  /MACHINE:x64 /OUT:npufunc.cp312-win_amd64.pyd npufunc.cp312-win_amd64.pyd.p/npufunc.c.obj "/release" "/nologo" "/OPT:REF" "/DLL" "/IMPLIB:npufunc.cp312-win_amd64.lib" "C:\Program Files\Python312\libs\python312.lib" "kernel32.lib" "user32.lib" "gdi32.lib" "winspool.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "uuid.lib" "comdlg32.lib" "advapi32.lib"       
   Creating library npufunc.cp312-win_amd64.lib and object npufunc.cp312-win_amd64.exp
npufunc.c.obj : error LNK2019: unresolved external symbol npy_half_to_float referenced in function half_float_logit       
npufunc.c.obj : error LNK2019: unresolved external symbol npy_float_to_half referenced in function half_float_logit       
npufunc.cp312-win_amd64.pyd : fatal error LNK1120: 2 unresolved externals
ninja: build stopped: subcommand failed.

I understand that dealing with 16-bit floating point numbers can be somewhat "iffy", due to a lack of hardware support on most CPUs. But I'm including the correct header "numpy/halffloat.h", that defines the two conversion functions npy_half_to_float and npy_float_to_half.

Why am I seeing this error? Any pointers in the right direction would be very much appreciated.

0

There are 0 best solutions below