Extension modules: marshalling void * to bytearray (and/or vice versa)

Question

Doing some experiments using python (means python3) for preparing data (also as sending them to wire - SPI) shows it is slow (having limited system). So I was thinking of creation of extension module written in C to defer the critical stuff to. I would like either:

• the python script would have access to memory block created by the malloc() in the extension module hopefully transparently convertible to bytearray
• the extension module would get the pointer to bytearray object created in python hopefully transparently convertible to void *

The goal is to have zero copy also as zero conversion memory block accessible by both python (as bytearray) and extension module (as void *).

Is there any method, how to achieve this?

Answer 1

OK, it seems to be simpler than expected ;-)

• bytearray provides direct support for accessing underlying memory block, which is exactly what is needed
• there is a format specifier for extracting bytearray object from function call argument list

C extension module [test.c]:

#include <Python.h>
#include <stdint.h>

/* Forward prototype declaration */
static PyObject *transform(PyObject *self, PyObject *args);

/* Methods exported by this extension module */
static PyMethodDef test_methods[] =
{
     {"transform", transform, METH_VARARGS, "testing buffer transformation"},
     {NULL, NULL, 0, NULL}
};


/* Extension module definition */
static struct PyModuleDef test_module =
{
   PyModuleDef_HEAD_INIT,
   "BytearrayTest",
   NULL,
   -1,
   test_methods,
};


/* 
 * The test function 
 */
static PyObject *transform(PyObject *self, PyObject *args)
{
    PyByteArrayObject *byte_array;
    uint8_t           *buff;
    int                buff_len = 0;
    int                i;


    /* Get the bytearray object */
    if (!PyArg_ParseTuple(args, "Y", &byte_array))
        return NULL;

    buff     = (uint8_t *)(byte_array->ob_bytes); /* data   */
    buff_len = byte_array->ob_alloc;              /* length */

    /* Perform desired transformation */
    for (i = 0; i < buff_len; ++i)
        buff[i] += 65;

    /* Return void */
    Py_INCREF(Py_None);
    return Py_None;
}


/* Mandatory extension module init function */
PyMODINIT_FUNC PyInit_BytearrayTest(void)
{
    return PyModule_Create(&test_module);
}

C extension module build/deployment script [setup.py]:

#!/usr/bin/python3
from distutils.core import setup, Extension

module = Extension('BytearrayTest', sources = ['test.c'])

setup (name = 'BytearrayTest',
       version = '1.0',
       description = 'This is a bytearray test package',
       ext_modules = [module])

Build/install the extension module:

# ./setup.py build
# ./setup.py install

Test it:

>>> import BytearrayTest
>>> a = bytearray(16); a
bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
>>> BytearrayTest.transform(a); a
bytearray(b'AAAAAAAAAAAAAAAA')
>>>