Fortran C interface for program containing non-c-interoperable derived type

Question

I have a large Fortran code base for which I am trying to write a Python interface. I decided I would go the Cython route. However, I have derived types that are not C-interoperable. As an example,

type :: SomeType
  real, allocatable :: not_interoperable
  [other members]
contains
  procedure :: init
end type

...

subroutine init(this) bind(c)
  class(SomeType), intent(inout) :: this
...
end subroutine init

I would be completely happy not being able to access the member not_interoperable from Python, and would actually prefer not to expose it to the user anyway. i.e. some python class that includes only the other members would be perfectly fine, as long as functions inside Fortran can still "see" them. Here, the compiler fails because init has a polymorphic dummy argument. I can remove init as a type-bound procedure, and then make the change class(SomeType) -> type(SomeType), but then I still get the error that this is a dummy argument to a bind(c) procedure but is not C interoperable because the derived type is not C interoperable.

Is there any way around this? The only solution I can see is removing all non-C-interoperable members out of SomeType and making them global/module variables in the Fortran code, but I'd rather avoid that. Basically, I want a way to hide variables from Python (like those that are not C interoperable) while still keeping the Fortran code modular/preventing global variables.

---

Some more details:

My program roughly works as:

type(SomeType) :: instance
instance%init() ! no input arguments, but reads a file
instance%do_some_stuff()
instance%do_stuff_with_args(...) ! args are all C-interoperable
instance%finalize()

I would like a similar workflow in Python

from FortranModule import SomeType
instance = SomeType(...) # does not read a file, variables passed directly
instance.do_stuff()
instance.member = 3
instance.do_stuff_with_args(...)
instance.finalize()

But in Python, not_interoperable is never accessed. I want my Fortran code to still have it and use it, and it is still affected by SomeType%init, but I do not want/need it in Python. In principle I guess I can hide it as a global variable which gets modified by this function, but then (a) I can only have one instance and (b) I don't like global variables as a general rule (general consensus is that it reduces modularity and maintainability).

Also, just for clarity: I have a lot of non-interoperable types, and they are not as simple as real, allocatable, so rewriting all of them is not really realistic.

Answer 1

AFAIK there's no way to interoperate with C a derived type containing allocatable component). A solution is to expose to C only an opaque handle (void* C pointer) to a derived type instance. Then there's the choice to keep the components completely hidden from C, or to give C some pointers to them.

Illustration code:

Fortran part:

! ===========================================
! file: FC_bind_handle_f.f90
module somemodule
   use, intrinsic :: iso_c_binding
   implicit none
    
   ! c_float is needed only if one wants to 
   ! access %a directly from C
   type :: sometype
      integer(c_int), allocatable :: a(:)
   end type
    
contains

   ! example routine that processes the content of the type
   subroutine sometype_x2_f(x)
      type(sometype), intent(inout) :: x
      
      x%a = 2 * x%a
   end subroutine
   
end module
! ===========================================

! ===========================================
! file: somemodule_wrap.f90 
module somemodule_wrap
   use, intrinsic :: iso_c_binding
   use somemodule 
   implicit none

contains

   ! function that creates a sometype instance,
   ! allocates the %a component,
   ! and returns a C pointer to the instance 
   type(c_ptr) function sometype_create(len) bind(c)
      integer(c_int), value :: len
      
      type(sometype), pointer :: p
      
      allocate(p)
      allocate(p%a(len))
      sometype_create = c_loc(p)
   end function

   ! routine that releases a handle 
   subroutine sometype_free(ptr) bind(c)
      type(c_ptr) :: ptr
      
      type(sometype), pointer :: p
      
      call c_f_pointer(ptr,p)
      deallocate(p%a)   ! would actually be automatically deallocated
      deallocate(p)
      ptr = c_null_ptr
   end subroutine
   
   ! function that returns a C pointer to the %a component, 
   ! allowing to work directly on it from C
   type(c_ptr) function sometype_a_ptr(ptr, n) bind(c)
      type(c_ptr), value :: ptr
      integer(c_int), intent(out) :: n
      
      type(sometype), pointer :: p
      
      call c_f_pointer(ptr,p)
      n = size(p%a)
      sometype_a_ptr = c_loc(p%a)
   end function

   ! wrapper routine to a Fortran routine that performs 
   ! some processing on a sometype instance
   subroutine sometype_x2(ptr) bind(c)
      type(c_ptr), value :: ptr
    
      type(sometype), pointer :: p
    
      call c_f_pointer(ptr,p)
      call sometype_x2_f(p)
   end subroutine
   
end module
! ===========================================

C part:

//===========================================
//file: FC_bind_handle_c.c

#include <stdio.h>

void*  sometype_create(int len);
void*  sometype_free(void** h);
int*   sometype_a_ptr(void* h, int* n);
void*  sometype_x2(void* h);

int main(void) 
{
    // creates an instance of sometype with 6 elements in %a
    void* h = sometype_create( 6 );
    
    // get a pointer to %a and fill it. Alternatively one could 
    // define a "put" routine to avoid any access to %a from C
    int n;
    int* a = sometype_a_ptr(h, &n);
    for (int i = 0; i < n; i++) {
        a[i] = i;
    }
    
    // print the content of the instance
    for (int i = 0; i < n; i++) printf("%d  ",a[i]);
    printf("\n\n");
    
    // double all the values
    sometype_x2(h);
    
    // print the content of the instance
    for (int i = 0; i < n; i++) printf("%d  ",a[i]);
    printf("\n\n");

    // releases the instance
    sometype_free(&h);
    
    return 0;
}
//===========================================

Result:

$ gfortran -c FC_bind_handle_f.f90 && gcc FC_bind_handle_c.c FC_bind_handle_f.o -lgfortran
$ ./a.out
0  1  2  3  4  5  

0  2  4  6  8  10  

However, managing C pointers from Python is maybe not convenient/possible (I don't know enough of python...). Then, instead of returning void* pointer handle, one can return identifiers, which refer to instances in a global array:

! ===========================================
! file: FC_bind_id_f.f90
module somemodule
   use, intrinsic :: iso_c_binding
   implicit none
    
   ! c_float is needed only if one wants to 
   ! access %a directly from C
   type :: sometype
      integer(c_int), allocatable :: a(:)
   end type
    
contains

   ! example routine that processes the content of the type
   subroutine sometype_x2_f(x)
      type(sometype), intent(inout) :: x
      
      x%a = 2 * x%a
   end subroutine
   
end module
! ===========================================

! ===========================================
! file: somemodule_wrap.f90 
module somemodule_wrap
   use, intrinsic :: iso_c_binding
   use somemodule 
   implicit none
   
   ! again, "target" is needed only if one wants to access the %a component from C
   type(sometype), allocatable, target :: h(:)
   logical, allocatable :: active(:)

contains

   ! function that creates a sometype instance,
   ! allocates the %a component,
   ! and returns a C pointer to the instance 
   integer(c_int) function sometype_create(len) bind(c)
      integer(c_int), value :: len
      
      integer :: id
      
      if (.not.allocated(h)) then
         allocate(h(0))
         allocate(active(0))
      end if
      id = findloc(active,.false.,dim=1)
      if (id == 0) then
         h = [h, sometype()]
         active = [active, .false.]
         id = size(h)
      end if
      active(id) = .true.
      allocate(h(id)%a(len))
      sometype_create = id
   end function

   ! routine that releases a handle 
   subroutine sometype_free(id) bind(c)
      integer(c_int) :: id
            
      deallocate(h(id)%a)
      active(id) = .false.
      if (id == size(h)) then
         do while (id > 0 .and. .not.active(id))
            h = h(1:id-1)
            active = active(1:id-1)
            id = id-1
         end do
      end if
      id = 0
   end subroutine
   
   ! function that returns a C pointer to the %a component, 
   ! allowing to work directly on it from C
   type(c_ptr) function sometype_a_ptr(id, n) bind(c)
      integer(c_int), value :: id
      integer(c_int), intent(out) :: n
            
      n = size(h(id)%a)
      sometype_a_ptr = c_loc(h(id)%a)
   end function

   ! wrapper routine to a Fortran routine that performs 
   ! some processing on a sometype instance
   subroutine sometype_x2(id) bind(c)
      integer(c_int), value :: id
        
      call sometype_x2_f(h(id))
   end subroutine
   
end module
! ===========================================

C part:

//===========================================
//file: FC_bind_id_c.c

#include <stdio.h>

int    sometype_create(int len);
void*  sometype_free(int* id);
int*   sometype_a_ptr(int id, int* n);
void*  sometype_x2(int id);

int main(void) 
{
    // creates an instance of sometype with 6 elements in %a
    int id = sometype_create( 6 );
    
    // get a pointer to %a and fill it. Alternatively one could 
    // define a "put" routine to avoid any access to %a from C
    int n;
    int* a = sometype_a_ptr(id, &n);
    for (int i = 0; i < n; i++) {
        a[i] = i;
    }
    
    // print the content of the instance
    for (int i = 0; i < n; i++) printf("%d  ",a[i]);
    printf("\n\n");
    
    // double all the values
    sometype_x2(id);
    
    // print the content of the instance
    for (int i = 0; i < n; i++) printf("%d  ",a[i]);
    printf("\n\n");

    // releases the instance
    sometype_free(&id);
    
    return 0;
}
//===========================================

Answer 2

An extension to PierU's excellent answer to Python (via Cython):

AFAIK there is no way to handle pointer's in Python, but that's not what I want to do anyway.

Add a subroutine to somemodule:

    subroutine sometype_init_f(x, n)
        type(sometype), intent(inout) :: x
        integer(c_int), intent(in) :: n
        integer(c_int) :: i
        x%a = [(i, i=0, n)]
    end subroutine sometype_init_f

and similarly in somemodule_wrap:

    subroutine sometype_init(ptr, n) bind(c, name="sometype_init")
       type(c_ptr), value :: ptr
       integer(c_int), value :: n

       type(sometype), pointer :: p

       call c_f_pointer(ptr,p)
       call sometype_init_f(p, n)
    end subroutine

Then for Cython, we can have

# FC_bind_handle.pyx
cdef extern from "FC_bind_handle_c.h":
    cdef extern void* sometype_create(int len)
    cdef extern void* sometype_free(void** h)
    cdef extern int*  sometype_a_ptr(void* h, int* n)
    cdef extern void* sometype_x2(void* h)
    cdef extern void* sometype_init(void* h, int n)


cdef class SomeTypeWrapper:
    cdef void* _handle

    def __cinit__(self, int len):
        cdef int c_len = len
        self._handle = sometype_create(c_len)
        sometype_init(self._handle, c_len)

    def __dealloc__(self):
        sometype_free(&self._handle)

    def a_ptr(self):
        cdef int n
        cdef int* ptr = sometype_a_ptr(self._handle, &n)
        return [ptr[i] for i in range(n)]

    def x2(self):
        sometype_x2(self._handle)

(following I got a lot of help from this blog post)

# setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy
npy_include_dir = numpy.get_include()

ext_modules = [
    Extension("FC_bind_handle",
        ["FC_bind_handle.pyx"],
        include_dirs = [npy_include_dir],
        libraries=["gfortran"],
        extra_objects=["FC_bind_handle_f.o", "somemodule_wrap.o"])
    ]

setup(name = 'Cython Example',
      cmdclass = {'build_ext': build_ext},
      ext_modules = ext_modules)

Compile this with (I have the two Fortran modules in separate files):

gfortran -fPIC -c FC_bind_handle_f.f90 somemodule_wrap.f90
python3 setup.py build_ext --inplace

Then you can do in Python

import FC_bind_handle
sometype = FC_bind_handle.SomeTypeWrapper(6)
a = sometype.a_ptr()
print(a) # [0, 1, 2, 3, 4, 5, 6]
sometype.x2()
print(a) # unchanged
a = sometype.a_ptr()
print(a) # [0, 2, 4, 6, 8, 10, 12]

This is the closest I could get to handling the a allocatable matrix, and in any case that wasn't what I wanted to do to begin with. I believe to access b, you also need to define getters and setters, which requires even more functions in the wrapper as well as some in the Cython code. I haven't tested it, but found more info here.