Is there an optimized way to convert a numpy array to fortran order when using memmaps

131 Views Asked by At

I have a memmapped numpy array:

arr = np.load("a.npy", mmap_mode='r')

It is bigger than my memory. For my further computation I need it in fortran order instead of C. So I can use np.asfortranarray to convert it and then use np.save to store it in a new file.

However when I do this my memory usage increases proportionally to the input and as such makes me think that an object is being created in memory, I would like it to be fully a, file to file interaction.

How can I convert a.npy into fortran order without having the full object in memory?


For example, I have array:

arr =array([[1, 2],
            [3, 4]])

This is stored on disk as follows by numpy:

fortran_order: False
01 02 03 04

I can do the following transformations:

1

np.asfortranarray(arr)
array([[1, 2],
       [3, 4]])

saved as:

fortran_order: True
01 03 02 04

2

np.asfortranarray(arr.T)
array([[1, 3],
       [2, 4]])

saved as:

fortran_order: True
01 02 03 04

3

arr.T
array([[1, 3],
       [2, 4]])

saved as:

fortran_order: True
01 02 03 04

arr.T only converts the high level accessiblity, I need the on disk ordering to be swapped (this will help with my overall task by keeping the array indexing with C in cache). This can be done by calling np.asfortranarray without arr.T however this incurs a full data copy instead of a transposed view being created and written in the transposed order.

0

There are 0 best solutions below