I have a memmapped numpy array:
arr = np.load("a.npy", mmap_mode='r')
It is bigger than my memory. For my further computation I need it in fortran order instead of C. So I can use np.asfortranarray
to convert it and then use np.save
to store it in a new file.
However when I do this my memory usage increases proportionally to the input and as such makes me think that an object is being created in memory, I would like it to be fully a, file to file interaction.
How can I convert a.npy
into fortran order without having the full object in memory?
For example, I have array:
arr =array([[1, 2],
[3, 4]])
This is stored on disk as follows by numpy:
fortran_order: False
01 02 03 04
I can do the following transformations:
1
np.asfortranarray(arr)
array([[1, 2],
[3, 4]])
saved as:
fortran_order: True
01 03 02 04
2
np.asfortranarray(arr.T)
array([[1, 3],
[2, 4]])
saved as:
fortran_order: True
01 02 03 04
3
arr.T
array([[1, 3],
[2, 4]])
saved as:
fortran_order: True
01 02 03 04
arr.T
only converts the high level accessiblity, I need the on disk ordering to be swapped (this will help with my overall task by keeping the array indexing with C in cache). This can be done by calling np.asfortranarray
without arr.T
however this incurs a full data copy instead of a transposed view being created and written in the transposed order.