Python: numpy memmap in parallel

644 Views Asked by At

Does it make sense to use numpy's memmap across multiple cores (MPI)?

I have a file on disk.

Can I create a separate memmap object on each core, and use it to read different slices from the file?

What about writing to it?

1

There are 1 best solutions below

0
On BEST ANSWER

Q : "Does it make sense to use numpy's memmap across multiple cores (MPI)?"

Yes ( ... even without MPI, using just Python native { thread- | process-}-based forms of concurrent-processing )

Q : "Can I create a separate memmap-object on each core, and use it to read different slices from the file?"

Yes.

Q : "What about writing to it?"

The same ( sure, if having been opened in write-able mode ... )