Shared memory with mpi

Question

I am trying to have a shared memory block using multiprocessing to contain a numpy array accessible to processes created using mpi4py. A basic tutorial is here. I made an simple code to implement it in the following script and run it from the terminal with mpirun -np 2 python myscript.py.

import numpy as np
from multiprocessing import shared_memory
from mpi4py import MPI

comm = MPI.COMM_WORLD
pid = comm.Get_rank()
npros = comm.Get_size()

npts = 10
dtyp = np.int64

if pid == 0:
    arr = np.arange(npts, dtype=dtyp)
    shm = shared_memory.SharedMemory(create=True, size=arr.nbytes, name='trial')
    brr = np.ndarray(arr.shape, dtype=arr.dtype, buffer=shm.buf)
    brr[:] = arr[:]
    del arr

comm.barrier()

if pid == 1:
    existing_shm = shared_memory.SharedMemory(name='trial')
    crr = np.ndarray((npts), dtype=dtyp, buffer=existing_shm.buf)
    print(crr)

comm.barrier()

if pid == 1:
    del crr
    existing_shm.close()

comm.barrier()

if pid == 0:
    del brr
    shm.close()
    shm.unlink()

However, it throws back some warning

/home/peedaruos/miniconda/envs/py39/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/peedaruos/miniconda/envs/py39/lib/python3.9/multiprocessing/resource_tracker.py:229: UserWarning: resource_tracker: '/trial': [Errno 2] No such file or directory: '/trial' warnings.warn('resource_tracker: %r: %s' % (name, e))

Not sure if there is a problem or could get serious down the line..

Answer 1

You're mixing to different modes of parallel programming.

1. You start this program as MPI, meaning that two python processes are running. One process then creates a multiprocessing.shared_memory buffer. And then there is no way for the other process to find that buffer.

2. Normally you'd start one python process, it creates a shared buffer, adn then you spawn multiple multiprocessing processes that read that buffer. Obviously they can see the buffer because they get it from their parent. In your scenario there is no parent.