I'm trying to build openmpi from source and use testsuite from mpich package. I have no problem, but subtest linked_list.c may be finished with error (3 fails from 10 runs):
[lemur-8:10278] *** An error occurred in MPI_Compare_and_swap
[lemur-8:10278] *** reported by process [4218159105,1]
[lemur-8:10278] *** on win rdma window 3
[lemur-8:10278] *** MPI_ERR_RMA_RANGE: invalid RMA address range
[lemur-8:10278] *** MPI_ERRORS_ARE_FATAL (processes in this win will now abort,
[lemur-8:10278] *** and potentially your MPI job)
This subtest checks posibility of creating linked list and adding elements by all processes. To prevent races or other errors operations are executed via MPI_Win_lock and MPI_Win_unlock. Also in this subtest dynamic window is used. To see source code of subtest: https://github.com/pmodels/mpich/blob/main/test/mpi/rma/linked_list.c
To find the error out I built OpenMPI with --enable-debug. Cut from log is below:
[lemur-8:10278] cswap: 0xc2deff907e20, 0x21c30, 0xc2deff907e10, MPI_INT, 0, 31849496, rdma window 3
[lemur-8:10278] looking for synchronization object for target 0
[lemur-8:10278] getting remote address for peer 0 target_disp 31849496. peer flags: 0x5
[lemur-8:10278] locating dynamic memory region matching: {1e5fc18, 1e5fc1c} (len 4)
[lemur-8:10278] checking memory region 0x1e5fc18-0x1e5fc1c against 0x1e5b000-0x1e5c000 (index 3) (min_index = 0, max_index = 6)
[lemur-8:10277] dynamic region 0: {0x1e0c000, 4096}
[lemur-8:10277] dynamic region 1: {0x1e57000, 4096}
[lemur-8:10277] dynamic region 2: {0x1e59000, 4096}
[lemur-8:10277] dynamic region 3: {0x1e5b000, 4096}
[lemur-8:10277] dynamic region 4: {0x1e5c000, 4096}
[lemur-8:10277] dynamic region 5: {0x1e5d000, 4096}
[lemur-8:10277] dynamic region 6: {0x1e5e000, 4096}
[lemur-8:10277] dynamic region 7: {0x1e5f000, 4096}
[lemur-8:10277] releasing exclusive lock 45dea53cf158 on peer 0
[lemur-8:10277] exclusive lock released
[lemur-8:10277] attach complete
[lemur-8:10277] lock: 1, 0, 0, rdma window 3
[lemur-8:10278] checking memory region 0x1e5fc18-0x1e5fc1c against 0x1e5d000-0x1e5e000 (index 5) (min_index = 4, max_index = 6)
[lemur-8:10278] checking memory region 0x1e5fc18-0x1e5fc1c against 0x1e5e000-0x1e5f000 (index 6) (min_index = 6, max_index = 6)
[lemur-8:10278] could not retrieve region for 1e5fc18 from window rank 0
Please, look at the last line. "Could not retrieve region for 1e5fc18 from window rank 0". I really dont understand, why the region had not been found.
Compiling subtest:
mpicc rma/linked_list.c -o linked_list -w -I rma -I include -L /usr/lib -lmpi -lpthread util/mtest.c
Command I use to run subtest:
mpirun --allow-run-as-root -np 4 linked_list
But if I use this command, no fail more:
mpirun --allow-run-as-root --mca osc '^rdma' -np 4 linked_list
There are https://github.com/open-mpi/ompi/issues/9580 and https://github.com/open-mpi/ompi/issues/10328, but these issues are not about my problem, as I understand.
Can you please explain, why region may be not retrieved? Also I coundn't find information about mpich RDMA support. Maybe this subtest is not valid for OpenMPI?
I found this one https://www.mpich.org/static/docs/v3.3/www3/MPI_Win_lock.html.
Implementations may restrict the use of RMA communication that is synchronized by lock calls to windows in memory allocated by MPI_Alloc_mem. Locks can be used portably only in such memory.