SO: Ubuntu 17.04. Until yesterday everything was fine with MPI, then something went wrong. The error is the one on the the title and it comes out running with mpirun -n[n] ./myprogram.
Before Opal the error was with Libmpi.so.12 and I couldn't resolve it despite I tried indicating the LD_LIBRARY_PATH on my .bashrc, as suggested in similar questions here. Then reinstalling OpenMPI the error has changed in this new one, and I don't know how to fix it. Actually the result of apt-file search libmpi.so is:
lam4-dev: /usr/lib/lam/lib/libmpi.so
libopenmpi-dev: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so
libopenmpi2: /usr/lib/x86_64-linux-gnu/libmpi.so.20
libopenmpi2: /usr/lib/x86_64-linux-gnu/libmpi.so.20.0.2
libopenmpi2: /usr/lib/x86_64-linux-gnu/openmpi/lib/libmpi.so.20.0.2
And ldd returns:
linux-vdso.so.1 => (0x00007ffc415e5000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fe717956000)
libmpi.so.12 => not found
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fe717738000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe717371000)
/lib64/ld-linux-x86-64.so.2 (0x000055ccd14f4000)
So the problem still seems to be libmpi.so.12 ? Before reinstalling OpenMPI, libmpi.so.12 was there, MPI couldn't find it, but was there.
[edit]
which mpirun returns:
/usr/bin/mpirun
And apt-file search mpicc returns:
lam4-dev: /usr/bin/mpicc.lam
lam4-dev: /usr/share/man/man1/mpicc.lam.1.gz
libmpich-dev: /usr/bin/mpicc.mpich
libmpich-dev: /usr/share/man/man1/mpicc.mpich.1.gz
libopenmpi-dev: /usr/bin/mpicc.openmpi
libopenmpi-dev: /usr/share/man/man1/mpicc.openmpi.1.gz
libsimgrid-dev: /usr/bin/smpicc
libsimgrid-dev: /usr/share/man/man1/smpicc.1.gz
mpich-doc: /usr/share/doc/mpich-doc/www1/mpicc.html
openmpi-common: /usr/share/openmpi/mpicc-wrapper-data.txt
openmpi-common: /usr/share/openmpi/mpicc.openmpi-wrapper-data.txt
it seems your application was built with an other MPI version (an older Open MPI or an other MPI library such as lam or mpich)
The easiest path is to rebuild your application with the freshly installed Open MPI
[EDIT]
this was diagnosed in the chat, and the root cause was system and user built Open MPI were inadvertently mixed.
once the environment was fixed so only the system Open MPI is used, the issue was resolved.