There are two options to get MPI working at Nersc using Shifter. 

  1. Manual installation of MPICH followed by pip installation (with no binary option) of mpi4py
    This assumes that you install both MPICH and mpi4py in the conda environment that will be used to install psana. Choosing this way will NOT require you to uninstall MPICH (–force flag) at the end of docker build script.
  2. Use conda install (from conda-forge)
    conda create --name lcls2_base mpich=3.3.2 psana (or psana-conda for lcls1)
    Choosing this way, you'll need to uninstall mpich at the end of the docker script.

We are currently investigating why (1) works but (2) does not. 

The correct installation will show ldd of mpiXX.so as:

(lcls2_base) monarin@nid02517:~$ ldd /img/conda.local/miniconda3/envs/lcls2_base/lib/python3.7/site-packages/mpi4py/MPI.cpython-37m-x86_64-linux-gnu.so 

 linux-vdso.so.1 (0x00002aaaaaad5000)

 libmpi.so.12 => /opt/udiImage/modules/mpich/lib64/libmpi.so.12 (0x00002aaaaacd5000)

 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00002aaaab297000)

 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00002aaaab4b6000)

 libxpmem.so.0 => /opt/udiImage/modules/mpich/lib64/dep/libxpmem.so.0 (0x00002aaaab8a7000)

 librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00002aaaabaab000)

 libugni.so.0 => /opt/udiImage/modules/mpich/lib64/dep/libugni.so.0 (0x00002aaaabcb3000)

 libudreg.so.0 => /opt/udiImage/modules/mpich/lib64/dep/libudreg.so.0 (0x00002aaaabf39000)

 libpmi.so.0 => /opt/udiImage/modules/mpich/lib64/dep/libpmi.so.0 (0x00002aaaac144000)

 libgfortran.so.5 => /opt/udiImage/modules/mpich/lib64/dep/libgfortran.so.5 (0x00002aaaac390000)

 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00002aaaac808000)

 libgcc_s.so.1 => /opt/udiImage/modules/mpich/lib64/dep/libgcc_s.so.1 (0x00002aaaacba6000)

 libquadmath.so.0 => /opt/udiImage/modules/mpich/lib64/dep/libquadmath.so.0 (0x00002aaaacdbf000)

 /lib64/ld-linux-x86-64.so.2 (0x00002aaaaaaab000)

 libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00002aaaad000000)

 libz.so.1 => /opt/udiImage/modules/mpich/lib64/dep/libz.so.1 (0x00002aaaad204000)



Note that after pulling a shifter image, the LD_LIBRARY_PATH gets set to:

(lcls2_base) monarin@nid02517:~$ echo $LD_LIBRARY_PATH

/opt/udiImage/modules/mpich/lib64




 

  • No labels