- can't create non-root user in docker image because singularity/shifter map the root user to the true username. creates issues on laptops where mpi doesn't like to be run as root
- with both Shifter and Singularity you can do
mpirun -n 3 <container>
to run on multiple nodes but not with Docker - psana1 containers are big (8GB for python2-10GB for python3)
- mounting right directories in the container is non-trivial for non-experts. singularity helps with this by mounting the user's home dir.
- some installations of MPI on the host define a variable called OPAL_PREFIX. In some cases this interferes with MPI inside the container, and needs to be unset before launching a container with MPI:
unset OPAL_PREFIX && mpirun -n 3 <container>
- precise version of cuda in the container must match the host
- major version of mpi in the container must match the host
- containers at NERSC are patched with NERSC-specific versions, we believe because of the interconnect, although this happens automatically when doing "docker pull" at NERSC
- we think for singularity each node has to access the container file on startup. we believe shifter has the ability to broadcast the container more efficiently to all nodes (presumably through a tree-like structure) to improve startup times. cpo's recollection is that the container-broadcast only worked for one of "mpirun shifter" or "shifter mpirun" (don't remember which).
Overview
Content Tools