• can't create non-root user in docker image because singularity/shifter map the root user to the true username.  creates issues on laptops where mpi doesn't like to be run as root
  • psana1 containers are big (8GB for python2-10GB for python3) (note by valerio: we are now in the range 1.6-3.2GB)
  • the following features are non-trivial for non experts when using docker: graphical applications, running MPI on multiple nodes with  mpirun -n 3 <container> , hardware acceleration and CUDA. singularity helps with all these. additionally mounting right directories in the container is also non-trivial when using docker.  singularity helps with this by mounting the user's home dir and a set of dirs defined by the administrators.
  • singularity requires vagrant on macos and windows (https://sylabs.io/guides/3.7/admin-guide/installation.html#installation-on-windows-or-mac, but the instructions are a bit outdated, and one should use VM=sylabs/singularity-3.7-ubuntu-bionic64)
  • some installations of MPI on the host define a variable called OPAL_PREFIX. In some cases this interferes with MPI inside the container, and needs to be unset before launching a container with MPI: unset OPAL_PREFIX && mpirun -n 3 <container> (note by valerio: this seems fixed now)
  • precise version of cuda in the container must match the host (note by valerio: CUDA seems forward compatible. For example 10.2 will run where 11 is supported)
  • major version of mpi in the container must match the host
  • containers at NERSC are patched with NERSC-specific versions, we believe because of the interconnect, although this happens automatically when doing "docker pull" at NERSC
  • we think for singularity each node has to access the container file on startup.  we believe shifter has the ability to broadcast the container more efficiently to all nodes (presumably through a tree-like structure) to improve startup times.  cpo's recollection is that the container-broadcast only worked for one of "mpirun shifter" or "shifter mpirun" (don't remember which).
  • need to do some work (we think minor) to integrate with Jupyter.
  • No labels