Versions Compared

Old Version 7

changes.mady.by.user Christopher O'Grady

Saved on

compared with

New Version Current

changes.mady.by.user Valerio Mariani

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
  • can't create non-root user in docker image because singularity/shifter map the root user to the true username.  creates issues on laptops where mpi doesn't like to be run as root
  • with both Shifter and Singularity you can do mpirun -n 3 <container> to run on multiple nodes but not with Docker
  • psana1 containers are big (8GB for python2-10GB for python3) (note by valerio: we are now in the range 1.6-3.2GB)
  • the following features are non-trivial for non experts when using docker: graphical applications, running MPI on multiple nodes with  mpirun -n 3 <container> , hardware acceleration and CUDA. singularity helps with all these. additionally mounting right directories in the container is also non-trivial when using docker.  singularity helps with this by mounting the user's home dir and a set of dirs defined by the administrators.
  • singularity requires vagrant on macos and windows (https://sylabs.io/guides/3.7/admin-guide/installation.html#installation-on-windows-or-mac, but the instructions are a bit outdated, and one should use VM=sylabs/singularity-3.7-ubuntu-bionic64)
  • some installations of MPI on the host define a variable called OPAL_PREFIX. In some cases this interferes with MPI inside the container, and needs to be unset before launching a container with MPI: unset OPAL_PREFIX && mpirun -n 3 <container> (note by valerio: this seems fixed now)
  • precise version of cuda in the container must match the host (note by valerio: CUDA seems forward compatible. For example 10.2 will run where 11 is supported)
  • major version of mpi in the container must match the host
  • containers at NERSC are patched with NERSC-specific versions, we believe because of the interconnect, although this happens automatically when doing "docker pull" at NERSC
  • we think for singularity each node has to access the container file on startup.  we believe shifter had has the ability to broadcast the container more efficiently to all nodes (presumably through a tree-like structure) to improve startup times.  cpo's recollection is that this depended on whether one did the container-broadcast only worked for one of "mpirun shifter" or "shifter mpirun" (don't remember which).
  • need to do some work (we think minor) to integrate with Jupyter.