Versions Compared

Old Version 9

changes.mady.by.user Christopher O'Grady

Saved on

compared with

New Version 10

changes.mady.by.user Christopher O'Grady

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
  • can't create non-root user in docker image because singularity/shifter map the root user to the true username.  creates issues on laptops where mpi doesn't like to be run as root
  • with both Shifter and Singularity you can do mpirun -n 3 <container> to run on multiple nodes but not with Docker
  • psana1 containers are big (8GB for python2-10GB for python3)
  • mounting right directories in the container is non-trivial for non-experts.  singularity helps with this by mounting the user's home dir.
  • some installations of MPI on the host define a variable called OPAL_PREFIX. In some cases this interferes with MPI inside the container, and needs to be unset before launching a container with MPI: unset OPAL_PREFIX && mpirun -n 3 <container>
  • precise version of cuda in the container must match the host
  • major version of mpi in the container must match the host
  • containers at NERSC are patched with NERSC-specific versions, we believe because of the interconnect, although this happens automatically when doing "docker pull" at NERSC
  • we think for singularity each node has to access the container file on startup.  we believe shifter has the ability to broadcast the container more efficiently to all nodes (presumably through a tree-like structure) to improve startup times.  cpo's recollection is that the container-broadcast only worked for one of "mpirun shifter" or "shifter mpirun" (don't remember which).