Versions Compared

Old Version 7

changes.mady.by.user Christopher O'Grady

Saved on

compared with

New Version 8

changes.mady.by.user Christopher O'Grady

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
  • can't create non-root user in docker image because singularity/shifter map the root user to the true username.  creates issues on laptops where mpi doesn't like to be run as root
  • with both Shifter and Singularity you can do mpirun -n 3 <container> to run on multiple nodes but not with Docker
  • psana1 containers are big (8GB for python2-10GB for python3)
  • some installations of MPI on the host define a variable called OPAL_PREFIX. In some cases this interferes with MPI inside the container, and needs to be unset before launching a container with MPI: unset OPAL_PREFIX && mpirun -n 3 <container>
  • precise version of cuda in the container must match the host
  • major version of mpi in the container must match the host
  • containers at NERSC are patched with NERSC-specific versions, we believe because of the interconnect, although this happens automatically when doing "docker pull" at NERSC
  • we think for singularity each node has to access the container file on startup.  we believe shifter had has the ability to broadcast the container more efficiently to all nodes (presumably through a tree-like structure) to improve startup times.  cpo's recollection is that this depended on whether one did the container-broadcast only worked for one of "mpirun shifter" or "shifter mpirun" (don't remember which).