Page History

...

The batch farm is made of sixty forty eight servers with the following general specifications:

• 8 24 cores, Xeon E5520X5675, 24GB memory, 500GB disk, 1Gb/s

Running OpenMPI Jobs in the PCDS Batch Farm

The RedHat supplied OpenMPI packages are installed on psdev, pslogin, psimport, psexport and all of the psana batch servers.

The system default has been set to the current version as supplied by RedHat.

$ mpi-selector --query
default:openmpi-1.4-gcc-x86_64
level:system

Your environment should be set up to use this version (unless you have used RedHat's mpi-selector script, or your login scripts, to override the default). You can check to see if your PATH is correct by issuing the command  which mpirun . Currently, this should return: /usr/lib64/openmpi/1.4-gcc/bin/mpirun  Future updates to the MPI version may change the exact details of this path. 

In addition, your LD_LIBRARY_PATH should include: /usr/lib64/openmpi/1.4-gcc/lib  (or something similar).

For notes on compiling examples; please see:

http://www.slac.stanford.edu/comp/unix/farm/mpi.html 

The following are examples of how to submit OpenMPI jobs to the PCDS batch queues:

bsub -q lclsq -a mympi -n 32 -o ~/output/%J.out ~/bin/hello

Will submit an OpenMPI job (-a mympi) requesting 32 processors (-n 32) to the lclsq batch queue (-q lclsq).

bsub -q psfehq -a mympi -n 18 -R "span[ptile=1]" -o ~/output/%J.out ~/bin/hello

Will submit an OpenMPI job (-a mympi) requesting 18 processors (-n 18) spanned as one processor per host (-R "span\[ptile=1\]") to the psfehq batch queue (-q psfehq).

Data Storage

LCLS provides space for all your experiment's data at no cost for you. This includes the measurements as well as derived data from your analysis software.

...