Page History

...

source /reg/g/psdm/etc/ana_env.csh    (for tcsh/csh) OR...
source /reg/g/psdm/etc/ana_env.sh     (for bash)

The following are examples of how to submit OpenMPI jobs to the PCDS psanaq batch queue:Starting with ana-0.18.0 (release April 28 2016), the recommended way to submit MPI batch jobs is

bsub -q psanaq -a mympi -n 32 -o ~/output/%J.out mpirun ~/bin/hello

Will this will submit an OpenMPI job (-a mympi) job  requesting 32 processors (-n 32) to the psanaq batch queue (-q psanaq). The environment variable SIT_RELEASE shows you the ana release number.

Prior to ana-0.18.0, one submits as follows:

bsub -q psanaq -a mympi -n 32 -o ~/output/%J.out ~/bin/hello

Usining -a mympi works prior to ana 0.18 as well. 

bsub -q psanaq -n 16 -R "span[ptile=1]" -o ~/output/%J.out mpirun ~/bin/hello       # for ana-0.18 and greater
bsub -q psanaq -a mympi -n 16 -R "span[ptile=1]" -o ~/output/%J.out ~/bin/hello     # prior to ana-0.18

Will submit an OpenMPI job (-a mympi) requesting 16 processors (-n 16) spanned as one processor per host (-R "span[ptile=1]") to the psanaq batch queue (-q psanaq).

bsub -q psanaq -n 12 -R "span[hosts=1]" -o ~/output/%J.out mpirun ~/bin/hello
bsub -q psanaq -a mympi -n 12 -R "span[hosts=1]" -o ~/output/%J.out ~/bin/hello

Will submit an OpenMPI job (-a mympi) requesting 12 processors (-n 12) spanned all on one host (-R "span[hosts=1]") to the psanaq batch queue (-q psanaq).

bsub "psana1503 psana1509" -q psnehq -n 12 -o ~/output/%J.out mpirun ~/bin/hello
bsub "psana1503 psana1509" -q psnehq -a mympi -n 12 -o ~/output/%J.out ~/bin/hello

Will submit an OpenMPI job (-a mympi) requesting 12 processors (-n 12) on two nodes (psana1503 and psana1509) to the psnehq batch queue (-q psnehq).

...