Page History

...

Depending on your data access you may need to submit jobs to a specific farm. This is accomplished by submitting to the appropriate LSF batch queue. Refer to the table below. Jobs for the running current experiment should be submitted to the high priority queues psnehq and psfehq . Multi-core OpenMPI jobs should be run in either the psnehmpiq or psfehmpiq batch queue, see the following section on "Submitting OpenMPI Batch Jobs"running against the Fast Feedback storage layer (FFB). Simulation jobs should be submitted to the low priority queues psfehidle and psfehidle(with idle in the name). CPU intensive jobs, which don't demand high data throughout, should be submitted to the psanacsq queue.

Submitting Batch Jobs

LSF (Load Sharing Facility) is the job scheduler used at SLAC to execute user batch jobs on the various batch farms. LSF commands can be run from a number of SLAC servers, but best to use psexport or psloginthe interactive psana farm. Login first to pslogin (from SLAC) or to psexport (from anywhere) and then to psana. From there you can submit a job with the following command:

...

The RedHat supplied OpenMPI packages are installed on pslogin, psexport and all of the psana batch servers. The system default has been set to the current version as supplied by RedHat.

$ mpi-selector --query
default:openmpi-1.4-gcc-x86_64
level:system

Your environment should be set up to use this version (unless you have used RedHat's mpi-selector script, or your login scripts, to override the default). You can check to see if your PATH is correct by issuing the commandcommand which mpirun. Currently, this should return /usr/lib64/openmpi/1.4-gcc/bin/mpirun. Future updates to the MPI version may change the exact details of this path.

...

The following are examples of how to submit OpenMPI jobs to the PCDS psnehmpiq psanaq batch queue:

bsub -q psnehmpiq psanaq-a mympi -n 32 -o ~/output/%J.out ~/bin/hello

Will submit an OpenMPI job (-a mympi) requesting 32 processors (-n 32) to the psnehmpiq psanaq batch queue (-q psnehmpiqpsanaq).

bsub -q psfehmpiqpsanaq -a mympi -n 16 -R "span[ptile=1]" -o ~/output/%J.out ~/bin/hello

Will submit an OpenMPI job (-a mympi) requesting 16 processors (-n 16) spanned as one processor per host (-R "span[ptile=1]") to the psfehmpiq psanaq batch queue (-q psfehmpiqpsanaq).

bsub -q psfehmpiqpsanaq -a mympi -n 12 -R "span[hosts=1]" -o ~/output/%J.out ~/bin/hello

Will submit an OpenMPI job (-a mympi) requesting 12 processors (-n 12) spanned all on one host (-R "span[hosts=1]") to the psfehmpiq psanaq batch queue (-q psfehmpiqpsanaq).

Common LSF Commands

Report status of all jobs (running, pending, finished, etc) submitted by the current user:

...

Report running or pending jobs for all users in the psnehq psanaq queue:

bjobs -w -u all -q psnehq

...

Jobs submitted to a high priority queue (eh psnehq) will automatically suspend jobs running on the same hardware, but in the lower priority queues (eg pnehmpiq and psnehidle), the ones with idle in the name. This suspension can take from a few seconds to up to a few minutes and when many small high priority jobs are submitted in rapid succession, the time taken to suspend lower priority jobs affects the performance of the high priority jobs. For this reason there is a mechanism to suspend the lower priority queues for any specified high priority queue. The command, which can be issued by LCLS scientists and PCDS people, is:

...

/reg/g/psdm/qcntrl/psniceq psnehq 30     # Suspend idle queues conflicting with psnehq for 30 minutes
/reg/g/psdm/qcntrl/psniceq psfehq 40     # Suspend idle queues conflicting with psfehq for 40 minutes
/reg/g/psdm/qcntrl/psniceq psanacspsanacsq 8h    # Suspend idle queues conflicting with psanacs for 8 hours