...
Single slot jobs are not allowed in these queues. You should specify the wallclock runtime using the -W <minutes> or -W <hours:minutes> bsub arguments. There is also a limit on the total number of cores (slots) in use by the bulletmpi and bulletmpi-large queues. You can check the current slot usage and the slot limits by running the blimits command. The output below shows the combined slot total for bulletmpi and bulletmpi-large is limited to 3072 slots. All 3072 slots are in use:
...
There are 2 flavors of mpi available at SLAC, the stock RedHat rpm version, and the RedHat version compiled with LSF hooks. They each use a slightly different bsub command to submit jobs which is described below.
Stock MPI
This version of mpi is the default on most of the public login machines. You can tell that this is the version that you will be running if you get the following response:
renata@rhel6-64f $ which mpirun
/usr/lib64/openmpi/bin/mpirun
A busb command which uses this mpi should look like:
bsub -q bulletmpi -a mympi -n <# cores> <mpi job>
MPI with LSF hooks
OpenMPI environment
We recommend you compile and run MPI jobs on the bullet cluster using the lsf-openmpi module. It is built from the RedHat OpenMPI source but compiled with support for the LSF batch job system. Login to one of the interactive bullet nodes via "ssh bullet", you will be redirected to either bullet0001 or bullet0002. Once logged in, run which mpirun. The command should return this path: This version of mpi is available to bsub when you ssh to bullet which will log you in to either bullet0001 or bullet0002 (as described earlier, submit email to unix-admin to request access to the bulletmpi queue(s)). The response to which mpirun in this case should look like:
/opt/lsf-openmpi/1.5.4/bin//mpirun
You can also check that the version of mpi with LSF hooks is loaded by runninglsf-openmpi is in use:
renata@bullet0002 $ module list
Currently Loaded Modulefiles:
1) lsf-openmpi_1.5.4-x86_64
If you are using lsf-openmpi, Make sure you do not override PATH or LD_LIBRARY_PATH with other OpenMPI directories. An example of a job submission using the lsf-openmpi module: A bsub command which uses this mpi should look like:
bsub -q bulletmpi mpirun -n <# cores> <mpi job>-W <runtime_minutes> mpirun <mpi_executable>