Slurm is a batch scheduler that enables users (you!) to submit long (or even short) compute 'jobs' to our compute clusters. It will queue up jobs such that the (limited) resources compute resources available are fairly shared and distributed for all users. This page describes basic usage of slurm at SLAC. It will provide some simple examples of how to request common resources.
Slurm is currently being tested and is scheduled for deployment on the SLAC Scientific Shared Data Facility. We welcome any suggestions and issues to be reported to unix-admin@slac.stanford.edu. Note that whilst we strive to keep the information on these pages up-to-date, there may be inconsistencies and/or incorrect information contained within.
You will need to request access to slurm whilst we work out more automated ways of adding users to the system. We will need to know which slurm Account to 'bill' you against (don't worry, there will be no $ charge for usage, it's purely for accounting and reporting). This Account will most likely be your immediate group/team that you work with. Please send your unix username and your group/team name to unix-admin@slac.stanford.edu.
We are also testing the ability for your group/team Slurm Administrator to have the ability to add users to their Accounts (delegated administration). If you wish to represent your group/team to do this, please contact us!
Why should I use Batch?
Whilst your desktop computer and or laptop computer has a fast processor and quick local access to data stored on its hard disk/ssd; you may want to run either very big and/or very large compute tasks that may require a lot of CPUs, GPUs, memory, or a lot of data. Our compute servers that are part of the Batch system allows your to do this. Our servers typically also have very fast access to centralised storage, have (some) common software already preinstalled, and will enable you to run these long tasks without impacting your local desktop/laptop resources.
Why should I use Slurm?
Historically, we have always use IBM's LSF as our Batch scheduler software. However, with new hardware such as GPU's, we have found that the user experience and the administrative accounting features of LSF to be lacking. Slurm is also commonly used across academic and laboratory environments and we hope that this commonality will facilitate easy usage for you, and simpler administration for us.
What should I know about using Batch?
The first thing to note is that you should probably be comfortable in a Unix 'command line' environment. LINKS?
When you submit a compute task to the batch system, this is called a Job. We need to charge each Job to an Account. You may also select what pool of servers to run the Jobs on - this is known as a Partition.
You should also acquaint yourself with slurm Accounts and Partitions.
What is a Slurm Account?
As the number of servers and GPUs in our environment is limited (but not small), we need to keep account of who uses what. In addition, as groups/teams can purchase their own servers to be added to the SDF we must provide a method of which allocated users can have priority access to the servers that were purchased for them. A slurm Account is basically something that you will charge your job against.
What is a Slurm Partition?
A Partition is a logical grouping of compute servers. These may be servers of a similar technical specification (eg Cascade Lake CPUs, Telsa GPUs etc), or by ownership of the servers - eg SUNCAT group may have purchased so many servers, so we put them all into a Partition.
Generally, all servers will be placed in the shared partition that everyone with a slac computer account will have access to (although at a low priority).
Users should contact their Coordinators to be added to appropriate group Partitions to get priority access to resources.
You can view the active Partitions on SDF with
$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST shared* up 7-00:00:00 21 unk* cryoem-gpu[02,04-09,11-15],ml-gpu[02-10] shared* up 7-00:00:00 10 idle cryoem-gpu[01,03,10,50],hep-gpu01,ml-gpu[01,11],nu-gpu[01-03] ml up infinite 9 unk* ml-gpu[02-10] ml up infinite 2 idle ml-gpu[01,11] neutrino up infinite 3 idle nu-gpu[01-03] cryoem up infinite 12 unk* cryoem-gpu[02,04-09,11-15] cryoem up infinite 4 idle cryoem-gpu[01,03,10,50]
What is a Slurm Allocation?
In order to provide appropriate access for users to the hardware, an Allocation is created that defines what User can run on what Partition and charge against what Account (there's a bit more in the backend to this).
How do I use Slurm?
We are still testing the best way to deploy Slurm at SLAC, and as such, some of the examples and instructions that follow may be subject to change. If you have any opinions and or suggestions, we would love to hear from.
Slurm is installed on a limited number of hosts currently. We recommend logging on using ssh via a terminal:
ssh ocio-gpu01.slac.stanford.edu
In order to get the slurm binaries available, you will need to use modules to add the slurm binaries into your path environment:
module load slurm
We will likely have the above command automatically run, so it may not be necessary later.
Common commands are:
| srun | request a quick job to be ran - eg an interactive terminal |
| sbatch | submit a batch job to run |
| squeue | show jobs |
| scancel | cancel a job |
| scontrol show job | show job details |
| sstat | show job usage details |
| sacctmgr | manage Associations |
How can I get an Interactive Terminal?
use the srun command
module load slurm srun -A shared -p shared -n 1 --pty /bin/bash
This will then execute /bin/bash on a (scheduled) server in the Partition shared and charge against Account shared. This will request a single CPU, launch a pseudo terminal (pty) where bash will run. You may be provided different Accounts and Partitions and should use them when possible.
Note that when you 'exit' the interactive session, it will relinquish the resources for someone else to use. This also means that if your terminal is disconnected (you turn your laptop off, loose network etc), then the Job will also terminate (similar to ssh).
If your interactive request doesn't immediately find resources, it will currently not actually return you a pty - even though the job actually does run. This results in what looks like a hanging process. We are investigating... salloc first?
How do I submit a Batch Job?
use the sbatch command, this primer needs to be elaborated:
Create a job submission script (text file) script.sh:
#!/bin/bash #SBATCH --account=myaccount #SBATCH --partition=shared #SBATCH --qos=scavenger # #SBATCH --job-name=test #SBATCH --output=output-%j.txt #SBATCH --error=output-%j.txt # #SBATCH --ntasks=1 #SBATCH --mem-per-cpu=100 # #SBATCH --time=10:00 # #SBATCH --gpu geforce_gtx_1080_ti:1 <commands here>
you will need an account (see below). All SLAC users have access to the "shared" partition with a quality of service of "scavenger". This is so that stakeholders of machines in the SDF will get priority access to their resources, whilst any user can use all resources as long as the 'owners' of the hardware isn't wanting to use it. As such, owners (or stakeholders) will have qos "normal" access to their partitions (of which such hosts are also within the shared partition).
Then, in order to submit the job:
module load slurm sbatch script.sh
You can then use the command to monitor your job progress:
squeue
And you can cancel the job with
scancel <jobid>
How can I request GPUs?
You can use the --gpus to specify gpus for your jobs: Using a number will request the number of any gpu that is available (what you get depends upon what your Account/Association is and what is available when you request it). You can also specify the type of gpus by prefixing the number with the model name. eg
# request single gpu srun -A shared -p shared -n 1 --gpus 1 --pty /bin/bash # request a gtx 1080 gpu srun -A shared -p shared -n 1 --gpus geforce_gtx_1080_ti:1 --pty /bin/bash # request a gtx 2080 gpu srun -A shared -p shared -n 1 --gpus geforce_rtx_2080_ti:1 --pty /bin/bash # request a v100 gpu srun -A shared -p shared -n 1 --gpus v100:1 --pty /bin/bash
How can I see what GPUs are available?
# sinfo -o "%12P %5D %14F %7z %7m %10d %11l %42G %38N %f" PARTITION NODES NODES(A/I/O/T) S:C:T MEMORY TMP_DISK TIMELIMIT GRES NODELIST AVAIL_FEATURES shared* 1 0/1/0/1 2:8:2 191567 0 7-00:00:00 gpu:v100:4 nu-gpu02 CPU_GEN:SKX,CPU_SKU:4110,CPU_FRQ:2.10GHz,GPU_GEN:VLT,GPU_SKU:V100,GPU_MEM:32GB,GPU_CC:7.0 shared* 8 0/1/7/8 2:12:2 257336 0 7-00:00:00 gpu:geforce_gtx_1080_ti:10 cryoem-gpu[02-09] CPU_GEN:HSW,CPU_SKU:E5-2670v3,CPU_FRQ:2.30GHz,GPU_GEN:PSC,GPU_SKU:GTX1080TI,GPU_MEM:11GB,GPU_CC:6.1 shared* 14 0/0/14/14 2:12:2 191552 0 7-00:00:00 gpu:geforce_rtx_2080_ti:10 cryoem-gpu[11-15],ml-gpu[02-10] CPU_GEN:SKX,CPU_SKU:5118,CPU_FRQ:2.30GHz,GPU_GEN:TUR,GPU_SKU:RTX2080TI,GPU_MEM:11GB,GPU_CC:7.5 shared* 1 0/1/0/1 2:12:2 257336 0 7-00:00:00 gpu:geforce_gtx_1080_ti:10(S:0) cryoem-gpu01 CPU_GEN:HSW,CPU_SKU:E5-2670v3,CPU_FRQ:2.30GHz,GPU_GEN:PSC,GPU_SKU:GTX1080TI,GPU_MEM:11GB,GPU_CC:6.1 shared* 3 0/3/0/3 2:12:2 191552 0 7-00:00:00 gpu:geforce_rtx_2080_ti:10(S:0) cryoem-gpu10,ml-gpu[01,11] CPU_GEN:SKX,CPU_SKU:5118,CPU_FRQ:2.30GHz,GPU_GEN:TUR,GPU_SKU:RTX2080TI,GPU_MEM:11GB,GPU_CC:7.5 shared* 3 0/3/0/3 2:8:2 191567 0 7-00:00:00 gpu:v100:4(S:0-1) cryoem-gpu50,nu-gpu[01,03] CPU_GEN:SKX,CPU_SKU:4110,CPU_FRQ:2.10GHz,GPU_GEN:VLT,GPU_SKU:V100,GPU_MEM:32GB,GPU_CC:7.0 shared* 1 0/1/0/1 2:12:2 257330 0 7-00:00:00 gpu:geforce_gtx_1080_ti:8(S:0),gpu:titan_x hep-gpu01 CPU_GEN:HSW,CPU_SKU:E5-2670v3,CPU_FRQ:2.30GHz,GPU_GEN:PSC,GPU_SKU:GTX1080TI,GPU_MEM:11GB,GPU_CC:6.1 ml 9 0/0/9/9 2:12:2 191552 0 infinite gpu:geforce_rtx_2080_ti:10 ml-gpu[02-10] CPU_GEN:SKX,CPU_SKU:5118,CPU_FRQ:2.30GHz,GPU_GEN:TUR,GPU_SKU:RTX2080TI,GPU_MEM:11GB,GPU_CC:7.5 ml 2 0/2/0/2 2:12:2 191552 0 infinite gpu:geforce_rtx_2080_ti:10(S:0) ml-gpu[01,11] CPU_GEN:SKX,CPU_SKU:5118,CPU_FRQ:2.30GHz,GPU_GEN:TUR,GPU_SKU:RTX2080TI,GPU_MEM:11GB,GPU_CC:7.5 neutrino 1 0/1/0/1 2:8:2 191567 0 infinite gpu:v100:4 nu-gpu02 CPU_GEN:SKX,CPU_SKU:4110,CPU_FRQ:2.10GHz,GPU_GEN:VLT,GPU_SKU:V100,GPU_MEM:32GB,GPU_CC:7.0 neutrino 2 0/2/0/2 2:8:2 191567 0 infinite gpu:v100:4(S:0-1) nu-gpu[01,03] CPU_GEN:SKX,CPU_SKU:4110,CPU_FRQ:2.10GHz,GPU_GEN:VLT,GPU_SKU:V100,GPU_MEM:32GB,GPU_CC:7.0 cryoem 8 0/1/7/8 2:12:2 257336 0 infinite gpu:geforce_gtx_1080_ti:10 cryoem-gpu[02-09] CPU_GEN:HSW,CPU_SKU:E5-2670v3,CPU_FRQ:2.30GHz,GPU_GEN:PSC,GPU_SKU:GTX1080TI,GPU_MEM:11GB,GPU_CC:6.1 cryoem 5 0/0/5/5 2:12:2 191552 0 infinite gpu:geforce_rtx_2080_ti:10 cryoem-gpu[11-15] CPU_GEN:SKX,CPU_SKU:5118,CPU_FRQ:2.30GHz,GPU_GEN:TUR,GPU_SKU:RTX2080TI,GPU_MEM:11GB,GPU_CC:7.5 cryoem 1 0/1/0/1 2:12:2 257336 0 infinite gpu:geforce_gtx_1080_ti:10(S:0) cryoem-gpu01 CPU_GEN:HSW,CPU_SKU:E5-2670v3,CPU_FRQ:2.30GHz,GPU_GEN:PSC,GPU_SKU:GTX1080TI,GPU_MEM:11GB,GPU_CC:6.1 cryoem 1 0/1/0/1 2:12:2 191552 0 infinite gpu:geforce_rtx_2080_ti:10(S:0) cryoem-gpu10 CPU_GEN:SKX,CPU_SKU:5118,CPU_FRQ:2.30GHz,GPU_GEN:TUR,GPU_SKU:RTX2080TI,GPU_MEM:11GB,GPU_CC:7.5 cryoem 1 0/1/0/1 2:8:2 191567 0 infinite gpu:v100:4(S:0-1) cryoem-gpu50 CPU_GEN:SKX,CPU_SKU:4110,CPU_FRQ:2.10GHz,GPU_GEN:VLT,GPU_SKU:V100,GPU_MEM:32GB,GPU_CC:7.0
How can I request a GPU with certain features and or memory?
TBA... something about using Constraints. Maybe get the gres for gpu memory working.
What Accounts are there?
Accounts are used to allow us to track, monitor and report on usage of SDF resources. As such, users who are members of stakeholders of SDF hardware, should use their relevant Account to charge their jobs against. We do not associate any monetary value to Accounts currently, but we do require all Jobs to be charged against an Account.
| Account Name | Description | Contact |
|---|---|---|
| shared | Everyone | Yee |
| cryoem | CryoEM Group | Yee |
| neutrino | Neutrino Group | Kazu |
| cryoem-daq | CryoEM data acquitision | Yee |
| ml | Machine Learning Initiative | Daniel |
| suncat | SUNCAT Group | Johannes |
| hps | HPS Group | Omar |
| LCLS | LCLS Group | Wilko |
What Partitions are there?
Partitions define a grouping of machines. In our use case the grouping to refer to science and engineering groups who have purchased servers for the SDF. We do this such that members (or associates) of those groups can have priority access to their hardware. Whilst we give everyone access to all hardware, by default, users who belong to groups who do not own any stake in SDF will have lower priority access and use of stakeholder's resources.
| Partition Name | Purpose | Contact |
|---|---|---|
| shared | General resources; this contains all shareable reasources, including GPUs | Yee |
| ml | Machine Learning Initiative GPU servers | Daniel / Yee |
| cryoem | CryoEM GPU servers | Yee |
| neutrino | Neutrino GPU servers | Kazu |
| suncat | SUNCAT AMD Rome Servers | Johannes |
| hps | HPS AMD Rome Servers | Omar |
| fermi | Fermi (LAT) AMD Rome Servers | Richard |
| lcls | LCLS AMD Rome Servers | Wilko |
Help! My Job takes a long time before it starts!
This is often due to limited resources. The simplest way is to request less CPU (-N) or less memory for your Job. However, this will also likely increase the amount of time that you need for the Job to complete. Note that perfect scaling is often very difficult (ie using 16 CPUs will run twice as fast as 8 CPUs), so it may be beneficial to submit many smaller Jobs where possible. You can also set the --time option to specify that your job will only run upto that amount of time so that the scheduler can better fit your job in.
The more expensive option is to buy more hardware to SDF and have it added to your group/teams Partition.
How can I restrict/contraint which servers to run my Job on?
You can use slurm Constraints. We tag each and every server that help identify specific Features that each has: whether that is the kind of CPU, or the kind of GPU that run on them.
You can view a servers specific Feature's using
$ module load slurm $ scontrol show node ml-gpu01 NodeName=ml-gpu01 Arch=x86_64 CoresPerSocket=12 CPUAlloc=0 CPUTot=48 CPULoad=1.41 AvailableFeatures=CPU_GEN:SKX,CPU_SKU:5118,CPU_FRQ:2.30GHz,GPU_GEN:TUR,GPU_SKU:RTX2080TI,GPU_MEM:11GB,GPU_CC:7.5 ActiveFeatures=CPU_GEN:SKX,CPU_SKU:5118,CPU_FRQ:2.30GHz,GPU_GEN:TUR,GPU_SKU:RTX2080TI,GPU_MEM:11GB,GPU_CC:7.5 Gres=gpu:geforce_rtx_2080_ti:10(S:0) NodeAddr=ml-gpu01 NodeHostName=ml-gpu01 Version=19.05.2 OS=Linux 3.10.0-1062.4.1.el7.x86_64 #1 SMP Fri Oct 18 17:15:30 UTC 2019 RealMemory=191552 AllocMem=0 FreeMem=182473 Sockets=2 Boards=1 State=IDLE ThreadsPerCore=2 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A Partitions=gpu BootTime=2019-11-12T11:18:04 SlurmdStartTime=2019-12-06T16:42:16 CfgTRES=cpu=48,mem=191552M,billing=48,gres/gpu=10 AllocTRES= CapWatts=n/a CurrentWatts=0 AveWatts=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
We are openly investigating additional Features to add. Comments and suggestions welcome.
Documentation PENDING.
Possibly add: GPU_DRV, OS_VER, OS_TYPE