Page History

...

• a partition and slurm account should be specified when submitting jobs. The slurm account would be lcls:<experiment-name> e.g. lcls:xpp123456. The account  is used for keeping track of resource usage per experiment

  Code Block
  % sbatch -p milano --account lcls:xpp1234 ........

• Submitting jobs using the lcls account allows only to submit preemptable jobs and requires to specify: --qos preemptable . The lcls account is set by --account lcls or --account lcls:default  (the lcls name gets automatically translated to lcls:default by slurm, and s-commands will show the default one).
• In the S3DF by default memory is limited to 4GB/core. Usually that is not an issue as processing jobs use many core (e.g. a job with 64 cores would request 256GB memory)
• the memory limit will be enforced and your job will fail with an OUT_OF_MEMORY status
• memory can be increased using the--memsbatch option (e.g.: --mem 16G, default unit is megabytes)
  
• Default total run time is 1 day, the --time option allows to increase/decrease it.
  

• Number of cores

  Warning
  Some cores of a milano batch node are exclusively used for file IO (WekaFS). Therefore although a milano node has 128 core only 120 can be used. submitting a task with  --nodes 1 --ntasks-per-node=128 would fail with:  Requested node configuration is not available

• Environment Varibales: sbatch option can also be set via environment variables which is useful if a program is executed that calls sbatch and doesn't allow to set options on the command line e.g.:

  Code Block
  language bash
  % SLURM_ACCOUNT=lcls:experiment executable-to-run [args] or % export SLURM_ACCOUNT=lcls:experiment % executable-to-run [args]

  The environment variables are SBATCH_MEM_PER_NODE (--mem), SLURM_ACCOUNT(--account) and SBATCH_TIMELIMIT (--time). The order arguments are selected is: command line, environment and withing sbatch script. 

Real-time Analysis Using A Reservation For A Running Experiment

Currently we are reserving nodes for experiments that need real-time processing.  This is an example of parameters that should be added to a slurm submission script:

...