Versions Compared

Old Version 39

changes.mady.by.user Christopher O'Grady

Saved on

compared with

New Version 40

changes.mady.by.user Wilko Kroeger

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • a partition and slurm account should be specified when submitting jobs. The slurm account would be lcls:<experiment-name> e.g. lcls:xpp123456. The account  is used for keeping track of resource usage per experiment

    Code Block
    % sbatch -p milano --account lcls:xpp1234 ........
  • In the S3DF by default memory is limited to 4GB/core. Usually that is not an issue as processing jobs use many core (e.g. a job with 64 cores would request 256GB memory)
  • the memory limit will be enforced and your job will fail with an OUT_OF_MEMORY status
  • memory can be increased using the--memsbatch option (e.g.: --mem 16G, default unit is megabytes)
  • Default total run time is 1 day, the --time option allows to increase/decrease it.

  • Environment Varibales: sbatch option can also be set via environment variables which is useful if a program is executed that calls sbatch and doesn't allow to set options on the command line e.g.:

    Code Block
    languagebash
    % SLURM_ACCOUNT=lcls:experiment  executable-to-run [args]
or
% export SLURM_ACCOUNT=lcls:experiment
% executable-to-run [args]

    The environment variables are SBATCH_MEM_PER_NODE (--mem), SLURM_ACCOUNT(--account) and SBATCH_TIMELIMIT (--time). The order arguments are selected is: command line, environment and withing sbatch script. 

MPI and Slurm

For running mpi jobs on the S3DF slurm cluster mpirun (or related tools) should be used. Using srun to run mpi4py will fail as it requires pmix which is not supported by the Slurm version.

Jupyter

Jupyter is provided by the onDemand service (We are not planning to run a standalone jupyterhub as is done at PCDS. For more information check the S3DF interactive compute docs).

...

The page will at first show a "Starting" status, after some time it will switch to "Running" and the "Connect to Jupyter" button will be available. Click on the "Connect to Jupyter" button to start working on Jupyter. The Session ID link instead allows to open the directory where the Jupyter is run and access to the files, included the logs.


Offsite data transfers

The S3DF provides a group of transfer nodes for moving data in/out. The nodes are accessed using the load balanced name:  s3dfdtn.slac.stanford.edu