View Source

Description in progress as of 2023-02-21.

As of this writing, access to S3DF is still by invitation.

This page gives a very brief introduction to SLAC's new S3DF (SLAC Shared Scientific Data Facility) cluster to help you get started. We assume you already have a Unix account and your main intent is to run the Fermitools/Fermipy.

See the main S3DF documentation for detailed information about how to log in, use the SLURM batch system, and so on. Specify --partition roma,milano.

Basically, ssh to s3dflogin.slac.stanford.edu and from there ssh to iana (no .slac.stanford.edu) to do actual interactive work. The login nodes are not meant for doing analysis or accessing data. Of course, real computational intensive tasks are meant for the batch system and not the interactive nodes either.

.bashrc:

a directory gets added to your home dir, called profile_d. It points back to the group equivalent in /sdf/group/fermi/sw/ and includes the contents of those conf files into your session's bashrc. Group-level settings go there, eg $LATCalibRoot.
don't overwrite your .bash_profile or you'll lose the code that does this:

# SLAC S3DF - source all files under ~/.profile.d
if [[ -e ~/.profile.d && -n "$(ls -A ~/.profile.d/)" ]]; then
source <(cat $(find -L ~/.profile.d -name '*.conf'))
fi

Disk space:

Your home directory is in weka (/sdf/home/<first letter of your userid>/<your userid>) with 30 GB of space. This space is backed up and is where code, etc., should go.
We have group space at /sdf/group/fermi/ which includes shared software, as well as Fermi-supplied user (i.e., on top of your home directory) and group space. You can find it in /sdf/group/fermi/u/<you>.
- after gpfs is retired in late 2023, this is where your larger user space will be.
your user/group space on the old clusters is not directly accesible from s3df - it currently needs to be copied over.
- We're still providing additional user space from the old cluster, available on request via the slac-helplist mailing list. It is not backed up. This space is natively gpfs. User directories are available under: /gpfs/slac/fermi/fs2/u/<your_dir>; we have not created any group directories on s3df yet.

If you were working on SDF, note that S3DF is completely separate (aside from the account name). Even though path names might look similar, they are on different file systems. You can still access all your SDF files by prepending "/fs/ddn/" to the paths you were used to.

Fermitools and other analysis software (e.g., 3ML) are available via shared Conda installation, so you don't need to install Conda yourself. See Fermitools/Conda Shared Installation at SLAC. If you do want your own Conda, you shouldn't install it in your home directory due to quota limits. Follow the S3DF documentation instructions to install Conda and set a prefix path for the Conda installation that will put it and any environments you create in your group space. However, you should use a prefix to your personal space, e.g., /sdf/group/fermi/u/$USER/miniconda3, instead of the path in their example.

You can also run a RHEL6 Singularity container (for apps that are not portable to RHEL/Centos7). See Using RHEL6 Singularity Container.

For generic advice on running in batch, see Running on SLAC Central Linux. Note that the actual batch system has changed and we have not updated the doc to reflect that. This is advice on copying data to local scratch, etc.