Get Started on S3DF - Cheat Sheet

You can modify your .ssh config to allow direct passwordless access from your device to iana, by adding this to your .ssh/config file on your end:

Host slac*
        User <you>

Host slacl
        Hostname s3dflogin.slac.stanford.edu

Host slacd
        Hostname iana
        ProxyJump slacl

and then add your e.g. ~/.ssh/id_rsa.pub from from your device to ~/.ssh/authorized_keys at SLAC, using:

ssh-copy-id <you>@s3dflogin.slac.stanford.edu

.bashrc:

• a directory gets added to your home dir, called profile_d. It points back to the group equivalent in /sdf/group/fermi/sw/ and includes the contents of those conf files into your session's bashrc. Group-level settings go there, eg $LATCalibRoot.
• don't overwrite your .bash_profile or you'll lose the code that does this:

# SLAC S3DF - source all files under ~/.profile.d
if [[ -e ~/.profile.d && -n "$(ls -A ~/.profile.d/)" ]]; then
 source <(cat $(find -L ~/.profile.d -name '*.conf'))
fi

• Your home directory is in weka (/sdf/home/<first letter of your userid>/<your userid>) with 30 GB of space. This space is backed up and is where code, etc., should go. 
• We have group space at /sdf/group/fermi/:
  • some directories are under /sdf/data/fermi/, but we provide links into the group directory tree for easier access
  • includes shared software, including conda envs for Fermitools and containers for running rhel6 executables
  • Fermi-supplied user (i.e., on top of your home directory) space.
    • You can find it in /sdf/group/fermi/u/<you>. There is a symlink to it, called "fermi-user", in your home directory for convenience.
    • after gpfs is retired in late 2023, this is where your larger user space will be.
  • group space in /sdf/group/fermi/g/ - a one-time copy has been done of all the gpfs g/ directories, under /nfs/farm/g/glast/g/.
  • all of glast afs has been copied to /sdf/group/fermi/a/
• your user/group space on the old clusters is not directly accessible from s3df - it currently needs to be copied over. (this access policy may get reversed soon)
  • We're still providing additional user space from the old cluster, available on request via the slac-helplist mailing list. It is not backed up. This space is natively gpfs.  User directories are available under: /gpfs/slac/fermi/fs2/u/<your_dir>.

If you were working on SDF, note that S3DF is completely separate (aside from the account name). Even though path names might look similar, they are on different file systems. You can still access all your SDF files by prepending "/fs/ddn/" to the paths you were used to.

Fermitools and other analysis software (e.g., 3ML) are available via shared Conda installation, so you don't need to install Conda yourself. See Fermitools/Conda Shared Installation at SLAC.  If you do want your own Conda, you shouldn't install it in your home directory due to quota limits; put it in your Fermi-supplied user space. Follow the S3DF documentation instructions to install Conda and set a prefix path for the Conda installation that will put it and any environments you create in your group-provided space. However, you should use a prefix to your personal space, e.g., /sdf/group/fermi/u/$USER/miniconda3, instead of the path in their example.

You can also run a RHEL6 Singularity container (for apps that are not portable to RHEL/Centos7). See Using RHEL6 Singularity Container.

For generic advice on running in batch, see Running on SLAC Central Linux.  Note that the actual batch system has changed and we have not updated the doc to reflect that. This is advice on copying data to local scratch, etc.

• LSB_JOBID -> SLURM_JOB_ID
• scratch space during job execution:
  • at job start, a directory is automatically created on the scratch of the worker: ${LSCRATCH} = /lscratch/${USER}/slurm_job_id_${SLURM_JOB_ID}
  • once all of a user's jobs on a node are completed/exited, their corresponding LSCRATCH directory on that host is deleted.

You can run cronjobs in S3DF. Users don't have to worry about token expiration like on AFS. Select one of the iana interactive nodes (and remember which one!) to run on.

Note: crontab does NOT inherit your environment. You'll need to set that up yourself.

Since crontab is per host (no trscrontab), if the node is reinstalled or removed, the crontab will be lost. It's probably best to save your crontab as a file in your home directory so that you can re-add your cronjobs if this happens:

crontab -l > ~/crontab.backup

Then to re-add the jobs back in:

crontab ~/crontab.backup

xrootd CLI commands are available via cvmfs:

module load osg/client-latest

which xrdcp

module unload osg/client-latest   #when done with xrootd

Oracle drivers etc have been installed in /sdf/group/fermi/sw/oracle/. The setup.sh file in the driver-version directory sets up everything needed to issue sqlplus commands from the command line.

/sdf/home/g/glast/a/datacat/prod/datacat

/sdf/home/g/glast/a/pipeline-II/prod/pipeline

For now, we are leaving the live cvs repo on nfs. The cvs client has been installed on the iana nodes.

Set:

CVSROOT=:ext:$<USER>@centaurusa.slac.stanford.edu:/nfs/slac/g/glast/ground/cvs