Page History

AFS Home Directory

By default you get a fairly small AFS quota on your home directory. You can use the AFS Quota Self-service Form to increase your total AFS quota up to 10GB. If you run into questions, please contact the unix-admin

NFS filesystems

Dedicated ATLAS NFS spaces are available. NFS are good for log file, batch job input and output, and scratch. These NFS spaces are not backed up. So code are recommended to remain on AFS space.

• /afs/slac.stanford.edu/g/atlas/work/<firstLetterOfUsername>/<username>. (It is actually an NFS disk space)
• /nfs/slac/g/atlas/u01
• /nfs/slac/g/atlas/u02

These spaces are not automatically cleaned up.

Xrootd spaces

There are two high performance xrootd spaces - The Tier 2 space and the Tier 3 space. These space are primarily for storing ROOT files and ATLAS datasets. They . Creating a new file or directory on a Xrootd space takes at least 5 seconds. So they are not good for large number of small files such as logs, codes, and small text files.

On ATLAS interactive nodes atlint0[1-4] (and only on them), one can see these spaces as mounted filesystems:

$ ssh atlint01.slac.stanford.edu
$ df -h /atlas /xrootd/atlas
Filesystem      Size  Used Avail Use% Mounted on
xrootdfs        741T  682T   60T  93% /atlas
xrootdfs        2.8P  1.9P  893T  69% /xrootd/atlas
$ ls -l /atlas
total 15
drwxrwxrwx 1 daemon daemon  140 Oct  7 11:43 dq2/
drwxrwxrwx 1 daemon daemon    6 Oct 25  2012 group/
drwxrwxrwx 1 daemon daemon 4096 Oct  7 12:27 local/
drwxrwxrwx 1 daemon daemon 4096 Dec 23  2014 output/
drwxrwxrwx 1 daemon daemon 4096 Aug  4 11:28 proof/
drwxrwxrwx 1 daemon daemon   37 Jan 23  2012 test/
$ cat /atlas/dq2/site-size
{"sizes": {"total": 813980054454272, "available": 65063350558720}, "time" :"2015-11-18T19:30:01"}
$ cd /atlas/local
$ rucio download ...

/xrootd/atlas is the Tier 2 space. /atlas is the Tier 3 space. Avoid using atlint03 for large number of filesystem access.

Tier 2 space

The Tier 2 space is for Tier 2 only so users should not write to it. However, SLAC ATLAS users are encouraged to use R2D2 to transfer official ATLAS datasets to the WT2 (the SLAC Tier 2. The corresponding DDM endpoint names are SLACXRD_*DISK) spaces (space token: USERDISK, GROUPDISK, LOCALGROUPDISK, SCRATCHDISK). Once they are at the Tier 2 storage, they are accessible (readonly of course) from all SLAC interactive nodes and batch nodes.

• On atlint0[1-4] this space is mounted at /xrootd/atlas/
• In your batch jobs, you can access them via the xrootd protocol (explained below). Files in xrootd have a URL-type format. The accessing URL is root://atl-xrdr//atlas/xrootd/atlas... (change: from /xrootd/atlas/XXX above to /atlas/xroot/XXX also works)

Tier 3 space

SLAC ATLAS group owns a proof private cluster. It is actually a proof cluster, a batch cluster and a xrootd storage cluster . (the Tier 3 space). It was also a proof cluster but the proof function is obsolete.

•  On atlint0[1-4] the xrootd storage cluster (the Tier 3 space) is mounted at /atlas .
• You can use R2D2 to transfer official ATLAS datasets to the space under directory /atlas/dq2. The Tier 3's name in R2D2 is SLAC-ATLAS-T3_GRIDFTP.
• Files under /atlas/proof and /atlas/output are used by the proof jobs.
• Space under /atlas/local are for users to read and write.
• In your batch jobs, you can access them via the xrootd protocol (explained below). The accessing URL is root://atlprf01:11094//atlas/...

Access the xrootd spaces

To read input data in batch jobs, we recommend either directly read the ROOT files using the xrootd protocol, or by copying files (ROOT file and non-ROOT file) via "xrdcp" to batch nodes' /scratch space.

...

filelist = []
filelist += ["root://atl-xrdr//atlas/rucio/data15_13TeV:DAOD_MUON0.06783725._000004.pool.root.1"]
filelist += ["root://atl-xrdr//atlas/rucio/data15_13TeV:DAOD_MUON0.06783725._000001.pool.root.1"]
...
athenaCommonFlags.PoolESDInput=filelist

Experimental: If the files are not at SLAC but are available in an ATLAS site that provides xrootd service, you can also access them by replacing "atl-xrdr" by "atlfax" (see below).

filelist = []
filelist += ["root://atlfax//atlas/rucio/data15_13TeV:DAOD_MUON0.06783725._000004.pool.root.1"]
filelist += ["root://atlfax//atlas/rucio/data15_13TeV:DAOD_MUON0.06783725._000001.pool.root.1"]
...
athenaCommonFlags.PoolESDInput=filelist

To read an xrootd file in ROOT:

root[0] TXNetFile f("root://atl-xrdratlprf01:11094//atlas/xrootd/usr/g/gowdy/local/myFile.root");

To copy files in and out of xrootd:

xrdcp myFile.root root://atl-xrdratlprf01:11094//atlas/xrootd/usr/g/gowdy/local/myFile.root
xrdcp root://atlprf01:11094//atlas/local/myFile.root .

Filesystem type access only available on atlint0[1-4]. 

$ ssh atlint01.slac.stanford.edu
$ df -h /atlas /xrootd/atlas
Filesystem      Size  Used Avail Use% Mounted on
xrootdfs        741T  682T   60T  93% /atlas
xrootdfs        2.8P  1.9P  893T  69% /xrootd/atlas
$ cat /atlas/dq2/site-size
$ rucio download ...

...

myFile.root