There are two high performance xrootd spaces - The Tier 2 space and the Tier 3 space. These space are primarily for storing ROOT files and ATLAS datasets. Creating a new file or directory on a Xrootd space takes at least 5 seconds. So they are not good for large number of small files such as logs, codes, and small text files.
On ATLAS interactive nodes atlint0[1-4] (and only on them), one can see these spaces as mounted filesystems:
| No Format |
|---|
$ ssh atlint01.slac.stanford.edu
$ df -h /atlas /xrootd/atlas
Filesystem Size Used Avail Use% Mounted on
xrootdfs 741T 682T 60T 93% /atlas
xrootdfs 2.8P 1.9P 893T 69% /xrootd/atlas
$ ls -l /atlas
total 15
drwxrwxrwx 1 daemon daemon 140 Oct 7 11:43 dq2/
drwxrwxrwx 1 daemon daemon 6 Oct 25 2012 group/
drwxrwxrwx 1 daemon daemon 4096 Oct 7 12:27 local/
drwxrwxrwx 1 daemon daemon 4096 Dec 23 2014 output/
drwxrwxrwx 1 daemon daemon 4096 Aug 4 11:28 proof/
drwxrwxrwx 1 daemon daemon 37 Jan 23 2012 test/
$ cat /atlas/dq2/site-size
{"sizes": {"total": 813980054454272, "available": 65063350558720}, "time" :"2015-11-18T19:30:01"}
$ cd /atlas/local
$ rucio download ... |
/xrootd/atlas is the Tier 2 space. /atlas is the Tier 3 space. Avoid using atlint03 for large number of filesystem access.
Tier 2 space
The Tier 2 space is for Tier 2 only so users should not write to it. However, SLAC ATLAS users are encouraged to use R2D2 to transfer official ATLAS datasets to the WT2 (the SLAC Tier 2. The corresponding DDM endpoint names are SLACXRD_*DISK) spaces (space token: USERDISK, GROUPDISK, LOCALGROUPDISK, SCRATCHDISK). Once they are at the Tier 2 storage, they are accessible (readonly of course) from all SLAC interactive nodes and batch nodes.
- On atlint0[1-4] this space is mounted at /xrootd/atlas/
- In your batch jobs, you can access them via the xrootd protocol (explained below). Files in xrootd have a URL-type format. The accessing URL is root://atl-xrdr//xrootd/atlas... (/atlas/xroot/XXX also works)
Tier 3 space
SLAC ATLAS group owns a private cluster. It is actually a batch cluster and a xrootd storage cluster (the Tier 3 space). It was also a proof cluster but the proof function is obsolete.
- On atlint0[1-4] the xrootd storage cluster is mounted at /atlas .
- You can use R2D2 to transfer official ATLAS datasets to the space under directory /atlas/dq2. The Tier 3's name in R2D2 is SLAC-ATLAS-T3_GRIDFTP.
Files under /atlas/proof and /atlas/output are used by the proof jobs.- Space under /atlas/local are for users to read and write.
- In your batch jobs, you can access them via the xrootd protocol (explained below). The accessing URL is root://atlprf01:11094//atlas/...
Access the xrootd spaces
To read input data in batch jobs, we recommend either directly read the ROOT files using the xrootd protocol, or by copying files (ROOT file and non-ROOT file) via "xrdcp" to batch nodes' /scratch space.
For output files, write the output to batch node's /scratch space first. At the end of your batch job, use "cp" to copy them back to the NFS space or "xrdcp" to copy them to the Tier 3 xrootd space.
To read an xrootd files in Athena using the xrootd protocol:
| Code Block |
|---|
AFS Home Directory
By default you get a fairly small AFS quota on your home directory.
You can ask unix-admin to increase it to 500MB or for other volumes to be added to your home directory with a quota of 500MB.
SCCS has a form to make this request.
NFS filesystems
Dedicated ATLAS scratch space is available.
You should create a directory for yourself with the format:/afs/slac.stanford.edu/g/atlas/work/<firstLetterOfUsername>/<username>.
It is actually an NFS disk space not backed up so that code are recommended to remain on your AFS space.
For the moment this space is not automatically cleaned up, if it fills up we'll need to start doing that.
Currently there are two 125GB volumes making up this space.
This space should be used for log files and other small files that don't need high performance access.
xrootd user space
This space is primarily for storing ROOT files, any type (POOL, ntuples, whatever). xrootd offers high performance access to these types of files.
Access to this area is either directly from ROOT or by copying files in or out via xrdcp.
The recommended access type is direct access for reading and to write via xrdcp
(write the output to a local disk on the machine your job is on, like /scratch on the batch workers).
Files in xrootd have a URL-type format. You can add them to a job and Athena knows how to use them. An example file would be:
| Code Block |
|---|
filelist = [] filelist += ["root://atl-xrdr//atlas/xrootd/atlasuserdisk/data09_cosrucio/DPDdata15_CALOCOMM13TeV/r733_p37/data09_cos.00121416.physics_L1Calo.merge.DPD_CALOCOMM.r733_p37_tid073973/DPD_CALOCOMM.073973._00004607/53/DAOD_MUON0.06783725._000004.pool.root.1"] filelist += ["root://atl-xrdr//atlas/xrootd/atlasuserdisk/data09_cosrucio/DPDdata15_CALOCOMM13TeV/r733_p37/data09_cos.00121416.physics_L1Calo.merge.DPD_CALOCOMM.r733_p37_tid073973/DPD_CALOCOMM.073973._0000164f/ca/DAOD_MUON0.06783725._000001.pool.root.1"] ... athenaCommonFlags.PoolESDInput=filelist |
To access an xrootd file in ROOT directly, doThe Tier 2 space supports FAX global logical file name (gLFN), so the above two files can be simplified as:
| Code Block |
|---|
root[0] TXNetFile f(filelist = [] filelist += ["root://atl-xrdr//atlas/xrootd/usr/g/gowdy/myFile.root"); |
To copy files in and out of xrootd:
| Code Block |
|---|
xrdcp myFile.root rucio/data15_13TeV:DAOD_MUON0.06783725._000004.pool.root.1"] filelist += ["root://atl-xrdr//atlas/xrootd/usr/g/gowdy/ xrdcp root://atl-xrdr//atlas/xrootd/usr/g/gowdy/myFile.root . |
Filesystem type access is not transparent currently due to the distributed nature of the xrootd system. However, the xrootd filesystem is MOUNTED on atlint01. You can should make a directory called /xrootd/atlas/usr/"first letter of username"/"username"/
rucio/data15_13TeV:DAOD_MUON0.06783725._000001.pool.root.1"]
...
athenaCommonFlags.PoolESDInput=filelist |
Experimental: If the files are not at SLAC but are available in an ATLAS site that provides xrootd service, you can also access them by replacing "atl-xrdr" by "atlfax" (see below).
| Code Block |
|---|
filelist = []
filelist += ["root://atlfax//atlas/rucio/data15_13TeV:DAOD_MUON0.06783725._000004.pool.root.1"]
filelist += ["root://atlfax//atlas/rucio/data15_13TeV:DAOD_MUON0.06783725._000001.pool.root.1"]
...
athenaCommonFlags.PoolESDInput=filelist |
To read an xrootd file in ROOT:
| Code Block |
|---|
root[0] TXNetFile f("root://atlprf01:11094//atlas/local/myFile.root");
|
To copy files in and out of xrootd:
| Code Block |
|---|
xrdcp myFile.root root://atlprf01:11094//atlas/local/myFile.root
xrdcp root://atlprf01:11094//atlas/local/myFile.root myFile.root |
| No Format |
ssh atlint01.slac.stanford.edu
df -h
xrootdfs 9.8T 972G 8.9T 10% /xrootd/atlas/usr
xrootdfs 9.8T 972G 8.9T 10% /xrootd/atlas/dq2
xrootdfs 9.8T 972G 8.9T 10% /xrootd/atlas/atlasdatadisk
xrootdfs 9.8T 972G 8.9T 10% /xrootd/atlas/atlasmcdisk
xrootdfs 9.8T 972G 8.9T 10% /xrootd/atlas/atlasuserdisk
xrootdfs 9.8T 972G 8.9T 10% /xrootd/atlas/atlasgroupdisk
xrootdfs 9.8T 972G 8.9T 10% /xrootd/atlas/atlashotdisk
cat /xrootd/atlas/usr/a/ahaas/test.txt
|