• Created by Unknown User (fizisist), last modified by Unknown User (ahaas) on Oct 08, 2009

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

This page documents some SLAC specific examples of using the GRID for dataset production and retrieval.

GRID Certificates

The centrally maintained ATLAS page for starting on the GRID is the place to start

DQ2 Setup at SLAC

DQ2 is the ATLAS data management system. There is significant documentation on it's general prinicples, usage, and troubleshooting here

In order to being using the DQ2 tools at SLAC one simply needs to source the following script

 source /afs/slac.stanford.edu/g/atlas/etc/hepix/GridSetup.sh

This script should be automatically run for you if you are using the standard ATLAS setup, described here:SLAC ATLAS Computing Environment

You then need to get a GRID ticket, which will last 12 hours or so:

voms-proxy-init -voms atlas

Some simple dq2 commands

To add your files to a personal dataset (which can also be used by others):

dq2-put --long-surls -s /xrootd/atlas/usr/f/fizisist/test user.DavidWilkinsMiller.misal1_csc11.005009.J0_pythia_jetjet.pile1sf01.AOD.v13003003.verylow.TEST9

The dataset name you use should conform to "user09.DN.name.datatype.version"
where DN is your identifier extracted from your certificate, and can be computed from:

python /afs/cern.ch/atlas/offline/external/GRID/ddm/Operations/utils/dn.py

To list the files in a dataset (note, you can use wildcards...):

dq2-ls -f user.DavidWilkinsMiller.misal1_csc11.005009.J0_pythia_jetjet.pile1sf01.AOD.*.verylow.TEST9

To get the dataset:

cd /tmp
dq2-get user.DavidWilkinsMiller.misal1_csc11.005009.J0_pythia_jetjet.pile1sf01.AOD.v13003003.verylow.TEST9
ls -l user.DavidWilkinsMiller.misal1_csc11.005009.J0_pythia_jetjet.pile1sf01.AOD.v13003003.verylow.TEST9/

Transfering large datasets

To request an import of a large dataset to SLAC (it must be avilable first at BNL!):


dq2-register-subscription --archive <dataSet> SLACXRD_USERDISK

(the --archive flag makes sure it doesn't automatically get deleted after a week)

It will take some time for the data to appear. You can check with:


dq2-ls -f <dataSet> -L SLACXRD

to see how many files are available locally.

The same works with containers:

dq2-register-subscription-container --archive  data09_cos.00121416.physics_L1Calo.merge.DPD_CALOCOMM.r733_p37/ SLACXRD_USERDISK
dq2-list-dataset-replicas-container data09_cos.00121416.physics_L1Calo.merge.DPD_CALOCOMM.r733_p37/
dq2-ls -f -H data09_cos.00121416.physics_L1Calo.merge.DPD_CALOCOMM.r733_p37/

And you can make a PoolFileCatalog.xml file directly:

dq2-ls -L SLACXRD -P <dataSet>
sed s%srm://osgserv04.slac.stanford.edu:8443/srm/v2/server?SFN=/xrootd/atlas%root://atl-xrdr//atlas/xrootd%g PoolFileCatalog.xml >! PoolFileCatalog.xml

PATHENA Setup at SLAC

A commonly used set of tools for distributed analysis is PANDA

In order to being using these tools at SLAC, one simply needs to source 1 script and set 1 environment variable

source /afs/slac/g/atlas/packages/panda-client/etc/panda/panda_setup.sh
export PATHENA_GRID_SETUP_SH="/afs/slac/package/vdt/wlcg-client/setup.sh"
  • No labels