status: Under Construction
last update: 15 February 2012
This page is the beginning of a description of a full reprocessing based on Pass 7 analysis.A 'full' Pass 7 reprocessing is currently envisioned to begin ~August 2011a record of the configuration and execution of the P202 reprocessing project, full reprocessing from DIGIs using Pass7 analysis code. This project involves reprocessing with Pass7 classification trees and new IRFs. This task will read DIGI files and emit RECON, MERIT, a selection of other GCR and CAL ROOT files, and an the standard array of FITS files. It will be a CPU-intensive and storage-intensive enterprise requiring months of elapsed time and somewhere around 3/4 Pbytes of order 1 Pbyte of storage. At the time of this task beginning, there will be about 1620,000 science runs in Fermi (3.5 years accumulation).
In order to To avoid occupying a new 750 TB 1 PB of disk space, the plan is to remove old RECON files once they have been reprocessed. This is a shell game that involves some amount of buffer space and then waiting until the new RECON file has been created and (to some extent) validated before removal. Further, the The old RECON file must first be backed up to HPSS. A straw-man proposal for this scheme follows.
Assume 50 TB of xroot buffer space.
Reprocess ~10% of the mission data. Once complete, prepare list of files to be removed (RECON, possibly plus others
like CAL and SVAC if they are also regenerated by the reprocessing task) and hand off to Wilko. He will ensure the files to be deleted have been backed up and then delete the disk version.
files will be retained on tape in the HPSS system and they will be available via xroot (but with some delay as these large files are staged in).
This page is a record of the configuration for the P202 reprocessing project, full reprocessing from DIGIs using Pass7 analysis code. This project involves reprocessing with Pass7 classification trees and new IRFs. The name "P202" derives from the word "processing" and the initial file version to be used for the output data products, e.g., r0123456789_v202_merit.root.
...
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P202-ROOT |
Task Status | http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp |
GlastRelease | 17-35-24-gr17 (SCons RHEL4-32 build) |
Run Selection | based on "standard" selection Public Release 4 (https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets ) |
s/c data | "standard" Public Release 2 ( https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets ) |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-ROOT/config/runList.txt |
photonFilter | CTBParticleType==0 && CTBClassLevel>0 |
electronFilter | CTBParticleType==1 |
Code Variants | redhat4-i686-32bit-gcc34 (Optimized) |
jobOpts | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-ROOT/config/doRecon.txt |
Output Data Products |
...