...
NOTE: One run, 242429468, of type TrigTest was declared 'good for science' but long after this task got started, so it and has been intentionally omittedincluded.
Wiki Markup |
---|
\[to be continued...\] |
Bookkeeping
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P202
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
...
Progress at the 1-year mark:
First run | 239557414 (MET), 2008-08-04 15:43:34 (UTC) |
| |
Last run | 271999199 (MET), 2009-08-15 03:19:57 (UTC) |
| |
Total runs | 5600 |
| |
Total input DIGI events | 11,928,911,465 |
| |
Total RECON events | 11,928,911,465 | 161.4 TB | |
Total CAL events | 11,928,911,465 | 36.1 TB | |
Total GCR events | 11,928,911,465 | 260.5 GB | |
Total MERIT events | 11,928,911,465 | 9.6 TB | all triggered events |
Total FILTEREDMERIT events | 1,572,783,868 | 1.3 TB | all photon event classes |
Total EXTENDEDFT1 | 1,572,783,826 | 143.7 GB | all photon event classes |
Total LS1 events | 1,572,783,826 | 255.0 GB | all photon event classes |
Total LS1 (FSSC selection) events | 271,923,333 | 44.2 GB | event classes (bits) 0,2,3,4 (transient, source, clean, ultraclean) |
Total FT1 (FSSC selection) events | 24,261,962 | 2.4 GB | event classes (bits) 2,3,4 (source, clean, ultraclean) |
Wiki Markup |
---|
\[to be continued...\] |
Bookkeeping
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P202
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
...
P202-ROOT
Anchor | ||||
---|---|---|---|---|
|
Status chronology
- 2/13/2012 - begin trials with final calibration and alignments from Leon; 5 runs reprocessed
- 2/14/2012 - trials continue with blocks of 15, 20, 25 and 50 runs reprocessed (each run generates ~20 batch jobs)
- 2/16/2012 - begin trickleStream production. Initial config:
Code Block
...
Status chronology
- 2/13/2012 - begin trials with final calibration and alignments from Leon; 5 runs reprocessed
- 2/14/2012 - trials continue with blocks of 15, 20, 25 and 50 runs reprocessed (each run generates ~20 batch jobs)
- 2/16/2012 - begin trickleStream production. Initial config:
Code Block ========================================================================================== TRICKLE PARMS =============================================================================== task = P202-ROOT maxRuns = 19172 firstStep = setupRun steps = [['/processRun processClump', 1500, 20], ['mergeClumps', 70, 1]] maxStreamsPerCycle = 20 timePerCycle = 300 ===============================================================================
- 2/21/2012 - One clump reprocessed with pointer to new mySQL DB (stream 710.0)
- 2/22/2012 - 776 runs complete. Pausing task.
- 3/15/2012 - resume task. New goal is 1-year of data (~5600 runs)
- 3/31/2012 - 1-year complete (5600 runs). DataCatalog summary:There have been a few nasty problems which need to be fixed before continuing.
S/W component | bug fix | ||
---|---|---|---|
New ROOT version | 5-min 'transaction timeout' triggered by xroot data server reboot | ||
New GlastRelease | 1) include new ROOT version (above); 2) exit with non-zero RC on ROOT write error | ||
New GPL_TOOLS | check size/checksum of file written to xroot with known size/checksum | ||
Tuned xroot on new Dell servers | silent file truncation when volume fills up JIRA | ||
New TSkim | 1) new ROOT version (above); 2) complain on ROOT write errors | ||
Name | Files | Events | Size |
FILTEREDMERIT | 5600 | 1,572,783,868 | 1.3 TB |
MERIT | 5600 | 11,928,911,465 | 9.6 TB |
CAL | 5600 | 11,928,911,465 | 36.1 TB |
GCR | 5600 | 11,928,911,465 | 260.5 GB |
RECON | 5600 | 11,928,911,465 | 161.4 TB |
Configuration
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P202-ROOT |
Task Status | http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp![]() |
GlastRelease | 17-35-24-gr17 (SCons RHEL4-32 build) |
Run Selection | based on a modified "standard" selection, see https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
s/c data | "standard" Public Release 2 https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-ROOT/config/runList.txt![]() |
photonFilter | CTBParticleType==1 && ((FT1EventClass & 0x00003EFF)!=0) |
electronFilter | CTBParticleType==1 |
Code Variants | redhat4-i686-32bit-gcc34 (Optimized) |
jobOpts | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-ROOT/config/doRecon.txt![]() |
Output Data Products |
...
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="6c5210c185bd597b-28adcbcb-47b94271-8127996e-4274148a05144c39bd71837d"><ac:plain-text-body><![CDATA[ | Data Product | destination | data content [1] | event selection [1] | makeFT1 | gtselect | gtdiffrsp | gtmktime | ]]></ac:plain-text-body></ac:structured-macro> |
---|---|---|---|---|---|---|---|---|---|
EXTENDEDFT1 | SLAC | FT1variables | ((FT1EventClass & 0x00003EFF)!=0) | ||||||
FT1 | FSSC+SLAC | FT1variables | 'source' and above |
| | (inherited) | |||
EXTENDEDLS1 | SLAC | LS1variables | ((FT1EventClass & 0x00003EFF)!=0) | ||||||
LS1 | FSSC+SLAC | LS1variables | 'transient' and above | (inherited) | |||||
ELECTRONFT1 | SLAC | FT1variables | CTBParticleType==1 |
...