status: Complete through 06 October 2012
last update: 17 22 Oct 2012
This page is a record of the configuration and execution of the P202 reprocessing project, full reprocessing from DIGIs using Pass7 analysis code. This project involves reprocessing with Pass7 classification trees and up-to-date alignment/calibration data. This task will read DIGI files and emit RECON, MERIT, GCR and CAL ROOT files, and the standard array of FITS files. It will be a CPU-intensive and storage-intensive enterprise requiring months of elapsed time and of order 0.7 Pbyte of storage. At the time of this task beginning, there will be about 20,000 science runs in Fermi (3.5 years accumulation).
...
- 3/2/2012 - Define block 1 as the 776 runs in P202-ROOT block 1. Configure trickleStream and begin (14:08)
- 3/31/2012 - Define block 2 as 5600 runs. Reconfig trickleStream and begin (18:05)
- 4/01/2012 - Block 2 complete (most of the 4824 jobs completed in about six hours w/1000 job limit).
- 5/31/2012
- Discover stream 5599 (run 271999199) requires rollback - new MERIT file version (v203->v205).
- Regenerate runlist with 20,229 input MERIT files.
- Minor config change: twoClumpMin=False (formerly True, but caused unnecessary extra processClump dummy jobs)
- 6/5/2012 - Final cleanup
- Five runs are responsible for discrepant event tallies:
Run
Stream
Reason
Action
239557414
0
bad TCut
rollback ROOT + FITS
241599746
352
bad processing order
rollback FITS
245403855
1019
bad start time
change start time in runList and rollback
332661583
16244
silent root errors
rollback ROOT
339081502
17416
silent root errors
rollback ROOT
- Code changes:
directory
script
modification
commonTools/00-01-00
repTools.py
added new getKey() function to extract #events from FITS event files
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="36a90194ef93ae37-4c57e0c9-4cb34df4-b7569205-e5719ca8fc8711fd7ed7267a"><ac:plain-text-body><![CDATA[
P202-FITS/config
config.py
added os.environ['HEADASNOQUERY']='true' to enable 'ftlist' to run in batch
]]></ac:plain-text-body></ac:structured-macro>
P202-FITS/config
makeFITS.py
add diagnostic print of #evts in FITS files for each processing sub-step
- After these actions, the dataCatalog tallies are now consistent: Note that the number of events in ELECTRONFT1 files are not currently tallied by the dataCatalog.
Name
Type
Files
Events
Size
Created (UTC)
Links
Group
20229
0
8.5 GB
02-Mar-2012 00:06:07
Group
20229
90,904,582
205.7 GB
25-Jan-2012 00:53:32
Group
20229
189,323,074
17.8 GB
02-Mar-2012 00:06:06
Group
20229
1,325,204,821
215.3 GB
02-Mar-2012 00:06:08
Group
20229
6,291,424,926
574.7 GB
02-Mar-2012 00:06:09
Group
20229
6,291,424,926
1,020.1 GB
02-Mar-2012 00:06:09
Group
20229
6,291,424,926
5.3 TB
25-Jan-2012 00:53:29
Group
20229
44,125,679,961
35.4 TB
25-Jan-2012 00:53:30
Group
20229
44,125,679,961
590.0 TB
25-Jan-2012 00:53:33
Group
20229
44,125,679,961
942.7 GB
25-Jan-2012 00:53:31
Group
20229
44,125,679,961
128.7 TB
25-Jan-2012 00:53:31
- Five runs are responsible for discrepant event tallies:
- 8/24/2012 - Configure and run backfill through 31 July 2012
- 10/7/2012 - Rollback the following seven streams to fix corrupt FITS files. The makeFT1 app received a large number of ROOT error, but terminated normally. The resulting FT1/LS1 files had multiple symptoms, including one discovered by the FSSC during transfer to them: one EVENT_ID was repeated ~1000 times. Scan of log files uncovered an additional five mergeClumps with these errors.
...
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="97ebf6eb7a8942a7-ba4ea1e5-40f845a5-b703bf99-1228e442a96adbe0c1e15892"><ac:plain-text-body><![CDATA[ | Data Product | destination | data content [1] | event selection [1] | makeFT1 | gtselect | gtdiffrsp | gtmktime | ]]></ac:plain-text-body></ac:structured-macro> |
---|---|---|---|---|---|---|---|---|---|
EXTENDEDFT1 | SLAC | FT1variables | ((FT1EventClass & 0x00003EFF)!=0) | | | | | ||
FT1 | FSSC+SLAC | FT1variables | 'source' and above | | | (inherited) | | ||
EXTENDEDLS1 | SLAC | LS1variables | ((FT1EventClass & 0x00003EFF)!=0) | | | | | ||
LS1 | FSSC+SLAC | LS1variables | 'transient' and above | | | (inherited) | | ||
ELECTRONFT1 | SLAC | FT1variables | CTBParticleType==1 | | | | |
...
- 8/8/2012 - Prepare task
- 8/18/2012 - 200 runs complete
- 8/25/2012 - Per Seth and Anders, add five runs and remove two:
- Add: 238421027, 238489647, 239108423, 239114152, 239208666
- Remove: 244395837, 244401823
(Note that removed runs have only been removed from dataCatalog – not from xroot)
But first, need FT2 files for these five runs!
- 10/19/2012
- P130-series FT2 files now available for early L&EO period
- The five runs above were never reprocessed; Seth blesses a new set of 64 runs to take their place
- Add 64 new runs to runList.txt (generate run list for entire L&EO period, then manually extract the new 64)
- Begin trickleStream
- 10/20/2012 262 runs complete (Original 200 - two 0244* runs + new 64). Note that only MERIT files for the two removed runs were removed from dataCatalog.
Configuration
Identical with P202-ROOT except for the list of runs to be processed...with one exception: to reprocess the four extra (out-of-order) L&EO runs, disable the event list sort.
...