P118 Reprocessing
status: Block 1 Complete
last update: 23 Feb 25 May 2011
This page is a record of the configuration for the P118 reprocessing project, recalculating diffuse response with new P6_V11 IRFs and adding to existing FT1 files. This processing will be applied to all science data, be incorporated into Level 1 processing, and results sent to the FSSC.
...
Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD (.doc, or PDF) on 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_ICD_RevA.pdf
Example:
Code Block |
---|
/glast/Data/Flight/Reprocess/P118/ft1/gll_ph_p116p118_r0239559565_v116v118.fit /glast/Data/Flight/Reprocess/P118/ls1/gll_ev_p116p118_r0239559565_v116v118.fit |
DataCatalog location and naming
...
Block | First run | First run | Last run | Last run | #Runs | #Evts | DataCatalog Source | Note | |
---|---|---|---|---|---|---|---|---|---|
1 | 239557414 | 2008-08-04 15:43:34 | 311669624 | 2010-11-17 06:53:42 | 12557 | 435,741,248 | /Data/Flight/Reprocess/P116/FT1 | entire P116 dataset | |
1 | 311675447 | 2010-11-17 08:30:45 | 319652280 | 2011-02-17 16:17:58 | 1410 | 55,911,339 | /Data/Flight/Level1/LPA/ | Standard Level 1 output | |
2 | 319658687 | 2011-02-17 18:04:45 | 324849509 | 2011-04-18 19:58:27 | 908 | 45,004,526 | /Data/Flight/Level1/LPA/ | Standard Level 1 output | |
3 | 324855475 | 2011-04-18 21:37:53 | 327999656 | 2011-05-25 07:00:54 | 552 | 22,469,614 | /Data/Flight/Level1/LPA/ | Standard Level 1 output | |
TOTAL |
|
|
|
|
| 15427 | 559,126,727 |
|
|
Final number of selected photon events (in the FT1 and LS1 files) = TBD
...
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P118
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
P118-FT1
Anchor | ||||
---|---|---|---|---|
|
(This task is, in configuration, equivalent to the P116-FT1 task but with , bypassing all but the gtdiffrsp step bypassed.)
Status chronology
5/25/2011 - Begin and complete reprocessing block3, 552 runs (324855475 through 327999656)
start 09:30, complete 11:40
4/20/2011 - Begin and complete reprocessing block2, 908 runs (319658690 through 324849509)
start 11:30, complete 14:00
3/24/2011 - Eight runs re-reprocessed due to updated FT1 files from Level 1:
Code Block |
---|
316703389
316739248
316744661
316761944
316767672
316773400
316776949
318795366
|
2/23/2011 - Block 1 complete
2/22/2011 - Begin trickleStream of block1
2/15/2011 - Begin configuration for this task
...
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P118-FT1 |
Task Status | |
Input Data Selection | FT1 |
Input Run List | http://glast-ground.slac.stanford.edu/Decorator/Decorate/u38/Reprocess-tasks/P118-FT1/config/runFile.txt |
Reprocessing Mode | doDiffRsp |
evtClassDefs | n/a (see P116) |
photon cut | n/a (see P116) |
eventClassifier | n/a (see P116) |
s/c data | FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production |
ScienceTools | 09-21-00 (SCons build) |
Allowed Code Variants | redhat4-i686-32bit-gcc34, redhat4-x86_64-64bit-gcc34, |
Diffuse Model | /afs/slac/g/glast/groups/diffuse/mapcubes/source_model_v02_P6_V11_DIFFUSE.xml ) |
Diffuse Response IRFs | P6_V11_DIFFUSE |
Output Data Products |
...
The execution of gtdiffrsp happens in the mergeClumps step and takes ~5 hequ-minutes per run (on average) to complete. Some individual runs (few percent) take considerably longer...up to 2 hours.
(update 5/25/2011) Statistics for mergeClump now show mean time of 14.4 CPU minutes (averaged across all CPU types, but mostly hequ and boer)