P130 Reprocessing
status: In Progress
last update: 3 February 2012
This page is a record of the configuration for the P130 reprocessing project, recalculating the diffuse response (see here for a description of the change). This task is identical with P120-FT1 [[please see] for details], run in July 2011, except for the following details.
- New version of ScienceTools (09-26-92) which includes new integration scheme for gtdiffrsp
- New run list (updated for data processed since the end of P120)
- Name of Pipeline task (P120->P130) as well as in file naming
Data Sample
=============== THIS SECTION NEEDS TO BE UPDATED FOR P130 ====================
The currently defined data sample for P120 reprocessing includes:
|First run|239557414 (MET), 2008-08-04 15:43:34 (UTC)|beginning of Science |
|Last run| 333880535 (MET), 2011-08-01 08:35:33 (UTC) | Official Pass7 release|
|Total runs | 16,459| |
|Total MERIT events|35,921,666,747|all "events"|
|Total FILTEREDMERIT/EXTENDEDFT1/LS1 events| 5,035,929,409|all photon event classes|
|Total ELECTRONMERIT/ELECTRONFT1 events| 68,055,849 | |
|Total LS1 (FSSC selection) events |1,025,359,231 |event classes (bits) 0,2,3,4 (transient, source, clean, ultraclean)|
|Total FT1 (FSSC selection) events | 142,042,060|event classes (bits) 2,3,4 (source, clean, ultraclean)|
|Total disk space used| 33.9 TB| |
Summary from DataCatalog as of 8/2/2011.
||Name || Files || Events || Size ||
|ELECTRONFT1 | 16459 | 68,055,849 | 6.4 GB |
|ELECTRONMERIT | 16459 | 68,055,849 | 147.4 GB |
|EXTENDEDFT1 | 16459 | 5,035,929,409 | 441.3 GB |
|EXTENDEDLS1 | 16459 | 5,035,929,409 | 816.5 GB |
|FILTEREDMERIT| 16459 | 5,035,929,409 | 4.0 TB |
|FT1 | 16459 |142,042,060 | 12.9 GB |
|LS1 | 16459 |1,025,359,231 | 166.6 GB |
|MERIT| 16459 |35,921,666,747 | 28.3 TB |
Bookkeeping
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P130
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
P130-FT1
This task generates all desired FITS data products. An example of the code processing chain appears on a child page.
Status chronology
- 1/26/2012 - Initial task configured, 1000 runs reprocessed. On hold for validation.
- 2/1/2012 - Okay to continue. Begin block 1 reprocessing, containing 19133 runs:
|
MET(sec) |
MJD(days) |
UNX(sec) |
UTC |
---|---|---|---|---|
1st run |
333886531 |
55774.427442129629 |
1312193731.000 |
2011-08-01 10:15:29 |
last run |
349266293 |
55952.433946759258 |
1327573493.000 |
2012-01-26 10:24:51 |
- 2/3/2012 - Block 1 reprocessing complete
Configuration
Task Location |
/nfs/farm/g/glast/u38/Reprocess-tasks/P130-FT1 |
Task Status |
http://glast-ground.slac.stanford.edu/Pipeline-II/task.jsp?task=74765664 |
Input Data |
MERIT (from P120-MERIT and Level 1) |
spacecraft data |
FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production |
Input Run List |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P130-FT1/config/runFile.txt |
Reprocessing Mode |
reFT1 |
meritFilter |
FT1EventClass!=0 |
evtClassDefs |
00-19-01 |
eventClassMap |
EvtClassDefs_P7V6.xml |
ScienceTools |
09-26-02 |
Code Variants |
redhat5-i686-32bit-gcc41 (Optimized) |
Diffuse Model |
based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 ) |
Diffuse Response |
'source' using P7SOURCE_V6 IRF |
IRFs |
P6V7, contained within ScienceTools release |
Output Data Products |
Processing chain for FITS data products
Data Product |
selection |
makeFT1 |
gtdiffrsp |
gtmktime |
gtltcube |
---|---|---|---|---|---|
FT1 (for FSSC) |
'source' and above |
||||
LS1 (for FSSC) |
'transient' and above |
||||
FT1EXTENDED |
FT1EventClass!=0 |
||||
LS1EXTENDED |
FT1EventClass!=0 |
||||
ELECTRONFT1 |
CTBParticleType==1 |
Note that diffuse response is calculated for 'source' and 'clean' event classes only.
Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.).
Timing and Performance
With a sample of the first 8128 runs, the CPU-intensive part of the task (mergeClumps) is consuming ~42 CPU-min/job (differences in machine class are small). TrickleStream configured to allow 1500 simultaneous mergeClumps jobs, and up to 150 job starts every 200 second cycle.