P130 Reprocessing
status: In Development
last update: 25 January 2012
This page is a record of the configuration for the P130 reprocessing project, recalculating the diffuse response (see here for a description of the change). This task is identical with P120-FT1, run in July 2011, except for updating the version of ScienceTools to reflect the change in gtdiffrsp.
=============== WHAT FOLLOWS IS TO BE UPDATED ====================
The name "P120" derives from the word "processing" and the initial file version to be used for the output data products, e.g., r0123456789_v120_merit.root.
- P120-MERIT - this task reads DIGI+RECON+MERIT and produces reprocessed MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT
- P120-LEO-MERIT - this task reads DIGI+RECON+MERIT and produces reprocessed MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT for 200 runs of earth limb (L&EO) data
Datafile names, versions and locations
Data file version numbers for this reprocessing will begin with v120.
XROOT location and file naming
Location template:
/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>
Locations for P120:
/glast/Data/Flight/Reprocess/P120/merit /glast/Data/Flight/Reprocess/P120/filteredmerit /glast/Data/Flight/Reprocess/P120/electronmerit /glast/Data/Flight/Reprocess/P120/ft1 /glast/Data/Flight/Reprocess/P120/extendedft1 /glast/Data/Flight/Reprocess/P120/electronft1 /glast/Data/Flight/Reprocess/P120/ls1 /glast/Data/Flight/Reprocess/P120/extendedls1
File naming:
Data Type |
aka |
Send to FSSC |
Naming template |
---|---|---|---|
MERIT |
|
No |
r<run#>_<version>_<dataType>.root |
FILTEREDMERIT |
|
No |
r<run#>_<version>_<dataType>.root |
ELECTRONMERIT |
|
No |
r<run#>_<version>_<dataType>.root |
ELECTRONFT1 |
|
No |
gll_el_p<procVer>_r<run#>_<version>.fit |
EXTENDEDFT1 |
|
No |
gll_xp_p<procVer>_r<run#>_<version>.fit |
FT1 |
LS-002 |
Yes |
gll_ph_p<procVer>_r<run#>_<version>.fit |
EXTENDEDLS1 |
|
No |
gll_xe_p<procVer>_r<run#>_<version>.fit |
LS1 |
LS-001 |
Yes |
gll_ev_p<procVer>_r<run#>_<version>.fit |
Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf
Example:
/glast/Data/Flight/Reprocess/P120/merit/r0239557414_v120_merit.root /glast/Data/Flight/Reprocess/P120/filteredmerit/r0239557414_v120_filteredmerit.root /glast/Data/Flight/Reprocess/P120/electronmerit/r0239557414_v120_electronmerit.root /glast/Data/Flight/Reprocess/P120/extendedft1/gll_xp_p120_r0239559565_v120.fit /glast/Data/Flight/Reprocess/P120/ft1/gll_ph_p120_r0239559565_v120.fit /glast/Data/Flight/Reprocess/P120/electronft1/gll_el_p120_r0239559565_v120.fit /glast/Data/Flight/Reprocess/P120/extendedls1/gll_xe_p120_r0239559565_v120.fit /glast/Data/Flight/Reprocess/P120/ls1/gll_ev_p120_r0239559565_v120.fit
DataCatalog location and naming
Logical directory and group template:
Data/Flight/Reprocess/<reprocessName>:<dataType>
Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.
Naming examples:
Data/Flight/Reprocess/P120:MERIT r0239557414 Data/Flight/Reprocess/P120:FILTEREDMERIT r0239557414 Data/Flight/Reprocess/P120:EXTENDEDFT1 r0239557414 Data/Flight/Reprocess/P120:FT1 r0239557414 Data/Flight/Reprocess/P120:ELECTRONFT1 r0239557414 Data/Flight/Reprocess/P120:EXTENDEDLS1 r0239557414 Data/Flight/Reprocess/P120:LS1 r0239557414
Data Sample
The currently defined data sample for P120 reprocessing includes:
First run |
239557414 (MET), 2008-08-04 15:43:34 (UTC) |
beginning of Science |
Last run |
333880535 (MET), 2011-08-01 08:35:33 (UTC) |
Official Pass7 release |
Total runs |
16,459 |
|
Total MERIT events |
35,921,666,747 |
all "events" |
Total FILTEREDMERIT/EXTENDEDFT1/LS1 events |
5,035,929,409 |
all photon event classes |
Total ELECTRONMERIT/ELECTRONFT1 events |
68,055,849 |
|
Total LS1 (FSSC selection) events |
1,025,359,231 |
event classes (bits) 0,2,3,4 (transient, source, clean, ultraclean) |
Total FT1 (FSSC selection) events |
142,042,060 |
event classes (bits) 2,3,4 (source, clean, ultraclean) |
Total disk space used |
33.9 TB |
|
Summary from DataCatalog as of 8/2/2011.
Name |
Files |
Events |
Size |
---|---|---|---|
ELECTRONFT1 |
16459 |
68,055,849 |
6.4 GB |
ELECTRONMERIT |
16459 |
68,055,849 |
147.4 GB |
EXTENDEDFT1 |
16459 |
5,035,929,409 |
441.3 GB |
EXTENDEDLS1 |
16459 |
5,035,929,409 |
816.5 GB |
FILTEREDMERIT |
16459 |
5,035,929,409 |
4.0 TB |
FT1 |
16459 |
142,042,060 |
12.9 GB |
LS1 |
16459 |
1,025,359,231 |
166.6 GB |
MERIT |
16459 |
35,921,666,747 |
28.3 TB |
NOTE: One run, 242429468, of type TrigTest was declared 'good for science' but long after this task got started, so it has been intentionally omitted.
8/17/2011 update: Four orphaned runs are being reprocessed, including one TrigTest run and four nadir-pointed runs.
Stream |
run |
type |
---|---|---|
16459 |
242429468 |
(TrigTest) |
16460 |
333355876 |
(nadirOps) |
16461 |
333358500 |
(nadirOps) |
16462 |
333365716 |
(nadirOps) |
Bookkeeping
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P120
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
P120-FT1
This task generates all desired FITS data products. An example of the code processing chain appears on a child page.
Status chronology
Configuration (version 2)
Task Location |
/nfs/farm/g/glast/u38/Reprocess-tasks/P120-FT1 |
Task Status |
http://glast-ground.slac.stanford.edu/Pipeline-II/task.jsp?task=63390236 |
Input Data |
MERIT (from P120-MERIT) |
spacecraft data |
FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production |
Input Run List |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P120-FT1/config/runFile.txt |
Reprocessing Mode |
reFT1 |
meritFilter |
FT1EventClass!=0 |
evtClassDefs |
00-19-01 |
eventClassMap |
EvtClassDefs_P7V6.xml |
ScienceTools |
09-24-00 |
Code Variants |
redhat5-i686-32bit-gcc41 (Optimized) |
Diffuse Model |
based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 ) |
Diffuse Response |
'source' using P7SOURCE_V6 IRF |
IRFs |
P6V7, contained within ScienceTools release |
Output Data Products |
Processing chain for FITS data products
Data Product |
selection |
makeFT1 |
gtdiffrsp |
gtmktime |
gtltcube |
---|---|---|---|---|---|
FT1 |
'source' and above |
true |
true |
true |
false |
LS1 |
'transient' and above |
true |
true |
true |
false |
FT1EXTENDED |
FT1EventClass!=0 |
true |
true |
true |
false |
LS1EXTENDED |
FT1EventClass!=0 |
true |
true |
true |
false |
ELECTRONFT1 |
CTBParticleType==1 |
true |
false |
true |
false |
Note that diffuse response is calculated for 'source' and 'clean' event classes only.
Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.).