status: Complete
last update: 17 August 2011
This page is a record of the configuration for the P120 reprocessing project, event reclassification using Pass 7.3/7.4/7.6. This project involves reprocessing with Pass7 classification trees and (ultimately) new IRFs. The name "P120" derives from the word "processing" and the initial file version to be used for the output data products, e.g., r0123456789_v120_merit.root.
Data file version numbers for this reprocessing will begin with v120.
Location template:
/glast/Data/Flight/Reprocess/<reprocessName>/<dataType> |
Locations for P120:
/glast/Data/Flight/Reprocess/P120/merit /glast/Data/Flight/Reprocess/P120/filteredmerit /glast/Data/Flight/Reprocess/P120/electronmerit /glast/Data/Flight/Reprocess/P120/ft1 /glast/Data/Flight/Reprocess/P120/extendedft1 /glast/Data/Flight/Reprocess/P120/electronft1 /glast/Data/Flight/Reprocess/P120/ls1 /glast/Data/Flight/Reprocess/P120/extendedls1 |
File naming:
Data Type |
aka |
Send to FSSC |
Naming template |
---|---|---|---|
MERIT |
|
No |
r<run#>_<version>_<dataType>.root |
FILTEREDMERIT |
|
No |
r<run#>_<version>_<dataType>.root |
ELECTRONMERIT |
|
No |
r<run#>_<version>_<dataType>.root |
ELECTRONFT1 |
|
No |
gll_el_p<procVer>_r<run#>_<version>.fit |
EXTENDEDFT1 |
|
No |
gll_xp_p<procVer>_r<run#>_<version>.fit |
FT1 |
LS-002 |
Yes |
gll_ph_p<procVer>_r<run#>_<version>.fit |
EXTENDEDLS1 |
|
No |
gll_xe_p<procVer>_r<run#>_<version>.fit |
LS1 |
LS-001 |
Yes |
gll_ev_p<procVer>_r<run#>_<version>.fit |
Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf
Example:
/glast/Data/Flight/Reprocess/P120/merit/r0239557414_v120_merit.root /glast/Data/Flight/Reprocess/P120/filteredmerit/r0239557414_v120_filteredmerit.root /glast/Data/Flight/Reprocess/P120/electronmerit/r0239557414_v120_electronmerit.root /glast/Data/Flight/Reprocess/P120/extendedft1/gll_xp_p120_r0239559565_v120.fit /glast/Data/Flight/Reprocess/P120/ft1/gll_ph_p120_r0239559565_v120.fit /glast/Data/Flight/Reprocess/P120/electronft1/gll_el_p120_r0239559565_v120.fit /glast/Data/Flight/Reprocess/P120/extendedls1/gll_xe_p120_r0239559565_v120.fit /glast/Data/Flight/Reprocess/P120/ls1/gll_ev_p120_r0239559565_v120.fit |
Logical directory and group template:
Data/Flight/Reprocess/<reprocessName>:<dataType> |
Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.
Naming examples:
Data/Flight/Reprocess/P120:MERIT r0239557414 Data/Flight/Reprocess/P120:FILTEREDMERIT r0239557414 Data/Flight/Reprocess/P120:EXTENDEDFT1 r0239557414 Data/Flight/Reprocess/P120:FT1 r0239557414 Data/Flight/Reprocess/P120:ELECTRONFT1 r0239557414 Data/Flight/Reprocess/P120:EXTENDEDLS1 r0239557414 Data/Flight/Reprocess/P120:LS1 r0239557414 |
The currently defined data sample for P120 reprocessing includes:
First run |
239557414 (MET), 2008-08-04 15:43:34 (UTC) |
beginning of Science |
Last run |
333880535 (MET), 2011-08-01 08:35:33 (UTC) |
Official Pass7 release |
Total runs |
16,459 |
|
Total MERIT events |
35,921,666,747 |
all "events" |
Total FILTEREDMERIT/EXTENDEDFT1/LS1 events |
5,035,929,409 |
all photon event classes |
Total ELECTRONMERIT/ELECTRONFT1 events |
68,055,849 |
|
Total LS1 (FSSC selection) events |
1,025,359,231 |
event classes (bits) 0,2,3,4 (transient, source, clean, ultraclean) |
Total FT1 (FSSC selection) events |
142,042,060 |
event classes (bits) 2,3,4 (source, clean, ultraclean) |
Total disk space used |
33.9 TB |
|
Summary from DataCatalog as of 8/2/2011.
Name |
Files |
Events |
Size |
---|---|---|---|
ELECTRONFT1 |
16459 |
68,055,849 |
6.4 GB |
ELECTRONMERIT |
16459 |
68,055,849 |
147.4 GB |
EXTENDEDFT1 |
16459 |
5,035,929,409 |
441.3 GB |
EXTENDEDLS1 |
16459 |
5,035,929,409 |
816.5 GB |
FILTEREDMERIT |
16459 |
5,035,929,409 |
4.0 TB |
FT1 |
16459 |
142,042,060 |
12.9 GB |
LS1 |
16459 |
1,025,359,231 |
166.6 GB |
MERIT |
16459 |
35,921,666,747 |
28.3 TB |
NOTE: One run, 242429468, of type TrigTest was declared 'good for science' but long after this task got started, so it has been intentionally omitted.
8/17/2011 update: Four orphaned runs are being reprocessed, including one TrigTest run and four nadir-pointed runs.
Stream |
run |
type |
---|---|---|
16459 |
242429468 |
(TrigTest) |
16460 |
333355876 |
(nadirOps) |
16461 |
333358500 |
(nadirOps) |
16462 |
333365716 |
(nadirOps) |
stream |
run |
UTC |
Crash location |
---|---|---|---|
16136 |
332054399 |
2011-07-11 05:19:57 |
Filtered Merit skim failure |
16151 |
332140182 |
2011-07-12 05:09:40 |
Electron Merit skim failure |
16156 |
332169056 |
2011-07-12 13:10:54 |
Filtered Merit skim failure |
Package |
Old Version |
New Version |
Reason for change |
---|---|---|---|
GPLtools |
v1r15p1 |
GPLtools-02-00-00 |
Checks size of files before and after move between disk and xroot |
skimmer |
07-07-00 |
08-01-00 |
detects failure to open input file |
run |
task stream |
# subStreams |
---|---|---|
306353950 |
11624 |
10 |
316611240 |
13431 |
8 |
320850543 |
14171 |
7 |
306353950 |
'bad chunk' (known previously) - awaiting GR update |
313483912 |
missing 700s, being worked on |
316611240 |
'bad chunk' - awaiting GR update |
Task Location |
/nfs/farm/g/glast/u38/Reprocess-tasks/P120-MERIT |
Task Status |
http://glast-ground.slac.stanford.edu/Pipeline-II/exp/Fermi/task.jsp?task=41146114 |
GlastRelease |
v17r35p8 v17r35p10 |
Input Data Selection |
"standard" from https://confluence.slac.stanford.edu/display/SCIGRPS/LAT+Dataset+Definitionsalong with "&& (RunQuality != "Bad" || is_null ( RunQuality )" |
s/c data |
FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production |
Input Run List |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P120-MERIT/config/runFile.txt |
photonFilter |
CTBParticleType==0 && CTBClassLevel>0 |
electronFilter |
CTBParticleType==1 |
jobOpts |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P120-MERIT/config/reClassify.txt |
Output Data Products |
Introduce new trickleStreams.py script to (partially) assess pipeline activity and only the number of jobs considered appropriate based on available data.
(block 1)
maxProcessClumps = 600 ## prevent overload of xroot maxMergeClumps = 20 ## prevent overload of xroot (inactive) maxStreamsPerCycle = 20 ## prevent overload of /u30 on startup timePerCycle = 900 ## 15 minutes: allow time for dust to settle |
With these parameters, it took ~ 5 hours to reach a point where fewer than 20 jobs per cycle were regularly submitted. Another 4.5 hours for the task to complete. On average, one run generated 7.5 processClump batch jobs.
For subsequent data (beyond block 2), xroot displayed such stress, that the maxProcessClumps limit was reduced to 250 or 300.
This task generates all desired FITS data products. An example of the code processing chain appears on a child page.
Task Location |
/nfs/farm/g/glast/u38/Reprocess-tasks/P120-FT1 |
Task Status |
http://glast-ground.slac.stanford.edu/Pipeline-II/exp/Fermi/task.jsp?task=65047878 |
Input Data |
MERIT (from P120-MERIT) |
spacecraft data |
FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production |
Input Run List |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P120-FT1/config/runFile.txt |
Reprocessing Mode |
reFT1 |
meritFilter |
FT1EventClass!=0 |
evtClassDefs |
00-19-01 |
eventClassMap |
EvtClassDefs_P7V6.xml |
ScienceTools |
09-24-00 |
Code Variants |
redhat5-i686-32bit-gcc41 (Optimized) |
Diffuse Model |
based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 ) |
Diffuse Response |
'source' using P7SOURCE_V6 IRF |
IRFs |
P6V7, contained within ScienceTools release |
Output Data Products |
Processing chain for FITS data products
Data Product |
selection |
makeFT1 |
gtdiffrsp |
gtmktime |
gtltcube |
---|---|---|---|---|---|
FT1 |
'source' and above |
true |
true |
true |
false |
LS1 |
'transient' and above |
true |
true |
true |
false |
FT1EXTENDED |
FT1EventClass!=0 |
true |
true |
true |
false |
LS1EXTENDED |
FT1EventClass!=0 |
true |
true |
true |
false |
ELECTRONFT1 |
CTBParticleType==1 |
true |
false |
true |
false |
Note that diffuse response is calculated for 'source' and 'clean' event classes only.
Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.).
Identical to the P120-MERIT task, except use FT2 files from P110 reprocessing.