status: Running
last update: 01 Aug 2013
This page is a record of the configuration and execution of the P300 reprocessing project, full reprocessing from DIGIs using Pass8 analysis code.
Pipeline tasks
- P300-ROOT - This task reads DIGI and produces reprocessed RECON + GCR + MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT
- P300-FITS - This task (once it has been created) reads MERIT and produces FT1 (photons) + EXTENDEDFT1 + LS1 (merit-like FITS file for photons) + EXTENDEDLS1 + ELECTRONFITS file
- An experimental P300x-FITS task is being used for early validation studies. Data will not survive in the long run...
Datafile names, versions and locations
Data file version numbers for this reprocessing will begin with v300.
XROOT location and file naming
Location template:
/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>
Locations for P300:
/glast/Data/Flight/Reprocess/P300/recon /glast/Data/Flight/Reprocess/P300/gcr /glast/Data/Flight/Reprocess/P300/relation/glast/Data/Flight/Reprocess/P300/merit /glast/Data/Flight/Reprocess/P300/filteredmerit /glast/Data/Flight/Reprocess/P300/electronmerit /glast/Data/Flight/Reprocess/P300/ft1 /glast/Data/Flight/Reprocess/P300/extendedft1 /glast/Data/Flight/Reprocess/P300/electronft1 /glast/Data/Flight/Reprocess/P300/ls1 /glast/Data/Flight/Reprocess/P300/extendedls1
File naming:
Data Type |
aka |
Send to FSSC |
Naming template |
---|---|---|---|
RECON |
|
No |
r<run#>_<version>_<dataType>.root |
RELATION |
|
No |
r<run#>_<version>_<dataType>.root |
GCR |
|
No |
r<run#>_<version>_<dataType>.root |
MERIT |
|
No |
r<run#>_<version>_<dataType>.root |
FILTEREDMERIT |
|
No |
r<run#>_<version>_<dataType>.root |
ELECTRONMERIT |
|
No |
r<run#>_<version>_<dataType>.root |
ELECTRONFT1 |
|
No |
gll_el_p<procVer>_r<run#>_<version>.fit |
EXTENDEDFT1 |
|
No |
gll_xp_p<procVer>_r<run#>_<version>.fit |
FT1 |
LS-002 |
Yes |
gll_ph_p<procVer>_r<run#>_<version>.fit |
EXTENDEDLS1 |
|
No |
gll_xe_p<procVer>_r<run#>_<version>.fit |
LS1 |
LS-001 |
Yes |
gll_ev_p<procVer>_r<run#>_<version>.fit |
Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf
Example:
/glast/Data/Flight/Reprocess/P300/recon/r0239557414_v202_recon.root
DataCatalog location and naming
Logical directory and group template:
Data/Flight/Reprocess/<reprocessName>:<dataType>
Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.
Naming examples:
Data/Flight/Reprocess/P300:RECON r0239557414
Data Sample
The currently defined data sample (as of 7/2/2013) for P300 reprocessing includes:
First run |
239557414 (MET), 2008-08-04 15:43:34 (UTC) |
Last run |
present |
Bookkeeping
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P300
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
P300-ROOT
Status chronology
- 4/23/2013 - Begin setting up P300 task for Pass8 reprocessing.
- 5/10/2013 - Initial version of task running. See this page for infomation about the task creation, and this page for performance comparison with P202
- 7/2/2013 - after weeks of testing, final GR, jobOpts, etc. come together and production tentatively starts.
- block 1 is defined with this run range: 239557414 through 392842073 (2008-08-04 15:43:37 through 2013-06-13 18:47:50 UTC)
- Update task to use the P202 generation of FT2 files (adds columns LAMBDA, RA_SUN, DEC_SUN, as well as meaningful BTIs)
- First 10 trial jobs launched. Too many ACD INFO messages 'Missed Poca for ID 602 at...' so modify jobOpts to disable.
- xroot disk space at the start of Pass8 = 508 TB (just after commissioning of fermi-xrd11)
- 7/29/2013 - a new GR is prepared (20-09-01) which fixes some problems in ACD tracking issues affecting ~32 runs so far (out of 5283). trickleStream disabled, wait for pending jobs to start, rollback other failed jobs, switch to new GR, rollback of ACD-failed jobs...
- This new GR continues to seg fault, so revert back to GR 20-09-00 and restart trickleStream.
- 7/31/2013 - A new GR (20-09-02) with patch for ACD tracking installed after stream 5970. At the time, there were 33 failed processClumps jobs. All rolled back.
- Last stream with GR 20-09-00 = 5970, run 274118943
- First stream with GR 20-09-02 = 5971, run 274124672
- Full list of rolled back substreams is here
- 10/15/2013 - Update job options to use the new "L1current" flavor of calibrations/alignment rather than "p7repro".
Configuration
Task Location |
/nfs/farm/g/glast/u38/Reprocess-tasks/P300-ROOT |
Task Status |
|
GlastRelease |
20-09-00 streams 0 - 5970 (02 Jul 2013) |
Run Selection |
based on a modified "standard" selection, see https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
s/c data |
P202 FT2SECONDS which will eventually become a "standard" Public Release https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
Input Run List |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300-ROOT/config/runList.txt |
photonFilter |
Not yet applicable, although this is defined by default: CTBParticleType==1 && ((FT1EventClass & 0x00003EFF)!=0) |
electronFilter |
Not yet applicable, although this is defined by default: CTBParticleType==1 |
Code Variants |
redhat5-i686-32bit-gcc41 (Optimized), note that rhel5-64 and rhel6-64 GR builds are not yet available |
jobOpts |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300-ROOT/config/Pass8Recon.txt |
calibrations & alignments |
"p7repro" and "L1current" (the latter to conform with new Level 1 flavor, and to include new ACD calibration for 9 Sep 2012) |
Output Data Products |
RECON, GCR, RELATION, MERIT, FILTEREDMERIT, ELECTRONMERIT (but note that FILTEREDMERIT and ELECTRONMERIT are empty or contain junk) |
Timing and Scaling
- Current performance results for the Pass8 code are on this page .
- For 20,000 event clumps:
job step
average CPU time
processClump
183 min
mergeClumps
10 min
- Plot of job throughput as of 20130813
From the plot, one can see a processing rate of ~208 runs/day. Assuming 28500 total runs, this would mean a repro time of 137 days or 4.5 months. - Another plot showing the number of run scratch directories cleaned up per day:
P300x-FITS
This task generates all desired FITS data products.
NOTE: a temporary task called P300x-FITS has been created. It generates only EXTENDEDFT1 files and nothing else. There is no diffuse calculation performed. An untagged version of evtClassDefs is used which contains Matthew Wood's initial event classification and selections. This will likely be a template for a future production P300-FITS task. All data produced by this task are "throw-away" and not expected to survive beyond initial validation studies.
Status chronology
- 8/1/2013 - Initial P300x-FITS task created. 10 trial runs processed. (See caveats above.)
- 8/7/2013 - First year of data run through the task, amounting to 5537 runs and 11790391606 events:
First run
239557417
2008-08-04 15:43:37
Last run
271850279
2009-08-13 09:57:59
- 8/28/2013 - Second year of data ready and processed -> extendedFT1. Note that 36 troublesome runs omitted from 1st year data were included in this sample. There should be no missing runs in this sample.
10,976 runs, 2,677,747,829 events, 234.7 GB
First run
239557417
2008-04-2008 15:43:37
Last run
302647722
2010-08-04 20:48:40
- 9/25/2013 - Third year of data ready.
This block contains 16,516 runs and 36,056,385,847 events, or an increase over year 2 of 5,540 runs.
First run
239557417
2008-04-2008 15:43:37
Last run
334184989
2011-08-04 21:09:47
Configuration
CAUTION: The following data describes the experimental P300x-FITS task, not the production task
Task Location |
/nfs/farm/g/glast/u38/Reprocess-tasks/P300x-FITS |
Task Status |
http://glast-ground.slac.stanford.edu/Pipeline-II/exp/Fermi/task.jsp?task=112641219 |
Input Data |
MERIT (direct from P300-ROOT) |
spacecraft data |
same as P300-ROOT |
Input Run List |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300x-FITS/config/runList.txt |
evtClassDefs |
untagged version from 20130801 |
eventClassMap |
EvtClassDefs_P8.xml |
ScienceTools |
09-32-05 (7/30/2013) |
Code Variants |
redhat5-x86_64-64bit-gcc41 & redhat6-x86_64-64bit-gcc44 (Optimized) |
Diffuse Model |
based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 /v3r0 |
Diffuse Response |
N/A |
IRFs |
N/A |
Output Data Products |
WARNING: THIS NEXT SECTION IS OBSOLETE
Generation of output data products:
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="baf89d82-5fad-477b-8de3-ce9097fb9347"><ac:plain-text-body><![CDATA[ |
Data Product |
destination |
data content [1] |
event selection [1] |
makeFT1 |
gtselect |
gtdiffrsp |
gtmktime |
]]></ac:plain-text-body></ac:structured-macro> |
---|---|---|---|---|---|---|---|---|---|
EXTENDEDFT1 |
SLAC |
FT1variables |
((FT1EventClass & 0x00003EFF)!=0) |
|
|
|
|
||
FT1 |
FSSC+SLAC |
FT1variables |
'source' and above |
|
|
(inherited) |
|
||
EXTENDEDLS1 |
SLAC |
LS1variables |
((FT1EventClass & 0x00003EFF)!=0) |
|
|
|
|
||
LS1 |
FSSC+SLAC |
LS1variables |
'transient' and above |
|
|
(inherited) |
|
||
ELECTRONFT1 |
SLAC |
FT1variables |
CTBParticleType==1 |
|
|
|
|
[1] /afs/slac/g/glast/ground/releases/volume04/evtClassDefs/00-19-04/data
Note that diffuse response is calculated for 'source' and 'clean' event classes only.
Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL5-32 (RHEL6-64 are built but not yet validated), while ScienceTools builds for RHEL5-64, RHEL5-64.