status: Running
last update: 18 Apr 2014
** UNDER CONSTRUCTION **
This page is a record of the configuration and execution of the P301 reprocessing project, regeneration of MERIT and FITS files from the P300 MERIT using TMine.
...
Locations for P301:
Code Block |
---|
/glast/Data/Flight/Reprocess/P301/recon /glast/Data/Flight/Reprocess/P301/gcr /glast/Data/Flight/Reprocess/P301/relation /glast/Data/Flight/Reprocess/P301/merit /glast/Data/Flight/Reprocess/P301/filteredmerit /glast/Data/Flight/Reprocess/P301/electronmerit [soon] /glast/Data/Flight/Reprocess/P301/ft1 /glast/Data/Flight/Reprocess/P301/extendedft1 /glast/Data/Flight/Reprocess/P301/electronft1 /glast/Data/Flight/Reprocess/P301/ls1 /glast/Data/Flight/Reprocess/P301/extendedls1 |
...
Code Block |
---|
/glast/Data/Flight/Reprocess/P301/recon/r0239557414_v301_reconmerit.root |
DataCatalog location and naming
...
Code Block |
---|
Data/Flight/Reprocess/P301:RECONMERIT r0239557414 |
Data Sample
The currently defined data sample (as of 4/18/2014) for P301 reprocessing includes:
...
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P301
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
...
P301-
...
MERIT
Anchor |
---|
...
|
...
|
...
|
...
|
Status chronology
- 43/2327/2013 2014 - Begin setting up P300 task for Pass8 reprocessing.
- 5/10/2013 - Initial version of task running. See this page for infomation about the task creation, and this page for performance comparison with P202
- 7/2/2013 - after weeks of testing, final GR, jobOpts, etc. come together and production tentatively starts.
- Block 1 is defined with this run range: 239557414 through 392842073 (2008-08-04 15:43:37 through 2013-06-13 18:47:50 UTC)
- Update task to use the P202 generation of FT2 files (adds columns LAMBDA, RA_SUN, DEC_SUN, as well as meaningful BTIs)
- First 10 trial jobs launched. Too many ACD INFO messages 'Missed Poca for ID 602 at...' so modify jobOpts to disable.
- xroot disk space at the start of Pass8 = 508 TB (just after commissioning of fermi-xrd11)
- 7/29/2013 - a new GR is prepared (20-09-01) which fixes some problems in ACD tracking issues affecting ~32 runs so far (out of 5283). trickleStream disabled, wait for pending jobs to start, rollback other failed jobs, switch to new GR, rollback of ACD-failed jobs...
- This new GR continues to seg fault, so revert back to GR 20-09-00 and restart trickleStream.
- 7/31/2013 - A new GR (20-09-02) with patch for ACD tracking installed after stream 5970. At the time, there were 33 failed processClumps jobs. All rolled back.
- Last stream with GR 20-09-00 = 5970, run 274118943
- First stream with GR 20-09-02 = 5971, run 274124672
- Full list of rolled back substreams is here
- 10/15/2013 - Update job options to use the new "L1current" flavor of calibrations/alignment rather than "p7repro". First stream with new configuration: 20804.
- 11/14/2013 - Block 1 complete except for routine cleanup
11/21/2013 - Block 1 complete
Problems encountered include:
** pipeline_summary file corruption causing loss of email or of it being ignored, finally jobs terminate via reaper (Tony notified)
** large blocks of failures due to dataCatalog not returning desired FT2 files (Brian notified, fix possible by moving query to scriptlet)
** jobs running out the clock due to anomalously slow xroot behavior (Wilko/Andy notified)
** problems with individual batch machines (Renata notified, and I also have the authority to remove batch machines from production)
Summary of events processed in Block 1 from dataCatalog:Name Type Files Events Size Created (UTC) Links ELECTRONMERIT Group 26929 0 1.4 GB 09-May-2013 22:22:19 Files FILTEREDMERIT Group 26929 0 1.4 GB 09-May-2013 22:22:17 Files GCR Group 26929 58,653,424,708 1.0 TB 09-May-2013 22:22:18 Files MERIT Group 26929 58,653,424,708 98.3 TB 09-May-2013 22:22:18 Files RECON Group 26929 58,653,424,708 816.1 TB 09-May-2013 22:22:20 Files RELATION Group 26929 58,653,424,708 7.7 TB 09-May-2013 22:22:20 Files - P301 task for Pass8 re-MERIT.
Configuration
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P300P301-ROOTMERIT |
Task Status | |
GlastRelease | 20-09-00 streams 0 - 5970 (02 Jul 2013) |
Run Selection | based on a modified "standard" selection, see https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
TMineExt | 3.3.8 (17 Apr 2014) as used from $GLAST_EXT/TMineExt |
Classification worksheet | v0r16 from $GLAST_EXT/extFiles |
Run Selection | All MERIT files produced by P300-ROOT task |
s/c data | P202 FT2SECONDS which will eventually become a "standard" Public Release https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300P301-ROOTMERIT/config/runList.txt |
photonFilter | Not yet applicable, although this is defined by default: CTBParticleType==1 && ((FT1EventClass & 0x00003EFF)!=0) |
electronFilter | Not yet applicable, although this is defined by default: CTBParticleType==1 |
Code Variants used | redhat5-i686-32bit-gcc41 (Optimized), note that rhel5-64 and rhel6-64 GR builds are not yet available |
jobOpts | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300-ROOT/config/Pass8Recon.txt |
calibrations & alignments | "p7repro" and "L1current" (the latter to conform with new Level 1 flavor, and to include new ACD calibration for 9 Sep 2012, redhat5-x86_64-64bit-gcc41, redhat6-x86_64-64bit-gcc44 (Optimized) |
Output Data Products | RECON, GCR, RELATION, MERIT, FILTEREDMERIT, ELECTRONMERIT (but note that FILTEREDMERIT and ELECTRONMERIT are empty or contain junk) |
Timing and Scaling
...
For 20,000 event clumps:
job step
average CPU time
processClump
183 min
mergeClumps
10 min
- Plot of job throughput as of 20130813
From the plot, one can see a processing rate of ~208 runs/day. Assuming 28500 total runs, this would mean a repro time of 137 days or 4.5 months. - Another plot showing the number of run scratch directories cleaned up per day:
...
TMine
~1M evts/hour
P301-FITS
Anchor |
---|
...
|
...
|
This task generates all desired FITS data products.
NOTE: a temporary task called P300x-FITS has been created. It generates only EXTENDEDFT1 files and nothing else. There is no diffuse calculation performed. An untagged version of evtClassDefs is used which contains Matthew Wood's initial event classification and selections. This will likely be a template for a future production P300-FITS task. All data produced by this task are "throw-away" and not expected to survive beyond initial validation studies.
Status chronology
...
8/7/2013 - First year of data run through the task, amounting to 5537 runs and 11790391606 events:
First run | 239557417 | 2008-08-04 15:43:37 |
Last run | 271850279 | 2009-08-13 09:57:59 |
...
8/28/2013 - Second year of data ready and processed -> extendedFT1. Note that 36 troublesome runs omitted from 1st year data were included in this sample. There should be no missing runs in this sample.
First run | 239557417 | 2008-04-2008 15:43:37 |
Last run | 302647722 | 2010-08-04 20:48:40 |
10,976 runs, 2,677,747,829 events, 234.7 GB
...
9/25/2013 - Third year of data ready.
First run | 239557417 | 2008-04-2008 15:43:37 |
Last run | 334184989 | 2011-08-04 21:09:47 |
This block contains 16,516 runs and 36,056,385,847 events, or an increase over year 2 of 5,540 runs.
...
** THE FOLLOWING INFORMATION IS INCORRECT, DO NOT USE **
Status chronology
...
Name | Type | Files | Events | Size | Created (UTC) | Links |
---|---|---|---|---|---|---|
EXTENDEDFT1 | Group | 26929 | 7,149,992,671 | 626.6 GB | 01-Aug-2013 17:29:46 | Files |
Configuration
CAUTION: The following data describes the experimental P300x-FITS task, not the production task
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P300xP301-FITS |
Task Status | http://glast-ground.slac.stanford.edu/Pipeline-II/exp/Fermi/task.jsp?task=112641219 |
Input Data | MERIT (direct from P300-ROOT) |
spacecraft data | same as P300-ROOT |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300x-FITS/config/runList.txt |
evtClassDefs | untagged version from 20130801 |
eventClassMap | EvtClassDefs_P8.xml |
ScienceTools | 09-32-05 (7/30/2013) |
Code Variants | redhat5-x86_64-64bit-gcc41 & redhat6-x86_64-64bit-gcc44 (Optimized) |
Diffuse Model | based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 /v3r0 |
Diffuse Response | N/A |
IRFs | N/A |
Output Data Products |
...
Timing and Scaling
(no data)
...
This task is a clone of the P300-ROOT task with the exception of the run list which contains the 264 L&EO runs.
12/10/2013 09:10 begin reprocessing
12/17/2013 Processing complete. Each of the RECON, MERIT, GCR and RELATION files contain 613,490,351 events.
P300x-LEO-FITS
This is a clone of the P300x-FITS task with the exception of the run list which contains the 264 L&EO runs.
12/17/2013 begin processing