...
- 3/2/2012 - Define block 1 as the 776 runs in P202-ROOT block 1. Configure trickleStream and begin (14:08)
- 3/31/2012 - Define block 2 as 5600 runs. Reconfig trickleStream and begin (18:05)
- 4/01/2012 - Block 2 complete (most of the 4824 jobs completed in about six hours w/1000 job limit).
- 5/31/2012
- Discover stream 5599 (run 271999199) requires rollback - new MERIT file version (v203->v205).
- Regenerate runlist with 20,229 input MERIT files.
- Minor config change: twoClumpMin=False (formerly True, but caused unnecessary extra processClump dummy jobs)
- 6/5/2012 - Final cleanup
- Five runs are responsible for discrepant event tallies:
Run
Stream
Reason
Action
239557414
0
bad TCut
rollback ROOT + FITS
241599746
352
bad processing order
rollback FITS
245403855
1019
bad start time
change start time in runList and rollback
332661583
16244
silent root errors
rollback ROOT
339081502
17416
silent root errors
rollback ROOT
- Code changes:
directory
script
modification
commonTools/00-01-00
repTools.py
added new getKey() function to extract #events from FITS event files
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="a2a8f3e7fd93544c-66083517-42f84387-bd6d87c8-c7e14e00c829b4fb73306ac0"><ac:plain-text-body><![CDATA[
P202-FITS/config
config.py
added os.environ['HEADASNOQUERY']='true' to enable 'ftlist' to run in batch
]]></ac:plain-text-body></ac:structured-macro>
P202-FITS/config
makeFITS.py
add diagnostic print of #evts in FITS files for each processing sub-step
- After these actions, the dataCatalog tallies are now consistent: Note that the number of events in ELECTRONFT1 files are not currently tallied by the dataCatalog.
Name
Type
Files
Events
Size
Created (UTC)
Links
Group
20229
0
8.5 GB
02-Mar-2012 00:06:07
Group
20229
90,904,582
205.7 GB
25-Jan-2012 00:53:32
Group
20229
189,323,074
17.8 GB
02-Mar-2012 00:06:06
Group
20229
1,325,204,821
215.3 GB
02-Mar-2012 00:06:08
Group
20229
6,291,424,926
574.7 GB
02-Mar-2012 00:06:09
Group
20229
6,291,424,926
1,020.1 GB
02-Mar-2012 00:06:09
Group
20229
6,291,424,926
5.3 TB
25-Jan-2012 00:53:29
Group
20229
44,125,679,961
35.4 TB
25-Jan-2012 00:53:30
Group
20229
44,125,679,961
590.0 TB
25-Jan-2012 00:53:33
Group
20229
44,125,679,961
942.7 GB
25-Jan-2012 00:53:31
Group
20229
44,125,679,961
128.7 TB
25-Jan-2012 00:53:31
- Five runs are responsible for discrepant event tallies:
- 8/24/2012 - Configure and run backfill through 31 July 2012
- 10/7/2012 - Rollback the following seven streams to fix corrupt FITS files. The makeFT1 app received a large number of ROOT error, but terminated normally. The resulting FT1/LS1 files had multiple symptoms, including one discovered by the FSSC during transfer to them: one EVENT_ID was repeated ~1000 times. Scan of log files uncovered an additional five mergeClumps with these errors.
Code Block %INFO: 20120331:20:34:32 - makeFITS(run)/line-127 - Running: makeFT1 ---------------- start commentary ---------------- About to run [time makeFT1 rootFile=/scratch/glastmp/P202-FITS/3345/r0259101994_v202_merit.root fitsFile=/scratch/glastmp/P202-FITS/3345/gll_xp_p202_r0259101994_v202.fit TCuts=/afs/slac.stanford.edu/g/glast/ground/releases//volume04/evtClassDefs/00-19-04/data/pass7.6_Extended_cuts_L1 dict_file=/afs/slac.stanford.edu/g/glast/ground/releases//volume04/evtClassDefs/00-19-04/data/FT1variables tstart=259101996.933000 tstop=259106272.085000 file_version=202 tempRootFile=/scratch/glastmp/P202-FITS/3345/dummy.root xml_classifier=/afs/slac.stanford.edu/g/glast/ground/releases//volume04/evtClassDefs/00-19-04/xml/EvtClassDefs_P7V6.xml evtclsmap=FT1EventClass chatter=4 debug=yes] at Sat Mar 31 20:34:32 2012 ---------------- start log ---------------- This is makeFT1 version ScienceTools-09-27-01 applying TCut: ((FT1EventClass & 0x00003EFF)!=0) && (EvtElapsedTime >= 259101996) && (EvtElapsedTime <= 259106273) Warning in <TClass::TClass>: no dictionary for class FileHeader is available Warning in <TClass::TClass>: no dictionary for class RootObj<int> is available R__unzip: error in header Error in <TBasket::ReadBasketBuffers>: fNbytes = 15550, fKeylen = 86, fObjlen = 31912, noutot = 0, nout=0, nin=11010118, nbuf=7864444 Error in <TBranch::GetBasket>: File: /scratch/glastmp/P202-FITS/3345/r0259101994_v202_merit.root at byte:459100837, branch:Vtx2LongDoca, entry:510338, badread=0, nerrors=1, basketnumber=1 [...]
Stream
Run
3345
259101994
<- found by FSSC
4122
263571912
4707
266893978
13927
319436826
16181
332306548
<- found by FSSC
17430
339161346
17479
339408141
- 10/15/2012 - Reconfigure for block 4 backfill and begin trickleStream.
- Discover run 22240 makeFT1 contains error messages as the mentioned last week (above). Rolling back the entire chain of processing, starting with P202-ROOT seemed to do the trick.
- 10/17/2012 - block4 backfill complete.
- 12/18/2012 - Prepare for block 5. Before:
Name
Type
Files
Events
Size
EXTENDEDFT1
Group
23141
7,252,996,601
662.6 GB
EXTENDEDLS1
Group
23141
7,252,996,601
1.1 TB
FT1
Group
23141
229,126,932
21.5 GB
LS1
Group
23141
1,549,133,207
251.7 GB
- 12/19/2012 - block5 backfill complete. Final DataCatalog numbers:
Name
Type
Files
Events
Size
EXTENDEDFT1
Group
24142
7,578,468,156
692.3 GB
EXTENDEDLS1
Group
24142
7,578,468,156
1.2 TB
FT1
Group
24142
240,949,332
22.6 GB
LS1
Group
24142
1,621,773,636
263.5 GB
- 3/6/2013 - Many changes!
- New task created (version 0.9 -> 1.0)
- Update to run native on RHEL5-64 and RHEL6-64
- New Interstellar Emission Model, v2r0 -> v3r0, provided by Luigi Tibaldo.
- Update ScienceTools from 09-27-01 to 09-31-01
- Update IRFS from P7*V6 to P7REP_V10 ( = CLEAN or SOURCE)
- This means that rather than rolling-back, the task starts from scratch.
- New evtClassDiffs version: 00-19-05, which changes P7V6 -> P7REP, the new designation.
- Reference list of all existing FITS files (with old diffuse model) generated and stored here: /nfs/farm/g/glast/u38/Reprocess-tasks/P202-FITS/config/task-v0.9
Code Block diffmodel_p7rep_clean_v10_reduced.xml diffmodel_p7rep_clean_v10.xml diffmodel_p7rep_source_v10_reduced.xml diffmodel_p7rep_source_v10.xml gal_p7rep_v10_v1.fits gal_p7rep_v10_v1_reduced.fits iso_p7rep_clean_v10_back_v1.txt iso_p7rep_clean_v10_front_v1.txt iso_p7rep_clean_v10_v1.txt iso_p7rep_source_v10_back_v1.txt iso_p7rep_source_v10_front_v1.txt iso_p7rep_source_v10_v1.txt
...
- 5/30/2013 - Green light given on new IRFs and diffuse model.
- Changes to config.py:
- New diffuse model: /afs/slac/g/glast/ground/GLAST_EXT/diffuseModels/v4r0
- New IRFs: P7REP_SOURCE_V15,P7REP_CLEAN_V15
- Awaiting a new ScienceTools release...
- Changes to config.py:
- 6/10/2013 - Several ST updates, and renaming of the diffuse model files. All appears good to go.
...
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P202-FITS |
Task Status | http://glast-ground.slac.stanford.edu/Pipeline-II/task.jsp?task=75031156 |
Input Data | MERIT (direct from P202-ROOT) |
spacecraft data | same as P202-ROOT |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-FITS/config/runList.txt |
evtClassDefs | 00-19-05 (March 2013, changed pass_ver to P7REP) |
eventClassMap | EvtClassDefs_P7V6.xml |
ScienceTools | 09-32-03 (6/7/2013) |
Code Variants | redhat5-x86_64-64bit-gcc41, redhat6-x86_64-64bit-gcc44 (Optimized) |
Diffuse Model | based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 /v3r0 v4r0 |
Diffuse Response | 'source' using P7SOURCE_V6 P7REP_SOURCE_V10 V15 IRF |
IRFs | P7REP_*_V10V15, contained within ScienceTools release |
Output Data Products |
...
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="8c977f6b29cf4928-43394e38-4e7f49c1-a801836c-491aee0589b33f02eab130e7"><ac:plain-text-body><![CDATA[ | Data Product | destination | data content [1] | event selection [1] | makeFT1 | gtselect | gtdiffrsp | gtmktime | ]]></ac:plain-text-body></ac:structured-macro> |
---|---|---|---|---|---|---|---|---|---|
EXTENDEDFT1 | SLAC | FT1variables | ((FT1EventClass & 0x00003EFF)!=0) | | | | | ||
FT1 | FSSC+SLAC | FT1variables | 'source' and above | | | (inherited) | | ||
EXTENDEDLS1 | SLAC | LS1variables | ((FT1EventClass & 0x00003EFF)!=0) | | | | | ||
LS1 | FSSC+SLAC | LS1variables | 'transient' and above | | | (inherited) | | ||
ELECTRONFT1 | SLAC | FT1variables | CTBParticleType==1 | | | | |
...