...
- 9/17/2013
- 13:33 Begin block 11 backfill, 539 runs in the range 398087830 through 401106634 (2013-09-17 11:48:16). Once complete, 28412 runs will have been reprocessed.
- 9/19/2013
- 08:30 block 11 complete. The bulk of runs completed within 14 hours, but stragglers, failed/terminated jobs, due to transient problems, bad batch machines, etc., delayed completion until this morning. DataCatalog reports a total of 61,827,628,706 events in {CAL,MERIT,RECON,GCR} and 8,991,163,530 events in FILTEREDMERIT.
- xroot scratch space cleaned up.
- 10/4/2013
- Prepare for block 12 backfill. Warren suggests ending with run 402560477 (2013-10-04 06:21:14 UTC). First run of block 12 is 401112810. There are 263 runs in this backfill block which will bring the reprocessed run total to 28675.
- 10/15/2013
Prepare for block 13 backfill. Warren suggests ending with run 403510814, 170 new runs.
Run
Run start time (UTC)
Task Stream
Block 13 start
402566464
2013-10-04 08:01:01
28675
Block 13 end
403510814
2013-10-15 06:20:11
28844
- 11/05/2013
- Today P7REP goes public.
- Prepare for block 14 backfill – the ultimate backfill for P7REP. Last run = 405329691 (2013-11-05 07:34:48)
313 new runs to reprocess...
Run
Run start time (UTC)
Task Stream
Block 14 start
403516762
2013-10-15 07:59:19
28845
Block 14 end
405329691
2013-11-05 07:34:48
29157
- 8/19/2014
- User reported problem with an event inside an interval with LIVETIME=0 in run 395891323 (2013-07-19 01:48:40 UTC). Problem traced to the FT2SECONDS file. Reprocessing took place 14 Aug 2013 using v002 FT2 file. Then on 19 Aug 2013, the FT2 file was updated to v003. M.E. has rebuilt a proper FT2 file in the Reprocess/P202 area, v203 and will roll back entire stream 27487 to rebuild ROOT files. To get this rollback to run, the following steps were necessary:
config.py - change ft2Selection from 'P105' to 'P202' (commonTools/repTools.py:findFT2() already knows about P202, so no changes needed there.)
commonTools/setupGR.sh - change svsopts from 'redhat5' back to 'redhat4' (TEMPORARILY) for processClump step
commonTools/setupSkimmer.py - a series of reversions:
- revert to skimmer 08-02-01
- revert to rootVersion = 'v5.26.00a-gl6'
- revert glastBuild from redhat5-i686-32bit-gcc41 to
redhat4-i686-32bit-gcc34 to get libraries for mergeClumps step.
- revert to old ROOTSYS definition that includes compiler version as last path item
- The mods to setupGR.sh and setupSkimmer.py have been preserved as OLD-setupGR.sh and OLD-setupSkimmer.py in the commonTools directory
- User reported problem with an event inside an interval with LIVETIME=0 in run 395891323 (2013-07-19 01:48:40 UTC). Problem traced to the FT2SECONDS file. Reprocessing took place 14 Aug 2013 using v002 FT2 file. Then on 19 Aug 2013, the FT2 file was updated to v003. M.E. has rebuilt a proper FT2 file in the Reprocess/P202 area, v203 and will roll back entire stream 27487 to rebuild ROOT files. To get this rollback to run, the following steps were necessary:
Configuration
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P202-ROOT |
Task Status | |
GlastRelease | 17-35-24-gr17 and 17-35-24-rp04 (SCons RHEL4-32 build) |
Run Selection | based on a modified "standard" selection, see https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
s/c data | "standard" Public Release 2 https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-ROOT/config/runList.txt |
photonFilter | CTBParticleType==1 && ((FT1EventClass & 0x00003EFF)!=0) |
electronFilter | CTBParticleType==1 |
Code Variants | redhat4-i686-32bit-gcc34 (Optimized) |
jobOpts | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-ROOT/config/doRecon.txt |
Output Data Products |
...
- 10/7/2013 - prepare block4 (corresponding to P202-ROOT block12).
10/8/2013 - block 4 complete
Name
Files
Events
Size
Created (UTC)
ELECTRONMERIT
28675
126,895,805
286.1 GB
25-Jan-2012 00:53:32
FT1
28675
301,166,782
27.1 GB
02-Mar-2012 00:06:06
LS1
28675
1,964,328,038
319.1 GB
02-Mar-2012 00:06:08
EXTENDEDLS1
28675
9,072,793,026
1.4 TB
02-Mar-2012 00:06:09
FILTEREDMERIT
28675
9,072,793,026
7.7 TB
25-Jan-2012 00:53:29
EXTENDEDFT1
28675
9,072,793,026
795.0 GB
02-Mar-2012 00:06:09
MERIT
28675
62,383,754,997
49.9 TB
25-Jan-2012 00:53:30
CAL
28675
62,383,754,997
180.6 TB
25-Jan-2012 00:53:31
GCR
28675
62,383,754,997
1.3 TB
25-Jan-2012 00:53:31
RECON
28675
62,383,754,997
831.0 TB
25-Jan-2012 00:53:33
- 10/17/2013 - prepare block5 backfill (P202-ROOT block13).
10/24/2013 - block 5 complete (as of 10/18/2013)
Name
Files
Events
Size
ELECTRONMERIT
28844
127,613,745
287.7 GB
FT1
28844
303,135,818
27.3 GB
LS1
28844
1,976,418,512
321.1 GB
EXTENDEDLS1
28844
9,127,005,732
1.4 TB
FILTEREDMERIT
28844
9,127,005,732
7.7 TB
EXTENDEDFT1
28844
9,127,005,732
799.8 GB
MERIT
28844
62,747,048,217
50.2 TB
CAL
28844
62,747,048,217
181.6 TB
GCR
28844
62,747,048,217
1.3 TB
RECON
28844
62,747,048,217
835.8 TB
11/7/2013 - block 6 complete and P7REP was made public two days ago. END OF PROJECT
Final tally by the dataCatalog:Name Files Events Size Created (UTC) CAL 29158 63,449,261,789 183.6 TB 25-Jan-2012 00:53:31 ELECTRONMERIT 29158 128,971,706 290.7 GB 25-Jan-2012 00:53:32 EXTENDEDFT1 29158 9,243,063,549 809.9 GB 02-Mar-2012 00:06:09 EXTENDEDLS1 29158 9,243,063,549 1.5 TB 02-Mar-2012 00:06:09 FILTEREDMERIT 29158 9,243,063,549 7.8 TB 25-Jan-2012 00:53:29 FT1 29158 310,326,817 27.9 GB 02-Mar-2012 00:06:06 GCR 29158 63,449,261,789 1.3 TB 25-Jan-2012 00:53:31 LS1 29158 2,007,123,864 326.1 GB 02-Mar-2012 00:06:08 MERIT 29158 63,449,261,789 50.8 TB 25-Jan-2012 00:53:30 RECON 29158 63,449,261,789 845.0 TB 25-Jan-2012 00:53:33
...