...
- 9/17/2013
- 13:33 Begin block 11 backfill, 539 runs in the range 398087830 through 401106634 (2013-09-17 11:48:16). Once complete, 28412 runs will have been reprocessed.
- 9/19/2013
- 08:30 block 11 complete. The bulk of runs completed within 14 hours, but stragglers, failed/terminated jobs, due to transient problems, bad batch machines, etc., delayed completion until this morning. DataCatalog reports a total of 61,827,628,706 events in {CAL,MERIT,RECON,GCR} and 8,991,163,530 events in FILTEREDMERIT.
- xroot scratch space cleaned up.
- 10/4/2013
- Prepare for block 12 backfill. Warren suggests ending with run 402560477 (2013-10-04 06:21:14 UTC). First run of block 12 is 401112810. There are 263 runs in this backfill block which will bring the reprocessed run total to 28675.
- 10/15/2013
Prepare for block 13 backfill. Warren suggests ending with run 403510814, 170 new runs.
Run
Run start time (UTC)
Task Stream
Block 13 start
402566464
2013-10-04 08:01:01
28675
Block 13 end
403510814
2013-10-15 06:20:11
28844
- 11/05/2013
- Today P7REP goes public.
- Prepare for block 14 backfill – the ultimate backfill for P7REP. Last run = 405329691 (2013-11-05 07:34:48)
313 new runs to reprocess...
Run
Run start time (UTC)
Task Stream
Block 14 start
403516762
2013-10-15 07:59:19
28845
Block 14 end
405329691
2013-11-05 07:34:48
29157
- 8/19/2014
- User reported problem with an event inside an a ~195s interval with LIVETIME=0 in run 395891323 (2013-07-19 01:48:40 UTC). Problem traced to the a bad FT2SECONDS file. Reprocessing took place 14 Aug 2013 using v002 FT2 file. Then on 19 Aug 2013, the FT2 file was updated to v003. M.E. has rebuilt a proper FT2 file in the Reprocess/P202 area, v203 and will roll back entire stream 27487 to rebuild ROOT files. To get this rollback to run, the following steps were necessary:
config.py - change ft2Selection from 'P105' to 'P202' (commonTools/repTools.py:findFT2() already knows about P202, so no changes needed there.)
commonTools/setupGR.sh - change svsopts from 'redhat5' back to 'redhat4' (TEMPORARILY) for processClump step
commonTools/setupSkimmer.py - a series of reversions:
- revert to skimmer 08-02-01
- revert to rootVersion = 'v5.26.00a-gl6'
- revert glastBuild from redhat5-i686-32bit-gcc41 to
redhat4-i686-32bit-gcc34 to get libraries for mergeClumps step.
- revert to old ROOTSYS definition that includes compiler version as last path item
- The mods to setupGR.sh and setupSkimmer.py have been preserved as OLD-setupGR.sh and OLD-setupSkimmer.py in the commonTools directory
- User reported problem with an event inside an a ~195s interval with LIVETIME=0 in run 395891323 (2013-07-19 01:48:40 UTC). Problem traced to the a bad FT2SECONDS file. Reprocessing took place 14 Aug 2013 using v002 FT2 file. Then on 19 Aug 2013, the FT2 file was updated to v003. M.E. has rebuilt a proper FT2 file in the Reprocess/P202 area, v203 and will roll back entire stream 27487 to rebuild ROOT files. To get this rollback to run, the following steps were necessary:
...