...
- 4/23/2013 - Begin setting up P300 task for Pass8 reprocessing.
- 5/10/2013 - Initial version of task running. See this page for infomation about the task creation, and this page for performance comparison with P202
- 7/2/2013 - after weeks of testing, final GR, jobOpts, etc. come together and production tentatively starts.
- block Block 1 is defined with this run range: 239557414 through 392842073 (2008-08-04 15:43:37 through 2013-06-13 18:47:50 UTC)
- Update task to use the P202 generation of FT2 files (adds columns LAMBDA, RA_SUN, DEC_SUN, as well as meaningful BTIs)
- First 10 trial jobs launched. Too many ACD INFO messages 'Missed Poca for ID 602 at...' so modify jobOpts to disable.
- xroot disk space at the start of Pass8 = 508 TB (just after commissioning of fermi-xrd11)
- 7/29/2013 - a new GR is prepared (20-09-01) which fixes some problems in ACD tracking issues affecting ~32 runs so far (out of 5283). trickleStream disabled, wait for pending jobs to start, rollback other failed jobs, switch to new GR, rollback of ACD-failed jobs...
- This new GR continues to seg fault, so revert back to GR 20-09-00 and restart trickleStream.
- 7/31/2013 - A new GR (20-09-02) with patch for ACD tracking installed after stream 5970. At the time, there were 33 failed processClumps jobs. All rolled back.
- Last stream with GR 20-09-00 = 5970, run 274118943
- First stream with GR 20-09-02 = 5971, run 274124672
- Full list of rolled back substreams is here
- 10/15/2013 - Update job options to use the new "L1current" flavor of calibrations/alignment rather than "p7repro". First stream with new configuration: 20804.
- 11/14/2013 - block Block 1 complete except for routine cleanup
11/21/2013 - block 1 complete Block 1 complete
Problems encountered include:
** pipeline_summary file corruption causing loss of email or of it being ignored, finally jobs terminate via reaper (Tony notified)
** large blocks of failures due to dataCatalog not returning desired FT2 files (Brian notified, fix possible by moving query to scriptlet)
** jobs running out the clock due to anomalously slow xroot behavior (Wilko/Andy notified)
** problems with individual batch machines (Renata notified, and I also have the authority to remove batch machines from production)
Summary of events processed in Block 1Name Type Files Events Size Created (UTC) Links ELECTRONMERIT Group 26929 0 1.4 GB 09-May-2013 22:22:19 Files FILTEREDMERIT Group 26929 0 1.4 GB 09-May-2013 22:22:17 Files GCR Group 26929 58,653,424,708 1.0 TB 09-May-2013 22:22:18 Files MERIT Group 26929 58,653,424,708 98.3 TB 09-May-2013 22:22:18 Files RECON Group 26929 58,653,424,708 816.1 TB 09-May-2013 22:22:20 Files RELATION Group 26929 58,653,424,708 7.7 TB 09-May-2013 22:22:20 Files
...