Versions Compared

Old Version 330

changes.mady.by.user Michael Kuss

Saved on

compared with

New Version 331

changes.mady.by.user Michael Kuss

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Run NumberReasonDateTimeDlsL1ProcGR versionrePipeL1Proc
622149260mergeEvt: 4 datagrams missing at the start of apid 9562020-09-1819:14:19-20:49:09200919001L1Proc-5.7GlastRelease-20-10-04-gr02(error)Michael(error)Michael
621835509mergeEvt: incomplete apid 956, probably 7 datagrams missing at the end (apid 957 has 7 more)2020-09-15 
200915005L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621824100mergeEvt: 8 missing datagrams in apid 9572020-09-15 
200915003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621806986mergeEvt: 6 missing datagrams in apid 9572020-09-14 
2009{14013,15001}L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621664373mergeEvt: 1 non closed datagram at the end of apid 957, gapfinder: no datagram gaps for 2020-09-132020-09-13 
200913004L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621612709mergeEvt: 5 missing datagrams in apid 957, gapfinder: no datagram gaps for 2020-09-122020-09-1214:11:48-15:27:3220091201[012]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621600789mergeEvt: 5 missing datagrams in apid 957, gapfinder: no datagram gaps for 2020-09-122020-09-1210:53:08-12:01:40200912009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621435528mergeEvt: 5 missing datagrams in apid 956, gapfinder: no datagram gaps for 2020-09-102020-09-1012:58:47-14:10:3020091001[01]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621236535mergeEvt: 6 missing datagrams in apid 956, gapfinder: there was a gap an hour earlier, retransmitted2020-09-0805:42:14-07:17:0420090800[56]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621081007mergeEvt: 5 missing datagrams in apid 956, gapfinder: no gaps for this run2020-09-0610:30:06-11:43:23200906010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620894265Michael screwed it up by accidentally rolling back the finished doRun.200904006.6208942652020-09-04 
20090400[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620460723mergeEvt: 6 missing datagrams in apid 957, gapfinder: no gaps for 2020-08-302020-08-3006:12:02-07:46:5320083000[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620375156missing datagrams2020-08-29 
200829007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620369451caused 200829007.mergeIndices to fail because of an extra evt file. Repipe created identical evt files, thus L1Proc will not be rolled back, the run just needs to be finalized.2020-08-29 
20082900[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
619313054mergeEvt: >1400 missing datagrams in both apids at the beginning of the run, gapfinder: no gaps around midnight of 2020-08-302020-08-1623:24:13-00:33:2020081700[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
618823233mergeEvt: 6 missing datagrams in apid 956, gapfinder: no gaps for the morning of 2020-08-112020-08-1107:20:32-08:36:4220073000[68]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
618805328mergeEvt: 8 missing datagrams in apid 957, gapfinder: no gaps for the morning of 2020-08-112020-08-1102:22:07-03:31:2620081100[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
617777656mergeEvt reports 8 missing datagrams in apid 956, but gapfinder doesn't report any gaps for 2020-07-302020-07-30 
20073000[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
617190465mergeEvt reports 8 missing datagrams in apid 957, but gapfinder doesn't report any gaps for 2020-07-232020-07-23 
200723007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615811484mergeEvt reports 6 missing datagrams, but gapfinder doesn't report any gaps2020-07-0710:44:43-12:19:342007070{09,10}L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615599718mergeEvt reports 6 missing datagrams, but gapfinder doesn't report any gaps2020-07-0404 23:55:17-05 01:06:3420070500[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615508121one digitization fails: "... 2 CalDigis are missing readouts for Event ...".2020-07-0322:28:40-23:37:3320070314, 200704001L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
615073887missing datagrams. Gapfinder doesn't report any gaps for the evening of 2020-06-28.2020-06-2821:51:26-23:06:4220062900[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
614967205mergeEvt reports 6 data gaps in apid 957. Gapfinder doesn't report any gaps 2020-06-27.2020-06-2716:13:24-17:48:1520062701[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
614162399mergeEvt reports 5 datagaps in apid 956. However gapfinder doesn't find any gaps.2020-06-18 
20061800[89]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612445752The run had data gaps, which were supposed to be filled by a redump. Unfortunately one piece was in the inaccessible part of the SSR. The run will stay incomplete.2020-05-29 
20052900[89]L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
612183340HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. There were several data gaps 2020-05-26. At least some were filled.2020-05-2610:55:39-12:30:30200526007, 20052700[125]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612171931HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. There were several data gaps 2020-05-26. At least some were filled.2020-05-2607:45:30-09:20:2120052600[567], 200527001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612166226HalfPipe labeled this run as "InComplete". MergeEvt said it's complete. However when repiping 200526005 (for 612171931) mergeIndices relabeled it, and failed with an error message hinting to this run. MK: the error was due to an evt file with slight different name. However, rolling back the HalfPipes resulted in exactly the same evt file contents. Thus, I didn't roll back L1Proc.2020-05-2606:10:25-07:14:1620052600[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
611880377HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. Gapfinder doesn't report a gap for the duration of the run.2020-05-2222:46:16-23:58:37200522015, 20052300[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)

...

Run Number

Reason

Date

TimeDls

L1Proc

GR version

rePipe

L1Proc

565721985Botched attempt at a repipe, repiped and marked as Complete2018-12-05  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
565942990stuck "in progress", repiped and marked as Complete2018-12-08  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

(tick)

566022500HP "in progress", repiped and marked as Complete2018-12-09  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
566235626HP "in progress", repiped and marked as Complete 2018-12-11  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
567301141stuck "in progress", repiped and marked as Complete2018-12-23  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

567598970

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
567581889

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
569069307in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-13

 
010 011L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
569213577

in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-15 
006 007L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572617673HP "in progress", repiped and marked as Complete 2019-02-23  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572853518stuck "in progress". Leftover chunkToken. Fixed.2019-02-26  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
574781844"in progress", incomplete data, missing datagrams2019-03-20  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575044322"in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized.2019-03-23 
013 014L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575762229"in progress", incomplete data, missing datagrams2019-03-31  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
576305031Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized.2019-04-07  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
576658880"in progress", incomplete data2019-04-11  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

578530168

corrupted data in monitoring2019-05-02  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
580907445in progress, mergeIndices reports 5 missing datagrams2019-05-30  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
582073028in progress, mergeIndices reports missing datagrams2019-06-13  

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
584730839in progress, complete data, missing datagrams2019-07-13  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585187986

This was actually complete. A botched rollback of halfPipe (190719003) produced leftover chunkTokens. Set to Complete and finalized.

2019-07-19  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585193691Both halfPipe and RePipe failed at delivering a complete set of data (this included a botched rollback of halfPipe 190719003). Set to Incomplete (due to 11 missing datagrams) and finalized.2019-07-19  

L1Proc-5.7GlastRelease-20-10-04-gr02(error)(tick)
585535991in progress, complete data, missing datagrams2019-07-23  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585690096in progress, missing datagrams2019-07-24  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586076888see thread. However it seems the recon file is complete. Reprocessing complete.  Everything seems to have worked including the cleanupCompleteRun process.2019-07-29  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586893790bunch of tasks in "Waiting" state preventing a rollback.  Once that is cleared a rollback should hopefully fix everything2019-08-07  

L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
586905200see thread . Seems the run is complete and can be just finalized. After checking, I've finalized this run as it looks fine -TS. MEM: same as 588742217, run was *not* finalized. 1. remove chunktokens as needed, 2. rollback checkRun, 3. make sure that cleanupCompleteRun is executed and recon file is merged. This is fine now. 2019-08-07  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
587042253Several failed tasks with errors of the form: "IOError: Can't stage in <filename>".  Looks like it was an xrootd hiccup.  Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete2019-08-09 12:46  

L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
587104373

This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success.  There is a chunk token directory but no token.  However, the verify log complains about closing reasons for both EUP0 and EPU1. I'm think that means we need to repipe everything.

2019-08-10 6:54  

L1Proc-5.7GlastRelease-20-10-04-gr02

(tick)

(tick)
588742217Run is incomplete according to the halfPipe. TS: I think this is okay as I don't see any errors in the verify log. MEM: run wasn't finalized because of leftover chunktokens. Please make sure that cleanupCompleteRun is executed before claiming success. This is fine now.2019-08-29  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
590065315Run still Incomplete in L1.  Looks like cleanupCompleteRun didn't get triggered.2019-09-13  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
591931054 failure in checkRun 191005003.591931054.  2 missing datagram.2019-10-05  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
5952403499 missing datagrams2019-11-12  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595405654missing datagrams2019-11-14  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595576882missing datagrams.  In progress2019-11-16  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595963106Missing datagrams due to decompression errors- Both halfPipe and rePipe report 6789 orphans (which translate to 53 missing datagrams). Set to Incomplete and finalized.2019-11-2017:18:24 - 18:32:18

191120013 191121001

L1Proc-5.7GlastRelease-20-10-04-gr02 (error) (tick)
596299562HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-11-24 21:30-24:002019-11-2414:46:00 - 16:01:21

191124009

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
596916997missing datagrams2019-12-01  

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
597315084HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-12-06 05:00-14:002019-12-0608:51:23 - 10:06:10191206009 010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598223809HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-162019-12-1621:16:48 - 22:51:39191217001 002L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598581385HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2100:36:24 - 01:53:08191221003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598645966HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2118:32:45 - 20:07:36191221009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598758794HalfPipe reports "inProgress"2019-12-2305:39:57 - 08:44:23191223003 004L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599272563HalfPipe reports "inProgress".  Datagram errors2019-12-292:22:25 - 4:58:04191229002 003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599746995HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2020-01-032020-01-0312:23:14 - 13:58:05200103009 010 011L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599809968HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2020-01-042020-01-0405:52:47 - 07:14:52200103009 010 011L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
600328891HalfPipe reports "inProgress".  Datagram errors2020-01-1010:43:13200110005L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
601777922HalfPipe reports "inProgress", but mergeEvt doesn't report any datagram errors!  Gapfinder doesn't report any data gaps for 2020-01-27.2020-01-2702:07:06 - 03:41:57200127002 003 004L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
601783627HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-27.2020-01-2700:32:01 - 02:06:55200127004 005 006L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
601890911HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for the morning of 2020-01-28.2020-01-2807:55:10 - 09:04:44200128006 008L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
602153762HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-31. No decompression errors in makeEvt.2020-01-3108:56:01 - 10:07:38200131007L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
603637679HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-02-17. No decompression errors in makeEvt.2020-02-1713:07:57 - 14:42:49200217009 010L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
603974262Run is complete.  mergeEvt.200221009.603974262 reports 6 missing datagrams. gapfinder doesn't report any data gaps for the time interval.2020-02-2110:37:41 - 12:12:31200221009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
604573262

HalfPipe reports "inProgress".  Missing datagrams. Gapfinder doesn't report any data gaps for 2020-02-28.

2020-02-2809:01:01 - 10:35:52200228008 - 009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
605908182HalfPipe reports "inProgress".  Missing datagrams. Gapfinder doesn't report any data gaps for 2020-03-15.2020-03-1519:49:41 - 21:24:31200315002L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
607059914Highly fragmented run (231 chunks). doRun/scanXxx processes fail.2020-03-28 
200328005, 006, 008L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
609843485HalfPipe reports "InProgress". GapFinder doesn't report any data gaps of 2020-04-29.2020-04-29 
200429007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610323623mergeIndices labeled this run as Incomplete, but mergeEvt as Complete! MK: Repipe resulted in the same chunks. Checking the old merit file didn't reveal any gaps within the run. Thus I relabeled the run to "Complete" and rolled-back checkRun.2020-05-05 
200505001, 002L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610711547mergeEvt reports missing datagrams. Gapfinder doesn't report any data gaps for 2020-05-09.2020-05-0910:05:46 - 11:23:53200509008, 009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610791211HalfPipe labels the run as Incomplete, mergeEvt reports missing datagrams. Gapfinder doesn't report any data gaps for 2020-05-10.2020-05-1008:13:30 - 09:30:17200510006, 007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610968381mergeEvt reports missing datagrams2020-05-12 
200512007, 008L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610996774mergeEvt reports missing datagrams2020-05-12 
200512014, 015L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
611162210mergeEvt reports missing datagrams2020-05-14 
200514011, 012L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
611782769HalfPipe labels the run as "Incomplete". MergeEvt reports missing datagrams. Gapfinder only reports a gap after 21:19.2020-05-2119:39:28 - 20:50:13200522001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)

...