Table of Contents | ||
---|---|---|
|
...
Run Number | Reason | Date | Time | Dls | L1Proc | GR version | rePipe | L1Proc |
---|---|---|---|---|---|---|---|---|
743660123 | MergeEvt found a gap of 6 datagrams in epu1. Nevertheless the halfPipe labeled the run as complete. Fortunately L1Proc.verify choked on it. Gapfinder doesn't report any gaps. | 2024-07-26 | 240716004 | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
742704925 | Puzzling. MergeEvt seems to have found all datagrams, but nevertheless labels both epu's as incomplete. L1Proc.verify claims that one datagram was dropped in epu0. | 2024-07-15 | 24071500[34] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
741148235 | MergeEvt reports 6 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run. | 2024-06-27 | 24062700[34] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
738549526 | L1Proc.verify claims to have found gaps of 5 datagrams each in both epus. mergeEvt reports the gap for epu0 but claims epu1 is complete. The HalfPipe labeled the run complete. Gapfinder doesn't report a gap for the duration of this run. | 2024-05-31 | 24053100[45] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
738549526 | mergeEvt reports 7 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run. | 2024-05-28 | 24052800[123] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
736930436 | mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report a gap for the duration of this run. | 2024-05-09 | 24050900[89] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
730458995 | 240224011.mergeEvt ran twice, creating an incomplete evt file on the second try. | 2024-02-24 | 24022401[01] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
729548905 | Highly fragmented run. 91, 15, and 61 chunks. This requires task RePipe. | 2024-02-14 | 24021400[345] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
727336262 | mergeEvt reports 6 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run. | 2024-01-19 | 240119003 | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
727324474 | mergeEvt reports missing datagrams at the end of epu1. Gapfinder doesn't report a gap for the duration of this run. | 2024-01-19 | 240119002 | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
726579387 | mergeEvt reports 5 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run. | 2024-01-10 | 24011000[78] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
726452942 | mergeEvt reports missing datagrams at the end of epu1. Gapfinder doesn't report a gap for the duration of this run. | 2024-01-09 | 00:29:01 - 01:41:00 | 240109004 | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||
723956756 | mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report a gap for the duration of this run. | 2023-12-11 | 03:05:55 - 04:39:36 | 23121100[23] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||
723939304 | mergeEvt reports 5 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run. | 2023-12-10 | 22:15:02 - 23:32:30 | 231211001 | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||
723865764 | mergeEvt reports 6 missing datagrams in epu1. Gapfinder doesn't report any gaps. | 2023-12-10 | 23121000[34] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
723665322 | mergeEvt reports 6 missing datagrams in epu1. Gapfinder found a gap for this run and sent just one retransmit request. | 2023-12-07 | 18:08:41 - 19:19:22 | 23120101[02] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||
723103279 | mergeEvt reports about 100 missing datagrams at the end of epu1. Gapfinder doesn't report any gaps. | 2023-12-01 | 23120100[78] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
717719650 | Dls 230929007 and 009 resulted from passes the FOT called "ratty". All run pieces in these dls were highly fragmented, and L1 didn't process them. Redumps were triggered, but only run 717708266 was completely filled in dl 230930008, 717713958 partially. Of the latter it's estimated that about half of the data are missing (1.2M events), while 717702333 (2.5M events) and 717719650 (2.7M events) should be almost filled. UPDATE: rePipe didn't result in more merged events. The runs were finalized as is, i.e. with the gaps. | 2023-09-29 | 23093000[123], (?230929009?) | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | - | ||
717713958 | 2023-09-29 | 230930008, (?230929009?) | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | - | |||
717702333 | 2023-09-29 | 230929008, 23093000[12], (?230929007?) | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | - | |||
710506750 | mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report any gaps. | 2023-07-08 | 23070800[89] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
709339013 | somehow the xrootd scratch area vanished before cleanupCompleteRun. The run needs to be repiped from the HalfPipe. | 2023-06-24 | 22:41:52 - 00:16:33 (+1) | 23062500[12] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||
708257402 | mergeEvt reports 8 missing datagrams in epu1. Gapfinder doesn't report any gaps. | 2023-06-12 | 23031200[56] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
700230335 | mergeEvt reports 5 missing datagrams in epu0. Gapfinder doesn't report any gaps. | 2023-02-11 | 230311009 | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
697691485 | mergeEvt reports 7 missing datagrams in epu1. | 2023-02-10 | 23021000[45] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
694642846 | verify found a datagram gap. Actually already mergeEvt found this gap of 12 datagrams in epu1, but somehow the run was labeled as complete. A HalfPipe repipe leaves one missing datagram, decompression error. Finalized with one datagram gap. | 2023-01-05 | 230105010, 230106001 | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | - | ||
693360593 | 003.mergeIndices failed and was processed last. When mergeEvt ran it didn't merge in the datagrams of 004. | 2022-12-22 | 22122200[234] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
687193623 | mergeReconChunks failed with a bus error but exit code 0. The merged recon file was not copied to xrootd but cleanupChunks removed all temporary files. | 2022-10-11 | 22101101[123] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
687119520 | mergeEvt reports 6 missing datagrams in epu0, gapfinder doesn't report any gaps. | 2022-10-10 | 221010013 | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
686035262 | dl's 220928003 and 004 came in together. It was not possible to disentangle both. 220928003.mergeIndices fails. | 2022-09-28 | 22092800[34] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
68602???? | 2022-09-28 | 220928003 | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||||
686022685 | 2022-09-28 | 22092800[23] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||||
685255472 | mergeEvt reports 7 missing datagrams in epu0, gapfinder doesn't report any gaps. | 2022-09-19 | 22091900[45] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
682569294 | Aug 19 (PDT) there were issues with a faulty nfs disk. This run was merged badly. | 2022-08-19 | 02:34:53 - 03:45:03 | 22081900[45] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||
682556983 | Aug 19 (PDT) there were issues with a faulty nfs disk. This run was merged badly. | 2022-08-18 | 23:09:42 - 00:29:45 (+1) | 22081900[34] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||
680740571 | mergeEvt reports 5 missing datagrams in epu1, gapfinder doesn't report any gaps. | 2022-07-28 | 22:36:10 - 23:58:01 | 22072900[12] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||
676577929 | mergeEvt reports 5 missing datagrams in epu0, gapfinder doesn't report any gaps. | 2022-06-10 | 22061001[01] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
675367448 | mergeEvt reports 8 missing datagram in epu0, additionally verify finds 7 missing datagrams in epu1. However gapfinder doesn't report any gaps. | 2022-05-27 | 18:04:07 - 19:28:29 | 220527012, 220528001 | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||
674884484 | mergeEvt reports 1 missing datagram in epu1, gapfinder doesn't report any gaps. MakeEvt reports a decompression error, finalizing with one datagram gap. | 2022-05-22 | 2205220[567] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | - | ||
674878528 | mergeEvt reports several gaps with missing datagrams in epu1, gapfinder doesn't report any gaps. MakeEvt reports decompression errors, finalizing with 14 scattered datagram gaps. | 2022-05-22 | 22052200[45] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | - | ||
674872543 | mergeEvt reports several gaps with missing datagrams in epu1, gapfinder doesn't report any gaps. MakeEvt reports decompression errors, finalizing with 16 missing datagrams scattered over 14 gaps. | 2022-05-22 | 22052200[34] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | - | ||
674052578 | mergeEvt reports 5 missing datagrams in epu1, gapfinder doesn't report any gaps. | 2022-05-12 | 220512010 | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
671827689 | verify reported missing data. Indeed also mergeEvt reports 5 missing datagrams in epu0, but somehow the run was labeled as complete. | 2022-04-16 | 220416014 | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
671754063 | missing datagrams at the end of epu1. gapfinder doesn't report any gaps. | 2022-04-16 | 22041600[23] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
670363411 | Dls 220331005 and 006 came in within the same second, and got mangled. 006 passed, but 005 fails at HalfPipe.mergeIndices. Needs to be repiped completely. | 2022-03-31 | 22033100[25] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
670357708 | Dls 220331005 and 006 came in within the same second, and got mangled. 006 passed, but 005 fails at HalfPipe.mergeIndices. Needs to be repiped completely. | 2022-03-31 | 22033100[345] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
661643650 | Highly fragmented run (138 chunks). Needs a "real" repipe. | 2021-12-20 | 21:54:09 - 23:28:58 | 21122000[124] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||
656744433 | Fall-out of the 211024 hypervisor failure. EPU0 is complete but EPU1 has a gap of ~5000 datagrams. Roll back of HalfPipe didn't work! Repipe did. But there was a gap in the magic7 file → second repipe | 2021-10-24 | 21102400[56] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
649557766 | Highly fragmented run (>400 chunks). Needs a "real" repipe. | 2021-08-02 | 21080200[234] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
648918184 | mergeEvt reports a gap of 5 datagrams in epu0. Gapfinder doesn't report any gaps. | 2021-07-25 | 15:03:03 - 16:13:05 | 21072501[01] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||
648862027 | mergeEvt reports a gap of 5 datagrams in epu1. Gapfinder doesn't report any gaps. | 2021-07-24 | 23:27:06 - 00:59:49[+1] | 21072500[12] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | ||
645896054 | One datagram in the inaccessible part of the SSR. The run will stay incomplete. | 2021-06-20 | 21062001[12] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | - | ||
645331200 | mergeEvt reports a gap of 9 datagrams in epu1. Gapfinder doesn't report any gaps. | 2021-06-14 | 21061400[34] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 | |||
644150495 | mergeEvt reports a gap of 9 datagrams in epu0. Gapfinder doesn't report any gaps. | 2021-05-31 | 21053100[78] | L1Proc-5.9 | GlastRelease-20-10-04-gr07 |
Run Number | Reason | Date | Time | Dls | L1Proc | GR version | rePipe | L1Proc |
---|---|---|---|---|---|---|---|---|
640254671 | mergeEvt reports a gap of 8 datagrams in epu1. Gapfinder doesn't report any gaps. | 2021-04-16 | 08:31:10-10:06:00 | 21041600[78] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
637031869 | mergeEvt reports a gap of 5 datagrams in epu1. Gapfinder doesn't report any gaps. | 2021-03-10 | 210310003 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
636598553 | missing datagrams at the end of epu0. Gapfinder doesn't report any gaps. | 2021-03-05 | 210305003 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
635205107 | Verify reports a datagram gap in epu1. Also none of the mergeEvt sees the run as complete. However somehow it got labeled "Complete" in L1Proc. Gapfinder doesn't report any gaps. | 2021-02-16 | 210216015, 210217001 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
633363581 | One chunk has one event only. Roll-back of HalfPipe.210126012 reproduced the one-event chunk. A "real" repipe worked. | |||||||
Run Number | Reason | Date | Time | Dls | L1Proc | GR version | rePipe | L1Proc |
633363581 | one chunk has one event only | 2021-01-26 | 21012601[23] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
632964783 | Missing data from the inaccessible part of the SSR. The run will stay incomplete. | 2021-01-21 | 23:33:02-01:07:56(+1) | 21012200[16] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | - | |
630962587 | Verify reports a datagram gap in epu0. Also none of the mergeEvt sees the run as complete. However somehow it got labeled "Complete" in L1Proc. GapFinder Gapfinder doesn't report any gaps. | 2020-12-29 | 20122901[012] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
627448410 | CAL FIFO overflow problem: digitization.201119003.627448410.3566554 fails with "3 CalDigis are missing readouts for Event: 4141638 Apid: 956 Event Index: 122309" | 2020-11-19 | 20111900[34] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | - | ||
626729954 | Highly fragmented run (>66 chunks). doRun/scanXxx processes fail. Moreover it's spread over 7 deliveries. | 2020-11-10 | 19:39:13-21:14:04 | 20111001[345], 20111100[1237] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
625160754 | mergeEvt: 6 missing datagrams in apid 956, gapfinder doesn't report any gaps | 2020-10-23 | 201023010 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
624820000 | switch from hequs to the bullets | 2020-10-19 | ||||||
624516636 | mergeEvt: 6 missing datagrams in apid 956, gapfinder doesn't report any gaps | 2020-10-16 | 20101600[45] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
623917505 | mergeEvt: 7 missing datagrams in apid 956 | 2020-10-09 | 06:25:04-08:02:32 | 201009007 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
622822397 | mergeEvt: 5 missing datagrams in apid 957 | 2020-09-26 | 14:13:16-15:48:06 | 20092601[12] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
622211988 | verify finds a gap of 3 datagrams in EPU0, and of 4 in EPU1 | 2020-09-19 | 20091901[02] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
622149260 | mergeEvt: 4 datagrams missing at the start of apid 956 | 2020-09-18 | 19:14:19-20:49:09 | 200919001 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
621835509 | mergeEvt: incomplete apid 956, probably 7 datagrams missing at the end (apid 957 has 7 more) | 2020-09-15 | 200915005 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
621824100 | mergeEvt: 8 missing datagrams in apid 957 | 2020-09-15 | 200915003 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
621806986 | mergeEvt: 6 missing datagrams in apid 957 | 2020-09-14 | 2009{14013,15001} | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
621664373 | mergeEvt: 1 non closed datagram at the end of apid 957, gapfinder: no datagram gaps for 2020-09-13 | 2020-09-13 | 200913004 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
621612709 | mergeEvt: 5 missing datagrams in apid 957, gapfinder: no datagram gaps for 2020-09-12 | 2020-09-12 | 14:11:48-15:27:32 | 20091201[012] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
621600789 | mergeEvt: 5 missing datagrams in apid 957, gapfinder: no datagram gaps for 2020-09-12 | 2020-09-12 | 10:53:08-12:01:40 | 200912009 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
621435528 | mergeEvt: 5 missing datagrams in apid 956, gapfinder: no datagram gaps for 2020-09-10 | 2020-09-10 | 12:58:47-14:10:30 | 20091001[01] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
621236535 | mergeEvt: 6 missing datagrams in apid 956, gapfinder: there was a gap an hour earlier, retransmitted | 2020-09-08 | 05:42:14-07:17:04 | 20090800[56] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
621081007 | mergeEvt: 5 missing datagrams in apid 956, gapfinder: no gaps for this run | 2020-09-06 | 10:30:06-11:43:23 | 200906010 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
620894265 | Michael screwed it up by accidentally rolling back the finished doRun.200904006.620894265 | 2020-09-04 | 20090400[67] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
620460723 | mergeEvt: 6 missing datagrams in apid 957, gapfinder: no gaps for 2020-08-30 | 2020-08-30 | 06:12:02-07:46:53 | 20083000[67] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
620375156 | missing datagrams | 2020-08-29 | 200829007 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
620369451 | caused 200829007.mergeIndices to fail because of an extra evt file. Repipe created identical evt files, thus L1Proc will not be rolled back, the run just needs to be finalized. | 2020-08-29 | 20082900[67] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
619313054 | mergeEvt: >1400 missing datagrams in both apids at the beginning of the run, gapfinder: no gaps around midnight of 2020-08-30 | 2020-08-16 | 23:24:13-00:33:20 | 20081700[23] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
618823233 | mergeEvt: 6 missing datagrams in apid 956, gapfinder: no gaps for the morning of 2020-08-11 | 2020-08-11 | 07:20:32-08:36:42 | 20073000[68] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
618805328 | mergeEvt: 8 missing datagrams in apid 957, gapfinder: no gaps for the morning of 2020-08-11 | 2020-08-11 | 02:22:07-03:31:26 | 20081100[45] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
617777656 | mergeEvt reports 8 missing datagrams in apid 956, but gapfinder doesn't report any gaps for 2020-07-30 | 2020-07-30 | 20073000[45] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
617190465 | mergeEvt reports 8 missing datagrams in apid 957, but gapfinder doesn't report any gaps for 2020-07-23 | 2020-07-23 | 200723007 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
615811484 | mergeEvt reports 6 missing datagrams, but gapfinder doesn't report any gaps | 2020-07-07 | 10:44:43-12:19:34 | 2007070{09,10} | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
615599718 | mergeEvt reports 6 missing datagrams, but gapfinder doesn't report any gaps | 2020-07-04 | 04 23:55:17-05 01:06:34 | 20070500[23] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
615508121 | one digitization fails: "... 2 CalDigis are missing readouts for Event ...". | 2020-07-03 | 22:28:40-23:37:33 | 20070314, 200704001 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | - | |
615073887 | missing datagrams. Gapfinder doesn't report any gaps for the evening of 2020-06-28. | 2020-06-28 | 21:51:26-23:06:42 | 20062900[12] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
614967205 | mergeEvt reports 6 data gaps in apid 957. Gapfinder doesn't report any gaps 2020-06-27. | 2020-06-27 | 16:13:24-17:48:15 | 20062701[23] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
614162399 | mergeEvt reports 5 datagaps in apid 956. However gapfinder doesn't find any gaps. | 2020-06-18 | 20061800[89] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
612445752 | The run had data gaps, which were supposed to be filled by a redump. Unfortunately one piece was in the inaccessible part of the SSR. The run will stay incomplete. | 2020-05-29 | 20052900[89] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | - | ||
612183340 | HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. There were several data gaps 2020-05-26. At least some were filled. | 2020-05-26 | 10:55:39-12:30:30 | 200526007, 20052700[125] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
612171931 | HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. There were several data gaps 2020-05-26. At least some were filled. | 2020-05-26 | 07:45:30-09:20:21 | 20052600[567], 200527001 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
612166226 | HalfPipe labeled this run as "InComplete". MergeEvt said it's complete. However when repiping 200526005 (for 612171931) mergeIndices relabeled it, and failed with an error message hinting to this run. MK: the error was due to an evt file with slight different name. However, rolling back the HalfPipes resulted in exactly the same evt file contents. Thus, I didn't roll back L1Proc. | 2020-05-26 | 06:10:25-07:14:16 | 20052600[45] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
611880377 | HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. Gapfinder doesn't report a gap for the duration of the run. | 2020-05-22 | 22:46:16-23:58:37 | 200522015, 20052300[12] | L1Proc-5.7 | GlastRelease-20-10-04-gr02 |
...
Run Number | Reason | Date | Time | Dls | L1Proc | GR version | rePipe | L1Proc |
---|---|---|---|---|---|---|---|---|
565721985 | Botched attempt at a repipe, repiped and marked as Complete | 2018-12-05 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
565942990 | stuck "in progress", repiped and marked as Complete | 2018-12-08 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
566022500 | HP "in progress", repiped and marked as Complete | 2018-12-09 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
566235626 | HP "in progress", repiped and marked as Complete | 2018-12-11 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
567301141 | stuck "in progress", repiped and marked as Complete | 2018-12-23 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
567598970 | Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized. | 2018-12-27 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
567581889 | Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized. | 2018-12-27 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
569069307 | in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized. | 2019-01-13 | 010 011 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
569213577 | in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized. | 2019-01-15 | 006 007 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
572617673 | HP "in progress", repiped and marked as Complete | 2019-02-23 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
572853518 | stuck "in progress". Leftover chunkToken. Fixed. | 2019-02-26 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
574781844 | "in progress", incomplete data, missing datagrams | 2019-03-20 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
575044322 | "in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized. | 2019-03-23 | 013 014 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
575762229 | "in progress", incomplete data, missing datagrams | 2019-03-31 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
576305031 | Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized. | 2019-04-07 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
576658880 | "in progress", incomplete data | 2019-04-11 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
578530168 | corrupted data in monitoring | 2019-05-02 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
580907445 | in progress, mergeIndices reports 5 missing datagrams | 2019-05-30 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
582073028 | in progress, mergeIndices reports missing datagrams | 2019-06-13 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
584730839 | in progress, complete data, missing datagrams | 2019-07-13 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
585187986 | This was actually complete. A botched rollback of halfPipe (190719003) produced leftover chunkTokens. Set to Complete and finalized. | 2019-07-19 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
585193691 | Both halfPipe and RePipe failed at delivering a complete set of data (this included a botched rollback of halfPipe 190719003). Set to Incomplete (due to 11 missing datagrams) and finalized. | 2019-07-19 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
585535991 | in progress, complete data, missing datagrams | 2019-07-23 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
585690096 | in progress, missing datagrams | 2019-07-24 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
586076888 | see thread. However it seems the recon file is complete. Reprocessing complete. Everything seems to have worked including the cleanupCompleteRun process. | 2019-07-29 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
586893790 | bunch of tasks in "Waiting" state preventing a rollback. Once that is cleared a rollback should hopefully fix everything | 2019-08-07 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | N/A | |||
586905200 | see thread . Seems the run is complete and can be just finalized. | 2019-08-07 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
587042253 | Several failed tasks with errors of the form: "IOError: Can't stage in <filename>". Looks like it was an xrootd hiccup. Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete | 2019-08-09 12:46 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | N/A | |||
587104373 | This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success. There is a chunk token directory but no token. However, the verify log complains about closing reasons for both EUP0 and EPU1. I'm think that means we need to repipe everything. | 2019-08-10 6:54 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
588742217 | Run is incomplete according to the halfPipe. | 2019-08-29 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
590065315 | Run still Incomplete in L1. Looks like cleanupCompleteRun didn't get triggered. | 2019-09-13 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
591931054 | failure in checkRun 191005003.591931054. 2 missing datagram. | 2019-10-05 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
595240349 | 9 missing datagrams | 2019-11-12 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
595405654 | missing datagrams | 2019-11-14 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
595576882 | missing datagrams. In progress | 2019-11-16 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
595963106 | Missing datagrams due to decompression errors- Both halfPipe and rePipe report 6789 orphans (which translate to 53 missing datagrams). Set to Incomplete and finalized. | 2019-11-20 | 17:18:24 - 18:32:18 | 191120013 191121001 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
596299562 | HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-11-24 21:30-24:00 | 2019-11-24 | 14:46:00 - 16:01:21 | 191124009 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
596916997 | missing datagrams | 2019-12-01 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||||
597315084 | HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-12-06 05:00-14:00 | 2019-12-06 | 08:51:23 - 10:06:10 | 191206009 010 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
598223809 | HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-16 | 2019-12-16 | 21:16:48 - 22:51:39 | 191217001 002 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
598581385 | HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-21 | 2019-12-21 | 00:36:24 - 01:53:08 | 191221003 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
598645966 | HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-21 | 2019-12-21 | 18:32:45 - 20:07:36 | 191221009 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
598758794 | HalfPipe reports "inProgress" | 2019-12-23 | 05:39:57 - 08:44:23 | 191223003 004 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
599272563 | HalfPipe reports "inProgress". Datagram errors | 2019-12-29 | 2:22:25 - 4:58:04 | 191229002 003 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
599746995 | HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2020-01-03 | 2020-01-03 | 12:23:14 - 13:58:05 | 200103009 010 011 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
599809968 | HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2020-01-04 | 2020-01-04 | 05:52:47 - 07:14:52 | 200103009 010 011 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
600328891 | HalfPipe reports "inProgress". Datagram errors | 2020-01-10 | 10:43:13 | 200110005 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
601777922 | HalfPipe reports "inProgress", but mergeEvt doesn't report any datagram errors! Gapfinder doesn't report any data gaps for 2020-01-27. | 2020-01-27 | 02:07:06 - 03:41:57 | 200127002 003 004 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
601783627 | HalfPipe reports "inProgress". Missing datagrams. Gapfinder doesn't report any data gaps for 2020-01-27. | 2020-01-27 | 00:32:01 - 02:06:55 | 200127004 005 006 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
601890911 | HalfPipe reports "inProgress". Missing datagrams. Gapfinder doesn't report any data gaps for the morning of 2020-01-28. | 2020-01-28 | 07:55:10 - 09:04:44 | 200128006 008 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
602153762 | HalfPipe reports "inProgress". Missing datagrams. Gapfinder doesn't report any data gaps for 2020-01-31. No decompression errors in makeEvt. | 2020-01-31 | 08:56:01 - 10:07:38 | 200131007 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
603637679 | HalfPipe reports "inProgress". Missing datagrams. Gapfinder doesn't report any data gaps for 2020-02-17. No decompression errors in makeEvt. | 2020-02-17 | 13:07:57 - 14:42:49 | 200217009 010 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
603974262 | Run is complete. mergeEvt.200221009.603974262 reports 6 missing datagrams. gapfinder doesn't report any data gaps for the time interval. | 2020-02-21 | 10:37:41 - 12:12:31 | 200221009 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
604573262 | HalfPipe reports "inProgress". Missing datagrams. Gapfinder doesn't report any data gaps for 2020-02-28. | 2020-02-28 | 09:01:01 - 10:35:52 | 200228008 - 009 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
605908182 | HalfPipe reports "inProgress". Missing datagrams. Gapfinder doesn't report any data gaps for 2020-03-15. | 2020-03-15 | 19:49:41 - 21:24:31 | 200315002 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
607059914 | Highly fragmented run (231 chunks). doRun/scanXxx processes fail. | 2020-03-28 | 200328005, 006, 008 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
609843485 | HalfPipe reports "InProgress". GapFinder Gapfinder doesn't report any data gaps of 2020-04-29. | 2020-04-29 | 200429007 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
610323623 | mergeIndices labeled this run as Incomplete, but mergeEvt as Complete! MK: Repipe resulted in the same chunks. Checking the old merit file didn't reveal any gaps within the run. Thus I relabeled the run to "Complete" and rolled-back checkRun. | 2020-05-05 | 200505001, 002 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
610711547 | mergeEvt reports missing datagrams. Gapfinder doesn't report any data gaps for 2020-05-09. | 2020-05-09 | 10:05:46 - 11:23:53 | 200509008, 009 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
610791211 | HalfPipe labels the run as Incomplete, mergeEvt reports missing datagrams. Gapfinder doesn't report any data gaps for 2020-05-10. | 2020-05-10 | 08:13:30 - 09:30:17 | 200510006, 007 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | ||
610968381 | mergeEvt reports missing datagrams | 2020-05-12 | 200512007, 008 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
610996774 | mergeEvt reports missing datagrams | 2020-05-12 | 200512014, 015 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
611162210 | mergeEvt reports missing datagrams | 2020-05-14 | 200514011, 012 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 | |||
611782769 | HalfPipe labels the run as "Incomplete". MergeEvt reports missing datagrams. Gapfinder only reports a gap after 21:19. | 2020-05-21 | 19:39:28 - 20:50:13 | 200522001 | L1Proc-5.7 | GlastRelease-20-10-04-gr02 |
...