Runs from May 31, 2021 onward (L1Proc-5.9), backward sorted

Run NumberReasonDateTimeDlsL1ProcGR versionrePipeL1Proc
730458995240224011.mergeEvt ran twice, creating an incomplete evt file on the second try.2024-02-24
24022401[01]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
729548905Highly fragmented run. 91, 15, and 61 chunks. This requires task RePipe.2024-02-14
24021400[345]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
727336262mergeEvt reports 6 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-19
240119003L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
727324474mergeEvt reports missing datagrams at the end of epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-19
240119002L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
726579387mergeEvt reports 5 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-10
24011000[78]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
726452942mergeEvt reports missing datagrams at the end of epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-0900:29:01 - 01:41:00240109004L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
723956756mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report a gap for the duration of this run.2023-12-1103:05:55 - 04:39:3623121100[23]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
723939304mergeEvt reports 5 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2023-12-1022:15:02 - 23:32:30231211001L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
723865764mergeEvt reports 6 missing datagrams in epu1. Gapfinder doesn't report any gaps.2023-12-10
23121000[34]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
723665322mergeEvt reports 6 missing datagrams in epu1. Gapfinder found a gap for this run and sent just one retransmit request.2023-12-0718:08:41 - 19:19:2223120101[02]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
723103279mergeEvt reports about 100 missing datagrams at the end of epu1. Gapfinder doesn't report any gaps.2023-12-01
23120100[78]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
717719650

Dls 230929007 and 009 resulted from passes the FOT called "ratty".  All run pieces in these dls were highly fragmented, and L1 didn't process them.  Redumps were triggered, but only run 717708266 was completely filled in dl 230930008, 717713958 partially.  Of the latter it's estimated that about half of the data are missing (1.2M events), while 717702333 (2.5M events) and 717719650 (2.7M events) should be almost filled.

UPDATE: rePipe didn't result in more merged events.  The runs were finalized as is, i.e. with the gaps.

2023-09-29
23093000[123], (?230929009?)L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
7177139582023-09-29
230930008, (?230929009?)L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
7177023332023-09-29
230929008, 23093000[12], (?230929007?)L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
710506750mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report any gaps.2023-07-08
23070800[89]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
709339013somehow the xrootd scratch area vanished before cleanupCompleteRun. The run needs to be repiped from the HalfPipe.2023-06-2422:41:52 - 00:16:33 (+1)23062500[12]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
708257402mergeEvt reports 8 missing datagrams in epu1. Gapfinder doesn't report any gaps.2023-06-12
23031200[56]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
700230335mergeEvt reports 5 missing datagrams in epu0.  Gapfinder doesn't report any gaps.2023-02-11
230311009L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
697691485mergeEvt reports 7 missing datagrams in epu1.2023-02-10
23021000[45]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
694642846verify found a datagram gap.  Actually already mergeEvt found this gap of 12 datagrams in epu1, but somehow the run was labeled as complete.  A HalfPipe repipe leaves one missing datagram, decompression error. Finalized with one datagram gap.2023-01-05
230105010, 230106001L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
693360593003.mergeIndices failed and was processed last. When mergeEvt ran it didn't merge in the datagrams of 004.2022-12-22
22122200[234]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
687193623mergeReconChunks failed with a bus error but exit code 0. The merged recon file was not copied to xrootd but cleanupChunks removed all temporary files.2022-10-11
22101101[123]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
687119520mergeEvt reports 6 missing datagrams in epu0, gapfinder doesn't report any gaps.2022-10-10
221010013L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
686035262dl's 220928003 and 004 came in together. It was not possible to disentangle both. 220928003.mergeIndices fails.2022-09-28
22092800[34]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
68602????2022-09-28
220928003L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
6860226852022-09-28
22092800[23]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
685255472mergeEvt reports 7 missing datagrams in epu0, gapfinder doesn't report any gaps.2022-09-19
22091900[45]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
682569294Aug 19 (PDT) there were issues with a faulty nfs disk. This run was merged badly.2022-08-1902:34:53 - 03:45:0322081900[45]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
682556983Aug 19 (PDT) there were issues with a faulty nfs disk. This run was merged badly.2022-08-1823:09:42 - 00:29:45 (+1)22081900[34]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
680740571mergeEvt reports 5 missing datagrams in epu1, gapfinder doesn't report any gaps.2022-07-2822:36:10 - 23:58:0122072900[12]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
676577929mergeEvt reports 5 missing datagrams in epu0, gapfinder doesn't report any gaps.2022-06-10
22061001[01]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
675367448mergeEvt reports 8 missing datagram in epu0, additionally verify finds 7 missing datagrams in epu1.  However gapfinder doesn't report any gaps.2022-05-2718:04:07 - 19:28:29220527012, 220528001L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
674884484mergeEvt reports 1 missing datagram in epu1, gapfinder doesn't report any gaps.  MakeEvt reports a decompression error, finalizing with one datagram gap.2022-05-22
2205220[567]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
674878528mergeEvt reports several gaps with missing datagrams in epu1, gapfinder doesn't report any gaps.  MakeEvt reports decompression errors, finalizing with 14 scattered datagram gaps.2022-05-22
22052200[45]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
674872543mergeEvt reports several gaps with missing datagrams in epu1, gapfinder doesn't report any gaps.  MakeEvt reports decompression errors, finalizing with 16 missing datagrams scattered over 14 gaps.2022-05-22
22052200[34]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
674052578mergeEvt reports 5 missing datagrams in epu1, gapfinder doesn't report any gaps.2022-05-12
220512010L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
671827689verify reported missing data. Indeed also mergeEvt reports 5 missing datagrams in epu0, but somehow the run was labeled as complete.2022-04-16
220416014L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
671754063missing datagrams at the end of epu1. gapfinder doesn't report any gaps.2022-04-16
22041600[23]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
670363411Dls 220331005 and 006 came in within the same second, and got mangled. 006 passed, but 005 fails at HalfPipe.mergeIndices. Needs to be repiped completely.2022-03-31
22033100[25]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
670357708Dls 220331005 and 006 came in within the same second, and got mangled. 006 passed, but 005 fails at HalfPipe.mergeIndices. Needs to be repiped completely.2022-03-31
22033100[345]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
661643650Highly fragmented run (138 chunks). Needs a "real" repipe.2021-12-2021:54:09 - 23:28:5821122000[124]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
656744433Fall-out of the 211024 hypervisor failure. EPU0 is complete but EPU1 has a gap of ~5000 datagrams.  Roll back of HalfPipe didn't work!  Repipe did. But there was a gap in the magic7 file → second repipe
2021-10-24
21102400[56]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
649557766Highly fragmented run (>400 chunks). Needs a "real" repipe.2021-08-02
21080200[234]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
648918184mergeEvt reports a gap of 5 datagrams in epu0. Gapfinder doesn't report any gaps.2021-07-2515:03:03 - 16:13:0521072501[01]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
648862027mergeEvt reports a gap of 5 datagrams in epu1. Gapfinder doesn't report any gaps.2021-07-2423:27:06 - 00:59:49[+1]21072500[12]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
645896054One datagram in the inaccessible part of the SSR. The run will stay incomplete.2021-06-20
21062001[12]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
645331200mergeEvt reports a gap of 9 datagrams in epu1. Gapfinder doesn't report any gaps.2021-06-14
21061400[34]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
644150495mergeEvt reports a gap of 9 datagrams in epu0. Gapfinder doesn't report any gaps.2021-05-31
21053100[78]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)

Runs from May 25, 2020 onward (L1Proc-5.7), backward sorted

Run NumberReasonDateTimeDlsL1ProcGR versionrePipeL1Proc
640254671mergeEvt reports a gap of 8 datagrams in epu1. Gapfinder doesn't report any gaps.2021-04-1608:31:10-10:06:0021041600[78]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
637031869mergeEvt reports a gap of 5 datagrams in epu1. Gapfinder doesn't report any gaps.2021-03-10
210310003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
636598553missing datagrams at the end of epu0. Gapfinder doesn't report any gaps.2021-03-05
210305003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
635205107Verify reports a datagram gap in epu1. Also none of the mergeEvt sees the run as complete. However somehow it got labeled "Complete" in L1Proc. Gapfinder doesn't report any gaps.2021-02-16
210216015, 210217001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
633363581One chunk has one event only.  Roll-back of HalfPipe.210126012 reproduced the one-event chunk.  A "real" repipe worked.2021-01-26
21012601[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
632964783Missing data from the inaccessible part of the SSR. The run will stay incomplete.2021-01-2123:33:02-01:07:56(+1)21012200[16]L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
630962587Verify reports a datagram gap in epu0. Also none of the mergeEvt sees the run as complete. However somehow it got labeled "Complete" in L1Proc.  Gapfinder doesn't report any gaps.2020-12-29
20122901[012]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
627448410CAL FIFO overflow problem: digitization.201119003.627448410.3566554 fails with "3 CalDigis are missing readouts for Event: 4141638 Apid: 956 Event Index: 122309"2020-11-19
20111900[34]L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
626729954Highly fragmented run (>66 chunks). doRun/scanXxx processes fail. Moreover it's spread over 7 deliveries.2020-11-1019:39:13-21:14:0420111001[345], 20111100[1237]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
625160754mergeEvt: 6 missing datagrams in apid 956, gapfinder doesn't report any gaps2020-10-23
201023010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
624820000switch from hequs to the bullets2020-10-19





624516636mergeEvt: 6 missing datagrams in apid 956, gapfinder doesn't report any gaps2020-10-16
20101600[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
623917505mergeEvt: 7 missing datagrams in apid 9562020-10-0906:25:04-08:02:32201009007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
622822397mergeEvt: 5 missing datagrams in apid 9572020-09-2614:13:16-15:48:0620092601[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
622211988verify finds a gap of 3 datagrams in EPU0, and of 4 in EPU12020-09-19
20091901[02]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
622149260mergeEvt: 4 datagrams missing at the start of apid 9562020-09-1819:14:19-20:49:09200919001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621835509mergeEvt: incomplete apid 956, probably 7 datagrams missing at the end (apid 957 has 7 more)2020-09-15
200915005L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621824100mergeEvt: 8 missing datagrams in apid 9572020-09-15
200915003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621806986mergeEvt: 6 missing datagrams in apid 9572020-09-14
2009{14013,15001}L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621664373mergeEvt: 1 non closed datagram at the end of apid 957, gapfinder: no datagram gaps for 2020-09-132020-09-13
200913004L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621612709mergeEvt: 5 missing datagrams in apid 957, gapfinder: no datagram gaps for 2020-09-122020-09-1214:11:48-15:27:3220091201[012]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621600789mergeEvt: 5 missing datagrams in apid 957, gapfinder: no datagram gaps for 2020-09-122020-09-1210:53:08-12:01:40200912009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621435528mergeEvt: 5 missing datagrams in apid 956, gapfinder: no datagram gaps for 2020-09-102020-09-1012:58:47-14:10:3020091001[01]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621236535mergeEvt: 6 missing datagrams in apid 956, gapfinder: there was a gap an hour earlier, retransmitted2020-09-0805:42:14-07:17:0420090800[56]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621081007mergeEvt: 5 missing datagrams in apid 956, gapfinder: no gaps for this run2020-09-0610:30:06-11:43:23200906010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620894265Michael screwed it up by accidentally rolling back the finished doRun.200904006.6208942652020-09-04
20090400[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620460723mergeEvt: 6 missing datagrams in apid 957, gapfinder: no gaps for 2020-08-302020-08-3006:12:02-07:46:5320083000[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620375156missing datagrams2020-08-29
200829007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620369451caused 200829007.mergeIndices to fail because of an extra evt file. Repipe created identical evt files, thus L1Proc will not be rolled back, the run just needs to be finalized.2020-08-29
20082900[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
619313054mergeEvt: >1400 missing datagrams in both apids at the beginning of the run, gapfinder: no gaps around midnight of 2020-08-302020-08-1623:24:13-00:33:2020081700[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
618823233mergeEvt: 6 missing datagrams in apid 956, gapfinder: no gaps for the morning of 2020-08-112020-08-1107:20:32-08:36:4220073000[68]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
618805328mergeEvt: 8 missing datagrams in apid 957, gapfinder: no gaps for the morning of 2020-08-112020-08-1102:22:07-03:31:2620081100[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
617777656mergeEvt reports 8 missing datagrams in apid 956, but gapfinder doesn't report any gaps for 2020-07-302020-07-30
20073000[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
617190465mergeEvt reports 8 missing datagrams in apid 957, but gapfinder doesn't report any gaps for 2020-07-232020-07-23
200723007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615811484mergeEvt reports 6 missing datagrams, but gapfinder doesn't report any gaps2020-07-0710:44:43-12:19:342007070{09,10}L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615599718mergeEvt reports 6 missing datagrams, but gapfinder doesn't report any gaps2020-07-0404 23:55:17-05 01:06:3420070500[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615508121one digitization fails: "... 2 CalDigis are missing readouts for Event ...".2020-07-0322:28:40-23:37:3320070314, 200704001L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
615073887missing datagrams. Gapfinder doesn't report any gaps for the evening of 2020-06-28.2020-06-2821:51:26-23:06:4220062900[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
614967205mergeEvt reports 6 data gaps in apid 957. Gapfinder doesn't report any gaps 2020-06-27.2020-06-2716:13:24-17:48:1520062701[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
614162399mergeEvt reports 5 datagaps in apid 956. However gapfinder doesn't find any gaps.2020-06-18
20061800[89]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612445752The run had data gaps, which were supposed to be filled by a redump. Unfortunately one piece was in the inaccessible part of the SSR. The run will stay incomplete.2020-05-29
20052900[89]L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
612183340HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. There were several data gaps 2020-05-26. At least some were filled.2020-05-2610:55:39-12:30:30200526007, 20052700[125]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612171931HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. There were several data gaps 2020-05-26. At least some were filled.2020-05-2607:45:30-09:20:2120052600[567], 200527001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612166226HalfPipe labeled this run as "InComplete". MergeEvt said it's complete. However when repiping 200526005 (for 612171931) mergeIndices relabeled it, and failed with an error message hinting to this run. MK: the error was due to an evt file with slight different name. However, rolling back the HalfPipes resulted in exactly the same evt file contents. Thus, I didn't roll back L1Proc.2020-05-2606:10:25-07:14:1620052600[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
611880377HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. Gapfinder doesn't report a gap for the duration of the run.2020-05-2222:46:16-23:58:37200522015, 20052300[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)

Runs from December 2018 onward (L1Proc-5.7)

Run Number

Reason

Date

TimeDls

L1Proc

GR version

rePipe

L1Proc

565721985Botched attempt at a repipe, repiped and marked as Complete2018-12-05

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
565942990stuck "in progress", repiped and marked as Complete2018-12-08

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

(tick)

566022500HP "in progress", repiped and marked as Complete2018-12-09

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
566235626HP "in progress", repiped and marked as Complete 2018-12-11

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
567301141stuck "in progress", repiped and marked as Complete2018-12-23

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

567598970

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
567581889

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
569069307in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-13


010 011L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
569213577

in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-15
006 007L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572617673HP "in progress", repiped and marked as Complete 2019-02-23

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572853518stuck "in progress". Leftover chunkToken. Fixed.2019-02-26

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
574781844"in progress", incomplete data, missing datagrams2019-03-20

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575044322"in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized.2019-03-23
013 014L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575762229"in progress", incomplete data, missing datagrams2019-03-31

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
576305031Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized.2019-04-07

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
576658880"in progress", incomplete data2019-04-11

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

578530168

corrupted data in monitoring2019-05-02

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
580907445in progress, mergeIndices reports 5 missing datagrams2019-05-30

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
582073028in progress, mergeIndices reports missing datagrams2019-06-13

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
584730839in progress, complete data, missing datagrams2019-07-13

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585187986

This was actually complete. A botched rollback of halfPipe (190719003) produced leftover chunkTokens. Set to Complete and finalized.

2019-07-19

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585193691Both halfPipe and RePipe failed at delivering a complete set of data (this included a botched rollback of halfPipe 190719003). Set to Incomplete (due to 11 missing datagrams) and finalized.2019-07-19

L1Proc-5.7GlastRelease-20-10-04-gr02(error)(tick)
585535991in progress, complete data, missing datagrams2019-07-23

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585690096in progress, missing datagrams2019-07-24

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586076888see thread. However it seems the recon file is complete. Reprocessing complete.  Everything seems to have worked including the cleanupCompleteRun process.2019-07-29

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586893790bunch of tasks in "Waiting" state preventing a rollback.  Once that is cleared a rollback should hopefully fix everything2019-08-07

L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
586905200see thread . Seems the run is complete and can be just finalized. After checking, I've finalized this run as it looks fine -TS. MEM: same as 588742217, run was *not* finalized. 1. remove chunktokens as needed, 2. rollback checkRun, 3. make sure that cleanupCompleteRun is executed and recon file is merged. This is fine now. 2019-08-07

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
587042253Several failed tasks with errors of the form: "IOError: Can't stage in <filename>".  Looks like it was an xrootd hiccup.  Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete2019-08-09 12:46

L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
587104373

This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success.  There is a chunk token directory but no token.  However, the verify log complains about closing reasons for both EUP0 and EPU1. I'm think that means we need to repipe everything.

2019-08-10 6:54

L1Proc-5.7GlastRelease-20-10-04-gr02

(tick)

(tick)
588742217Run is incomplete according to the halfPipe. TS: I think this is okay as I don't see any errors in the verify log. MEM: run wasn't finalized because of leftover chunktokens. Please make sure that cleanupCompleteRun is executed before claiming success. This is fine now.2019-08-29

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
590065315Run still Incomplete in L1.  Looks like cleanupCompleteRun didn't get triggered.2019-09-13

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
591931054 failure in checkRun 191005003.591931054.  2 missing datagram.2019-10-05

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
5952403499 missing datagrams2019-11-12

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595405654missing datagrams2019-11-14

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595576882missing datagrams.  In progress2019-11-16

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595963106Missing datagrams due to decompression errors- Both halfPipe and rePipe report 6789 orphans (which translate to 53 missing datagrams). Set to Incomplete and finalized.2019-11-2017:18:24 - 18:32:18

191120013 191121001

L1Proc-5.7GlastRelease-20-10-04-gr02 (error) (tick)
596299562HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-11-24 21:30-24:002019-11-2414:46:00 - 16:01:21

191124009

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
596916997missing datagrams2019-12-01

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
597315084HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-12-06 05:00-14:002019-12-0608:51:23 - 10:06:10191206009 010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598223809HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-162019-12-1621:16:48 - 22:51:39191217001 002L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598581385HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2100:36:24 - 01:53:08191221003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598645966HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2118:32:45 - 20:07:36191221009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598758794HalfPipe reports "inProgress"2019-12-2305:39:57 - 08:44:23191223003 004L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599272563HalfPipe reports "inProgress".  Datagram errors2019-12-292:22:25 - 4:58:04191229002 003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599746995HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2020-01-032020-01-0312:23:14 - 13:58:05200103009 010 011L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599809968HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2020-01-042020-01-0405:52:47 - 07:14:52200103009 010 011L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
600328891HalfPipe reports "inProgress".  Datagram errors2020-01-1010:43:13200110005L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
601777922HalfPipe reports "inProgress", but mergeEvt doesn't report any datagram errors!  Gapfinder doesn't report any data gaps for 2020-01-27.2020-01-2702:07:06 - 03:41:57200127002 003 004L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
601783627HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-27.2020-01-2700:32:01 - 02:06:55200127004 005 006L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
601890911HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for the morning of 2020-01-28.2020-01-2807:55:10 - 09:04:44200128006 008L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
602153762HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-31. No decompression errors in makeEvt.2020-01-3108:56:01 - 10:07:38200131007L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
603637679HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-02-17. No decompression errors in makeEvt.2020-02-1713:07:57 - 14:42:49200217009 010L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
603974262Run is complete.  mergeEvt.200221009.603974262 reports 6 missing datagrams. gapfinder doesn't report any data gaps for the time interval.2020-02-2110:37:41 - 12:12:31200221009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
604573262

HalfPipe reports "inProgress".  Missing datagrams. Gapfinder doesn't report any data gaps for 2020-02-28.

2020-02-2809:01:01 - 10:35:52200228008 - 009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
605908182HalfPipe reports "inProgress".  Missing datagrams. Gapfinder doesn't report any data gaps for 2020-03-15.2020-03-1519:49:41 - 21:24:31200315002L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
607059914Highly fragmented run (231 chunks). doRun/scanXxx processes fail.2020-03-28
200328005, 006, 008L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
609843485HalfPipe reports "InProgress". Gapfinder doesn't report any data gaps of 2020-04-29.2020-04-29
200429007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610323623mergeIndices labeled this run as Incomplete, but mergeEvt as Complete! MK: Repipe resulted in the same chunks. Checking the old merit file didn't reveal any gaps within the run. Thus I relabeled the run to "Complete" and rolled-back checkRun.2020-05-05
200505001, 002L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610711547mergeEvt reports missing datagrams. Gapfinder doesn't report any data gaps for 2020-05-09.2020-05-0910:05:46 - 11:23:53200509008, 009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610791211HalfPipe labels the run as Incomplete, mergeEvt reports missing datagrams. Gapfinder doesn't report any data gaps for 2020-05-10.2020-05-1008:13:30 - 09:30:17200510006, 007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610968381mergeEvt reports missing datagrams2020-05-12
200512007, 008L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610996774mergeEvt reports missing datagrams2020-05-12
200512014, 015L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
611162210mergeEvt reports missing datagrams2020-05-14
200514011, 012L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
611782769HalfPipe labels the run as "Incomplete". MergeEvt reports missing datagrams. Gapfinder only reports a gap after 21:19.2020-05-2119:39:28 - 20:50:13200522001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)

How to use the RePipe facility

/afs/slac/g/glast/ground/bin/pipeline createStream --define "downlinkID=990101001,runID=237063408" RePipe
/nfs/farm/g/glast/u28/RePipe/990101001 will contain the RePiped output (should this have a --stream <run number> flag in the command for clarity in the monitoring pages?)

if you set an environment variable, rn, to the run number (i.e. setenv rn <run number>) you can do the following command to start the repipe and have the output in the format expected for the instruction linked to below.

/afs/slac/g/glast/ground/bin/pipeline createStream --stream $rn --define "downlinkID=$rn,runID=$rn" RePipe

More details: Instructions for rolling back a halfPipe stream and susbsequent L1 stream and Reprocessing runs from a RePiped job

Runs before December 2018 but after July 27 2011 (L1Proc-5.6 and earlier)

Run Number

Reason

Date

L1Proc

GR version

rePipe

L1(Re)Proc

Export
369152685 ACQ Complete, L1 not finalized. Fixed in P300x.2012-09-12L1Proc-4.0GR-17-35-24-gr35
4.0
478706331 ACQ Complete, L1 not finalized. Fixed in P300x.2016-03-03L1Proc-5.4GR-20-10-02
5.4
478756313ACQ Complete, L1 not finalized. Fixed in P300x.2016-03-04L1Proc-5.4

GR-20-10-02

5.4
504919670Leap Second Run. Set to Incomplete and finalized.2016-12-31L1Proc-5.6

GR-20-10-04-gr01

(error)
5.6
508046861ACQ InProgress. Repiped and reprocessed.2017-02-09L1Proc-5.6

GR-20-10-04-gr01

5.7
512115535ACQ Complete, L1 Complete/Failed. Fixed in P300x.2017-03-25L1Proc-5.6

GR-20-10-04-gr01

5.6
516617903ACQ Complete, L1 Complete/Failed. Fixed in P300x. 2017-05-16L1Proc-5.6

GR-20-10-04-gr01

5.6
531419768ACQ Complete, L1 Complete , recon on xrootd (root://fermi-gpfs05//glast/Data/Flight/Level1/LPA/prod/5.6/recon/r0531419768_v000_recon.root) but not in datacat. Run was reprocessed correctly in P300x. No follow-up needed.2017-11-03L1Proc-5.6GR-20-10-04-gr01
5.6
534500370ACQ Complete, L1 Complete/Failed. Fixed in P300x. 2017-12-09L1Proc-5.6

GR-20-10-04-gr01

5.6
549415721 ACQ Complete, L1 Complete , recon on xrootd (root://fermi-gpfs05//glast/Data/Flight/Level1/LPA/prod/5.6/recon/r0549415721_v000_recon.root) but not in datacat. Run was reprocessed correctly in P300x. No follow-up needed. 2018-05-31L1Proc-5.6GR-20-10-04-gr01
5.6
553318321ACQ Complete, L1 Failed. Repiped and reprocessed.2018-07-16L1Proc-5.6

GR-20-10-04-gr01

5.7
554179852ACQ Complete, L1 Failed. Repiped and reprocessed. 2018-07-25L1Proc-5.6

GR-20-10-04-gr01

5.7
554396875 ACQ Complete, cleanupCompleteRun.getTimes terminated by reaper. Run was reprocessed correctly in P300x. No follow-up needed. 2018-07-27L1Proc-5.6GR-20-10-04-gr01
5.6
555737292ACQ Complete, L1 InProgress. Finalized.2018-08-12L1Proc-5.6

GR-20-10-04-gr01

5.6
557026900ACQ Complete, L1 InProgress. Finalized.2018-08-27L1Proc-5.6

GR-20-10-04-gr01

5.6
559965007Truncated Run. Set to Incomplete and finalized. 2018-09-30L1Proc-5.6

GR-20-10-04-gr01

(error)
5.6
562253064Missing Data. Set to Incomplete and finalized. Missing data sent by MOC 9/24/19, reprocessed with 5.7 - versions numbers generated are less than the v305 version from the reprocessing so ingest is not happening at the FSSC. File version edited by hand and re-exported.
2018-10-26L1Proc-5.6

GR-20-10-04-gr01

(tick)
5.7
562258990This run was missing entirely. Retrieved by MOC 9/24/19 and reprocessed with 5.7.2018-10-26L1Proc-5.6

GR-20-10-04-gr01

(tick)5.7 (tick)
562264544Missing Data. Set to Incomplete and finalized. Missing data sent by MOC 9/24/19, reprocessed with 5.7 - versions numbers generated are less than the v305 version from the reprocessing so ingest is not happening at the FSSC. File version edited by hand and re-exported.
2018-10-27L1Proc-5.6

GR-20-10-04-gr01

(tick)5.7 (tick)

564602913

Repiped, reprocessed with 5.7, re-exported.2018-11-22L1Proc-5.6

GR-20-10-04-gr01

5.7 (tick)(tick)

Runs from LEO through 2009

Run Number

Reason

Date

FSW

L1Proc

GR version

rePipe

L1Proc

237911263

silent NFS read error

2008-07-16

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

237916725

silent NFS read error

2008-07-16

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

(tick)

238008393

silent NFS read error

2008-07-17

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

(tick)

238014123

silent NFS read error

2008-07-17

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238207772

silent NFS read error

2008-07-20

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238759073

silent NFS read error

2008-07-26

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238856333

silent NFS read error

2008-07-26

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

239125611

silent NFS read error

2008-07-30

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

239129222

silent NFS read error

2008-07-30

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

241125159

1 missing Datagram

2008-08-22

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

241491796

missing chunk in in gcr

2008-08-27

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

241501554

1 missing Datagram

2008-08-27

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

243170470

afs cache corruption?

2008-09-15

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243197713

afs cache corruption?

2008-09-15

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243307762

2 missing Datagrams

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243313738

2 missing Datagrams

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243336617

afs cache corruption?

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243491490

4 missing Datagrams

2008-09-19

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243873727

afs cache corruption?

2008-09-23

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243926731

afs cache corruption?

2008-09-24

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

244143826

afs cache corruption?

2008-09-26

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

244167563

afs cache corruption?

2008-09-27

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

245667862

irretrievable packets missing from two datagrams

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(error)

(error)

245673843

This one was complete, but a retransmission changed its status

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245685781

all packets available; repiping will produce complete data-set

2008-10-14

B1-1-3

L1Proc-1.69

Version-v15r47p1

(tick)

(tick)

245691461

all packets available; repiping will produce complete data-set

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245697061

This one was already complete

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245702790

irretrievable packet missing from one datagram

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(error)

(error)

246390295

Merging problems

2008-10-22

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

246405636

Merging problems

2008-10-22

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

246447791

Merging problems

2008-10-23

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

253539991

Delivery problems ?

2009-01-13

B1-1-3

L1Proc-1.70

Version-v15r47p1

(tick)

(tick)

260478011

1 Missing datagram ???

2009-04-03

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261148032

Insane number of chunks

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261154012

Insane number of chunks

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261406015

1 Missing datagram ???

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

269048825

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269054554

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269060283

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269066012

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269071741

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269075743

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269081879

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269087937

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269093942

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269099929

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

272627642

1 bad event (seg fault)

2009-08-22

B1-1-3

L1Proc-1.75

Version-v15r47p12

(tick)

(tick)

  • No labels

8 Comments

  1. Tom Stephens

    Run 564602913 was marked as successfully done on this page but in looking at it, there is no directory for that run in /nfs/farm/g/glast/u28/RePipe and the L1Proc for it is still failed.  Plus the FSSC received files for it last week that had a shortened time range.

    Was this done outside the standard data monitoring tasks?  If so, how to we reconcile the different time ranges?  I've removed the check marks from that run but if they are correct (somewhere other than the main data processing page) feel free to put them back but we need to explain why the time ranges are different from the original files sent to the FSSC.

    1. Tom Stephens

      And looking at the RePipe task, it was run on that run back in December which is probably the botched repipe attempt mentioned in the notes (and probably done by me) so I don't actually think this has been repiped. 

      1. Tom Stephens

        I'm going to RePipe this run and see if I can do it all from start to finish.

        1. Maria Elena Monzani

          Hi Tom. Please leave 564602913 alone. I redid it last week and re-exported to FSSC. It was not updated with the latest file because it would have overwritten the P305 version of the files. There was a change in format and L1Proc version which we cannot revert. If something is marked as done, it means that it's done. Please don't undo other people's work.

  2. Maria Elena Monzani

    I was in the middle of fixing a bunch of the old runs. Sorry I didn't put my name on them.

    1. Maria Elena Monzani

      Run 565721985 was not merged correctly (the two halves of the run were never merged with each other). I'm going to redo it again.

      1. Maria Elena Monzani

        OrbitPlot for run 565721985, showing the missing data.
        One should check the logs for the latest verify process in each run to make sure that there are no missing data at the beginning/end of each run.

        1. Maria Elena Monzani

          Run 565721985 is fixed now.