Versions Compared

Old Version 249

changes.mady.by.user Michael Kuss

Saved on

compared with

New Version 456

changes.mady.by.user Michael Kuss

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
maxLevel3

Runs from

...

May 31, 2021 onward (L1Proc-5.

...

9), backward sorted

Run NumberReasonDateTimeDlsL1ProcGR versionrePipeL1Proc
565721985Botched attempt at a repipe, repiped and marked as Complete2018-12-05 
736930436mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report a gap for the duration of this run.2024-05-09
24050900[89]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07(tick)(tick)
565942990stuck "in progress", repiped and marked as Complete2018-12-08 
730458995240224011.mergeEvt ran twice, creating an incomplete evt file on the second try.2024-02-24
24022401[01]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07(tick)(tick)
566022500HP "in progress", repiped and marked as Complete2018-12-09 
729548905Highly fragmented run. 91, 15, and 61 chunks. This requires task RePipe.2024-02-14
24021400[345]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07(tick)(tick)
566235626HP "in progress", repiped and marked as Complete 2018-12-11 
727336262mergeEvt reports 6 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-19
240119003
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07(tick)(tick)
567301141stuck "in progress", repiped and marked as Complete2018-12-23 
727324474mergeEvt reports missing datagrams at the end of epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-19
240119002
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)

567598970

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27 
726579387mergeEvt reports 5 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-10
24011000[78]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
(error)
(tick)(tick)
567581889

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27 
726452942mergeEvt reports missing datagrams at the end of epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-0900:29:01 - 01:41:00240109004
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07(error)
(tick)
(error)
569069307in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-13

 010 011
723956756mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report a gap for the duration of this run.2023-12-1103:05:55 - 04:39:3623121100[23]L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
569213577

in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-15 
723939304mergeEvt reports 5 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2023-12-1022:15:02 - 23:32:30231211001
006 007
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
572617673HP "in progress", repiped and marked as Complete 2019-02-23 
723865764mergeEvt reports 6 missing datagrams in epu1. Gapfinder doesn't report any gaps.2023-12-10
23121000[34]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
572853518stuck "in progress". Leftover chunkToken. Fixed.2019-02-26 
723665322mergeEvt reports 6 missing datagrams in epu1. Gapfinder found a gap for this run and sent just one retransmit request.2023-12-0718:08:41 - 19:19:2223120101[02]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
574781844"in progress", incomplete data, missing datagrams2019-03-20 
723103279mergeEvt reports about 100 missing datagrams at the end of epu1. Gapfinder doesn't report any gaps.2023-12-01
23120100[78]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
717719650

Dls 230929007 and 009 resulted from passes the FOT called "ratty".  All run pieces in these dls were highly fragmented, and L1 didn't process them.  Redumps were triggered, but only run 717708266 was completely filled in dl 230930008, 717713958 partially.  Of the latter it's estimated that about half of the data are missing (1.2M events), while 717702333 (2.5M events) and 717719650 (2.7M events) should be almost filled.

UPDATE: rePipe didn't result in more merged events.  The runs were finalized as is, i.e. with the gaps.

2023-09-29
23093000[123], (?230929009?)L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
7177139582023-09-29
230930008, (?230929009?)L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
7177023332023-09-29
230929008, 23093000[12], (?230929007?)L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
710506750mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report any gaps.2023-07-08
23070800[89]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
709339013somehow the xrootd scratch area vanished before cleanupCompleteRun. The run needs to be repiped from the HalfPipe.2023-06-2422:41:52 - 00:16:33 (+1)23062500[12]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
708257402mergeEvt reports 8 missing datagrams in epu1. Gapfinder doesn't report any gaps.2023-06-12
23031200[56]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
700230335mergeEvt reports 5 missing datagrams in epu0.  Gapfinder doesn't report any gaps.2023-02-11
230311009L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
697691485mergeEvt reports 7 missing datagrams in epu1.2023-02-10
23021000[45]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
694642846verify found a datagram gap.  Actually already mergeEvt found this gap of 12 datagrams in epu1, but somehow the run was labeled as complete.  A HalfPipe repipe leaves one missing datagram, decompression error. Finalized with one datagram gap.2023-01-05
230105010, 230106001L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
693360593003.mergeIndices failed and was processed last. When mergeEvt ran it didn't merge in the datagrams of 004.2022-12-22
22122200[234]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
687193623mergeReconChunks failed with a bus error but exit code 0. The merged recon file was not copied to xrootd but cleanupChunks removed all temporary files.2022-10-11
22101101[123]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
687119520mergeEvt reports 6 missing datagrams in epu0, gapfinder doesn't report any gaps.2022-10-10
221010013L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
686035262dl's 220928003 and 004 came in together. It was not possible to disentangle both. 220928003.mergeIndices fails.2022-09-28
22092800[34]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
68602????2022-09-28
220928003L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
6860226852022-09-28
22092800[23]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
685255472mergeEvt reports 7 missing datagrams in epu0, gapfinder doesn't report any gaps.2022-09-19
22091900[45]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
682569294Aug 19 (PDT) there were issues with a faulty nfs disk. This run was merged badly.2022-08-1902:34:53 - 03:45:0322081900[45]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
682556983Aug 19 (PDT) there were issues with a faulty nfs disk. This run was merged badly.2022-08-1823:09:42 - 00:29:45 (+1)22081900[34]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
680740571mergeEvt reports 5 missing datagrams in epu1, gapfinder doesn't report any gaps.2022-07-2822:36:10 - 23:58:0122072900[12]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
676577929mergeEvt reports 5 missing datagrams in epu0, gapfinder doesn't report any gaps.2022-06-10
22061001[01]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
675367448mergeEvt reports 8 missing datagram in epu0, additionally verify finds 7 missing datagrams in epu1.  However gapfinder doesn't report any gaps.2022-05-2718:04:07 - 19:28:29220527012, 220528001L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
674884484mergeEvt reports 1 missing datagram in epu1, gapfinder doesn't report any gaps.  MakeEvt reports a decompression error, finalizing with one datagram gap.2022-05-22
2205220[567]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
674878528mergeEvt reports several gaps with missing datagrams in epu1, gapfinder doesn't report any gaps.  MakeEvt reports decompression errors, finalizing with 14 scattered datagram gaps.2022-05-22
22052200[45]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
674872543mergeEvt reports several gaps with missing datagrams in epu1, gapfinder doesn't report any gaps.  MakeEvt reports decompression errors, finalizing with 16 missing datagrams scattered over 14 gaps.2022-05-22
22052200[34]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
674052578mergeEvt reports 5 missing datagrams in epu1, gapfinder doesn't report any gaps.2022-05-12
220512010L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
671827689verify reported missing data. Indeed also mergeEvt reports 5 missing datagrams in epu0, but somehow the run was labeled as complete.2022-04-16
220416014L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
671754063missing datagrams at the end of epu1. gapfinder doesn't report any gaps.2022-04-16
22041600[23]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
670363411Dls 220331005 and 006 came in within the same second, and got mangled. 006 passed, but 005 fails at HalfPipe.mergeIndices. Needs to be repiped completely.2022-03-31
22033100[25]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
670357708Dls 220331005 and 006 came in within the same second, and got mangled. 006 passed, but 005 fails at HalfPipe.mergeIndices. Needs to be repiped completely.2022-03-31
22033100[345]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
661643650Highly fragmented run (138 chunks). Needs a "real" repipe.2021-12-2021:54:09 - 23:28:5821122000[124]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
656744433Fall-out of the 211024 hypervisor failure. EPU0 is complete but EPU1 has a gap of ~5000 datagrams.  Roll back of HalfPipe didn't work!  Repipe did. But there was a gap in the magic7 file → second repipe
2021-10-24
21102400[56]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
649557766Highly fragmented run (>400 chunks). Needs a "real" repipe.2021-08-02
21080200[234]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
648918184mergeEvt reports a gap of 5 datagrams in epu0. Gapfinder doesn't report any gaps.2021-07-2515:03:03 - 16:13:0521072501[01]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
648862027mergeEvt reports a gap of 5 datagrams in epu1. Gapfinder doesn't report any gaps.2021-07-2423:27:06 - 00:59:49[+1]21072500[12]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
645896054One datagram in the inaccessible part of the SSR. The run will stay incomplete.2021-06-20
21062001[12]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
645331200mergeEvt reports a gap of 9 datagrams in epu1. Gapfinder doesn't report any gaps.2021-06-14
21061400[34]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
644150495mergeEvt reports a gap of 9 datagrams in epu0. Gapfinder doesn't report any gaps.2021-05-31
21053100[78]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)

Runs from May 25, 2020 onward (L1Proc-5.7), backward sorted

Run NumberReasonDateTimeDlsL1ProcGR versionrePipeL1Proc
640254671mergeEvt reports a gap of 8 datagrams in epu1. Gapfinder doesn't report any gaps.2021-04-1608:31:10-10:06:0021041600[78]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
637031869mergeEvt reports a gap of 5 datagrams in epu1. Gapfinder doesn't report any gaps.2021-03-10
210310003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
636598553missing datagrams at the end of epu0. Gapfinder doesn't report any gaps.2021-03-05
210305003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
635205107Verify reports a datagram gap in epu1. Also none of the mergeEvt sees the run as complete. However somehow it got labeled "Complete" in L1Proc. Gapfinder doesn't report any gaps.2021-02-16
210216015, 210217001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
633363581One chunk has one event only.  Roll-back of HalfPipe.210126012 reproduced the one-event chunk.  A "real" repipe worked.2021-01-26
21012601[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
632964783Missing data from the inaccessible part of the SSR. The run will stay incomplete.2021-01-2123:33:02-01:07:56(+1)21012200[16]L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
630962587Verify reports a datagram gap in epu0. Also none of the mergeEvt sees the run as complete. However somehow it got labeled "Complete" in L1Proc.  Gapfinder doesn't report any gaps.2020-12-29
20122901[012]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
627448410CAL FIFO overflow problem: digitization.201119003.627448410.3566554 fails with "3 CalDigis are missing readouts for Event: 4141638 Apid: 956 Event Index: 122309"2020-11-19
20111900[34]L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
626729954Highly fragmented run (>66 chunks). doRun/scanXxx processes fail. Moreover it's spread over 7 deliveries.2020-11-1019:39:13-21:14:0420111001[345], 20111100[1237]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
625160754mergeEvt: 6 missing datagrams in apid 956, gapfinder doesn't report any gaps2020-10-23
201023010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
624820000switch from hequs to the bullets2020-10-19





624516636mergeEvt: 6 missing datagrams in apid 956, gapfinder doesn't report any gaps2020-10-16
20101600[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
623917505mergeEvt: 7 missing datagrams in apid 9562020-10-0906:25:04-08:02:32201009007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
622822397mergeEvt: 5 missing datagrams in apid 9572020-09-2614:13:16-15:48:0620092601[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
622211988verify finds a gap of 3 datagrams in EPU0, and of 4 in EPU12020-09-19
20091901[02]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
622149260mergeEvt: 4 datagrams missing at the start of apid 9562020-09-1819:14:19-20:49:09200919001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621835509mergeEvt: incomplete apid 956, probably 7 datagrams missing at the end (apid 957 has 7 more)2020-09-15
200915005L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621824100mergeEvt: 8 missing datagrams in apid 9572020-09-15
200915003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621806986mergeEvt: 6 missing datagrams in apid 9572020-09-14
2009{14013,15001}L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621664373mergeEvt: 1 non closed datagram at the end of apid 957, gapfinder: no datagram gaps for 2020-09-132020-09-13
200913004L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621612709mergeEvt: 5 missing datagrams in apid 957, gapfinder: no datagram gaps for 2020-09-122020-09-1214:11:48-15:27:3220091201[012]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621600789mergeEvt: 5 missing datagrams in apid 957, gapfinder: no datagram gaps for 2020-09-122020-09-1210:53:08-12:01:40200912009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621435528mergeEvt: 5 missing datagrams in apid 956, gapfinder: no datagram gaps for 2020-09-102020-09-1012:58:47-14:10:3020091001[01]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621236535mergeEvt: 6 missing datagrams in apid 956, gapfinder: there was a gap an hour earlier, retransmitted2020-09-0805:42:14-07:17:0420090800[56]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621081007mergeEvt: 5 missing datagrams in apid 956, gapfinder: no gaps for this run2020-09-0610:30:06-11:43:23200906010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620894265Michael screwed it up by accidentally rolling back the finished doRun.200904006.6208942652020-09-04
20090400[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620460723mergeEvt: 6 missing datagrams in apid 957, gapfinder: no gaps for 2020-08-302020-08-3006:12:02-07:46:5320083000[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620375156missing datagrams2020-08-29
200829007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620369451caused 200829007.mergeIndices to fail because of an extra evt file. Repipe created identical evt files, thus L1Proc will not be rolled back, the run just needs to be finalized.2020-08-29
20082900[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
619313054mergeEvt: >1400 missing datagrams in both apids at the beginning of the run, gapfinder: no gaps around midnight of 2020-08-302020-08-1623:24:13-00:33:2020081700[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
618823233mergeEvt: 6 missing datagrams in apid 956, gapfinder: no gaps for the morning of 2020-08-112020-08-1107:20:32-08:36:4220073000[68]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
618805328mergeEvt: 8 missing datagrams in apid 957, gapfinder: no gaps for the morning of 2020-08-112020-08-1102:22:07-03:31:2620081100[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
617777656mergeEvt reports 8 missing datagrams in apid 956, but gapfinder doesn't report any gaps for 2020-07-302020-07-30
20073000[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
617190465mergeEvt reports 8 missing datagrams in apid 957, but gapfinder doesn't report any gaps for 2020-07-232020-07-23
200723007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615811484mergeEvt reports 6 missing datagrams, but gapfinder doesn't report any gaps2020-07-0710:44:43-12:19:342007070{09,10}L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615599718mergeEvt reports 6 missing datagrams, but gapfinder doesn't report any gaps2020-07-0404 23:55:17-05 01:06:3420070500[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615508121one digitization fails: "... 2 CalDigis are missing readouts for Event ...".2020-07-0322:28:40-23:37:3320070314, 200704001L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
615073887missing datagrams. Gapfinder doesn't report any gaps for the evening of 2020-06-28.2020-06-2821:51:26-23:06:4220062900[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
614967205mergeEvt reports 6 data gaps in apid 957. Gapfinder doesn't report any gaps 2020-06-27.2020-06-2716:13:24-17:48:1520062701[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
614162399mergeEvt reports 5 datagaps in apid 956. However gapfinder doesn't find any gaps.2020-06-18
20061800[89]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612445752The run had data gaps, which were supposed to be filled by a redump. Unfortunately one piece was in the inaccessible part of the SSR. The run will stay incomplete.2020-05-29
20052900[89]L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
612183340HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. There were several data gaps 2020-05-26. At least some were filled.2020-05-2610:55:39-12:30:30200526007, 20052700[125]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612171931HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. There were several data gaps 2020-05-26. At least some were filled.2020-05-2607:45:30-09:20:2120052600[567], 200527001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612166226HalfPipe labeled this run as "InComplete". MergeEvt said it's complete. However when repiping 200526005 (for 612171931) mergeIndices relabeled it, and failed with an error message hinting to this run. MK: the error was due to an evt file with slight different name. However, rolling back the HalfPipes resulted in exactly the same evt file contents. Thus, I didn't roll back L1Proc.2020-05-2606:10:25-07:14:1620052600[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
611880377HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. Gapfinder doesn't report a gap for the duration of the run.2020-05-2222:46:16-23:58:37200522015, 20052300[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)

Runs from December 2018 onward (L1Proc-5.7)

Run Number

Reason

Date

TimeDls

L1Proc

GR version

rePipe

L1Proc

565721985Botched attempt at a repipe, repiped and marked as Complete2018-12-05

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
565942990stuck "in progress", repiped and marked as Complete2018-12-08

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

(tick)

566022500HP "in progress", repiped and marked as Complete2018-12-09

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
566235626HP "in progress", repiped and marked as Complete 2018-12-11

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
567301141stuck "in progress", repiped and marked as Complete2018-12-23

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)

567598970

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
567581889

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
569069307in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-13


010 011L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
569213577

in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-15
006 007L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
572617673HP "in progress", repiped and marked as Complete 2019-02-23

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
572853518stuck "in progress". Leftover chunkToken. Fixed.2019-02-26

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
574781844"in progress", incomplete data, missing datagrams2019-03-20

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
575044322"in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized.2019-03-23
013 014L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
575762229"in progress", incomplete data, missing datagrams2019-03-31

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
576305031Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized.2019-04-07

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
576658880"in progress", incomplete data2019-04-11

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)

578530168

corrupted data in monitoring2019-05-02
575044322"in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized.2019-03-23 013 014


L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Modified
(tick)
575762229
580907445
"
in progress
", incomplete data,
, mergeIndices reports 5 missing datagrams2019-
03-31  
05-30

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Modified
(tick)
576305031Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized.
582073028in progress, mergeIndices reports missing datagrams2019-
04-07  
06-13

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)
(tick)(tick)
576658880
584730839
"
in progress
"
,
incomplete
complete data, missing datagrams2019-
04-11  
07-13

L1Proc-5.7GlastRelease-20-10-04-gr02
Image Removed
(tick)(tick)

578530168

585187986

This was actually complete. A botched rollback of halfPipe (190719003) produced leftover chunkTokens. Set to Complete and finalized.

corrupted data in monitoring

2019-
05-02  
07-19

L1Proc-5.7GlastRelease-20-10-04-gr02
Image Removed
(tick)(tick)
580907445in progress, mergeIndices reports 5 missing datagrams2019-05-30 
585193691Both halfPipe and RePipe failed at delivering a complete set of data (this included a botched rollback of halfPipe 190719003). Set to Incomplete (due to 11 missing datagrams) and finalized.2019-07-19
 


L1Proc-5.7GlastRelease-20-10-04-gr02
Image Removed
(error)(tick)
582073028
585535991in progress, complete data,
mergeIndices reports
missing datagrams2019-
06-13  
07-23

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
584730839
585690096in progress
, complete data
, missing datagrams2019-07-
13  
24

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585187986This was actually complete. A botched rollback of halfPipe (190719003) produced leftover chunkTokens. Set to Complete and finalized
586076888see thread. However it seems the recon file is complete. Reprocessing complete.  Everything seems to have worked including the cleanupCompleteRun process.2019-07-
19  
29

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585193691Both halfPipe and RePipe failed at delivering a complete set of data (this included a botched rollback of halfPipe 190719003). Set to Incomplete (due to 11 missing datagrams) and finalized.2019-07-19 
586893790bunch of tasks in "Waiting" state preventing a rollback.  Once that is cleared a rollback should hopefully fix everything2019-08-07
 


L1Proc-5.7GlastRelease-20-10-04-gr02
(error)(tick)
N/A(tick)
586905200see thread . Seems the run is complete and can be just finalized. After checking, I've finalized this run as it looks fine -TS. MEM: same as 588742217, run was *not* finalized. 1. remove chunktokens as needed, 2. rollback checkRun, 3. make sure that cleanupCompleteRun is executed and recon file is merged. This is fine now. 2019-08-07
585535991in progress, complete data, missing datagrams2019-07-23  


L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585690096in progress, missing datagrams2019-07-24  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)586076888see thread. However it seems the recon file is complete. Reprocessing complete.  Everything seems to have worked including the cleanupCompleteRun process.2019-07-29 
587042253Several failed tasks with errors of the form: "IOError: Can't stage in <filename>".  Looks like it was an xrootd hiccup.  Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete2019-08-09 12:46
 


L1Proc-5.7GlastRelease-20-10-04-gr02
(tick)
N/A(tick)
586893790bunch of tasks in "Waiting" state preventing a rollback.  Once that is cleared a rollback should hopefully fix everything2019-08-07 
587104373

This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success.  There is a chunk token directory but no token.  However, the verify log complains about closing reasons for both EUP0 and EPU1. I'm think that means we need to repipe everything.

2019-08-10 6:54
 


L1Proc-5.7GlastRelease-20-10-04-gr02
N/A

(tick)

(tick)
586905200see thread . Seems the run is complete and can be just finalized. After checking, I've finalized this run as it looks fine -TS. MEM: same as 588742217, run was *not* finalized. 1. remove chunktokens as needed, 2. rollback checkRun, 3.
588742217Run is incomplete according to the halfPipe. TS: I think this is okay as I don't see any errors in the verify log. MEM: run wasn't finalized because of leftover chunktokens. Please make sure that cleanupCompleteRun is executed
and recon file is merged
before claiming success. This is fine now.2019-08-
07  
29

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
587042253Several failed tasks with errors of the form: "IOError: Can't stage in <filename>".  Looks like it was an xrootd hiccup.  Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete2019-08-09 12:46 
590065315Run still Incomplete in L1.  Looks like cleanupCompleteRun didn't get triggered.2019-09-13
 


L1Proc-5.7GlastRelease-20-10-04-gr02
N/A
(tick)(tick)
587104373This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success.  There is a chunk token directory but no token.  However, the verify log complains about closing reasons for both EUP0 and EPU1. I'm think that means we need to repipe everything
591931054 failure in checkRun 191005003.591931054.  2 missing datagram.2019-
08-10 6:54  
10-05

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
588742217Run is incomplete according to the halfPipe. TS: I think this is okay as I don't see any errors in the verify log. MEM: run wasn't finalized because of leftover chunktokens. Please make sure that cleanupCompleteRun is executed before claiming success. This is fine now.2019-08-29 
5952403499 missing datagrams2019-11-12

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595405654missing datagrams2019-11-14

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595576882missing datagrams.  In progress2019-11-16
 


L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
590065315Run still Incomplete in L1.  Looks like cleanupCompleteRun didn't get triggered
595963106Missing datagrams due to decompression errors- Both halfPipe and rePipe report 6789 orphans (which translate to 53 missing datagrams). Set to Incomplete and finalized.2019-
09
11-
13 
2017:18:24 - 18:32:18

191120013 191121001

 

L1Proc-5.7GlastRelease-20-10-04-gr02
(tick)
(error) (tick)
591931054 failure in checkRun 191005003.591931054.  2 missing datagram.2019-10-05 
596299562HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-11-24 21:30-24:002019-11-2414:46:00 - 16:01:21

191124009

 

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595240349
596916997
9
missing datagrams2019
-11
-12
  
-01

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595405654
597315084HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-12-06 05:00-14:00
missing datagrams
2019-
11
12-
14
0608:51:23 - 10:06:10191206009 010
  
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595576882missing datagrams.  In progress2019-11-16 
598223809HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-162019-12-1621:16:48 - 22:51:39191217001 002
 
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595963106Missing datagrams due to decompression errors- Both halfPipe and rePipe report 6789 orphans (which translate to 53 missing datagrams). Set to Incomplete and finalized.2019-11-2017:18:24 - 18:32:18
598581385HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2100:36:24 - 01:53:08191221003
191120013 191121001
L1Proc-5.7GlastRelease-20-10-04-gr02
(error)
(tick)(tick)
596299562
598645966HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-
11
12-
24
21
:30-24:00
2019-
11
12-
24
21
14
18:
46
32:
00
45 -
16
20:
01
07:
21
36
191124009
191221009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
596916997
598758794HalfPipe reports "inProgress"
missing datagrams
2019-12-
01
2305:39:57 - 08:44:23191223003 004
  
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
597315084
599272563HalfPipe reports "inProgress"
, gapfinder doesn't report any data gaps for 2019-12-06 05:00-14:00
.  Datagram errors2019-12-
06
29
08
2:
51
22:
23
25 -
10
4:
06
58:
10
04
191206009 010
191229002 003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598223809
599746995HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for
2019
2020-
12
01-
16
03
2019
2020-
12
01-
16
03
21
12:
16
23:
48
14 -
22
13:
51
58:
39
05200103009 010 011
191217001 002
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598581385
599809968HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for
2019
2020-
12
01-
21
04
2019
2020-
12
01-
21
04
00
05:
36
52:
24
47 -
01
07:
53
14:
08
52
191221003
200103009 010 011L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598645966
600328891HalfPipe reports "inProgress"
, missing datagrams, gapfinder doesn't report any data gaps for 2019-12-21
.  Datagram errors2020-01-1010:43:13200110005
2019-12-2118:32:45 - 20:07:36191221009
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598758794
601777922HalfPipe reports "inProgress"
, but mergeEvt doesn't report any datagram errors!  Gapfinder doesn't report any data gaps for 2020-01-27.2020-01-2702:07:06 - 03:41:57200127002 003
2019-12-2305:39:57 - 08:44:23191223003
004L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
599272563
601783627HalfPipe reports "inProgress"
.  Datagram errors
.  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-27.2020-01-2700:32:01 - 02:06:55200127004 005 006
2019-12-292:22:25 - 4:58:04191229002 003
L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
599746995
601890911HalfPipe reports "inProgress"
, missing datagrams, gapfinder
.  Missing datagrams.  Gapfinder doesn't report any data gaps for the morning of 2020-01-
03
28.2020-01-
03
28
12
07:
23
55:
14
10 -
13
09:
58
04:
05
44200128006 008
200103009 010 011
L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
599809968
602153762HalfPipe reports "inProgress"
, missing datagrams, gapfinder
.  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-
04
31. No decompression errors in makeEvt.2020-01-
04
31
05
08:
52
56:
47
01 - 10:07:
14:52
38200131007
200103009 010 011
L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
600328891HalfPipe reports "inProgress".  Datagram errors2020-01-1010:43:13

603637679HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-02-17. No decompression errors in makeEvt.2020-02-1713:07:57 - 14:42:49200217009 010
200110005
L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
601777922HalfPipe reports "inProgress", but mergeEvt doesn't report any datagram errors!  Gapfinder
603974262Run is complete.  mergeEvt.200221009.603974262 reports 6 missing datagrams. gapfinder doesn't report any data gaps for
2020-01-27
the time interval.2020-
01
02-
27
21
02
10:
07
37:
06
41 -
03
12:
41
12:
57
31
200127002 003 004
200221009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
601783627
604573262

HalfPipe reports "inProgress".  Missing datagrams.

 

Gapfinder doesn't report any data gaps for 2020-

01

02-

27

28.

2020-
01
02-
27
28
00
09:
32
01:01 -
02
10:
06
35:
55
52200228008 - 009
200127004 005 006
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
601890911
605908182HalfPipe reports "inProgress".  Missing datagrams.
 
Gapfinder doesn't report any data gaps for
the morning of
2020-
01
03-
28
15.2020-
01
03-
28
15
07
19:
55
49:
10
41 -
09
21:
04
24:
44
31
200128006 008
200315002L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
602153762HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-31. No decompression errors in makeEvt
607059914Highly fragmented run (231 chunks). doRun/scanXxx processes fail.2020-
01
03-
3108:56:01 - 10:07:38
28
200328005, 006, 008
200131007
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
603637679
609843485HalfPipe reports "
inProgress
InProgress"
.  Missing datagrams
.
 
Gapfinder doesn't report any data gaps
for
of 2020-
02-17. No decompression errors in makeEvt
04-29.2020-
02
04-
1713:07:57 - 14:42:49
29
200429007
200217009 010
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
603974262Run is complete.  mergeEvt.200221009.603974262 reports 6 missing datagrams. gapfinder doesn't report any data gaps for the time interval.2020-02-2110:37:41 - 12:12:31
610323623mergeIndices labeled this run as Incomplete, but mergeEvt as Complete! MK: Repipe resulted in the same chunks. Checking the old merit file didn't reveal any gaps within the run. Thus I relabeled the run to "Complete" and rolled-back checkRun.2020-05-05
200505001, 002
200221009
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
604573262
610711547
HalfPipe reports "inProgress".  Missing
mergeEvt reports missing datagrams. Gapfinder doesn't report any data gaps for 2020-
02
05-
28
09.2020-
02
05-
28
09
09
10:
01
05:
01
46 -
10
11:
35
23:
52
53
200228008 -
200509008, 009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
605908182
610791211HalfPipe
reports "inProgress".  Missing
labels the run as Incomplete, mergeEvt reports missing datagrams. Gapfinder doesn't report any data gaps for 2020-
03
05-
15
10.2020-
03
05-
15
10
19
08:
49
13:
41
30 -
21
09:
24
30:
31
17
200315002
200510006, 007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
607059914Highly fragmented run (231 chunks). doRun/scanXxx processes fail.
610968381mergeEvt reports missing datagrams2020-
03
05-
28 
12
200512007
200328005, 006
, 008L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
609843485
610996774
HalfPipe reports "InProgress". GapFinder doesn't report any data gaps of 2020-04-29.
mergeEvt reports missing datagrams2020-
04
05-
29 
12
200512014, 015
200429007
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610323623mergeIndices labeled this run as Incomplete, but mergeEvt as Complete!
611162210mergeEvt reports missing datagrams2020-05-
05200505001
14
 

200514011,
002
012L1Proc-5.7GlastRelease-20-10-04-gr02(tick)
(error)Michael
(tick)
610711547mergeEvt reports missing datagrams. Gapfinder doesn't report any data gaps for 2020-05-09
611782769HalfPipe labels the run as "Incomplete". MergeEvt reports missing datagrams. Gapfinder only reports a gap after 21:19.2020-05-
09
21
10
19:
05
39:
46
28 -
11
20:
23
50:
53
13
200509008, 009
200522001L1Proc-5.7GlastRelease-20-10-04-gr02
(error) Michael
(tick)
(error) Michael
(tick)

How to use the RePipe facility

...

Runs before December 2018 but after July 27 2011 (L1Proc-5.6 and earlier)

Run Number

Reason

Date

L1Proc

GR version

rePipe

L1(Re)Proc

Export
369152685 ACQ Complete, L1 not finalized. Fixed in P300x.2012-09-12L1Proc-4.0GR-17-35-24-gr35
Image Modified
4.0 Image Modified
Image Modified
478706331 ACQ Complete, L1 not finalized. Fixed in P300x.2016-03-03L1Proc-5.4GR-20-10-02
Image Modified
5.4 Image Modified
Image Modified
478756313ACQ Complete, L1 not finalized. Fixed in P300x.2016-03-04L1Proc-5.4

GR-20-10-02

Image Modified
5.4 Image Modified
Image Modified
504919670Leap Second Run. Set to Incomplete and finalized.2016-12-31L1Proc-5.6

GR-20-10-04-gr01

(error)
5.6 Image Modified
Image Modified
508046861ACQ InProgress. Repiped and reprocessed.2017-02-09L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.7 Image Modified
Image Modified
512115535ACQ Complete, L1 Complete/Failed. Fixed in P300x.2017-03-25L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.6 Image Modified
Image Modified
516617903ACQ Complete, L1 Complete/Failed. Fixed in P300x. 2017-05-16L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.6 Image Modified
Image Modified
531419768ACQ Complete, L1 Complete , recon on xrootd (root://fermi-gpfs05//glast/Data/Flight/Level1/LPA/prod/5.6/recon/r0531419768_v000_recon.root) but not in datacat. Run was reprocessed correctly in P300x. No follow-up needed.2017-11-03L1Proc-5.6GR-20-10-04-gr01
Image Modified
5.6 Image Modified
Image Modified
534500370ACQ Complete, L1 Complete/Failed. Fixed in P300x. 2017-12-09L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.6 Image Modified
Image Modified
549415721 ACQ Complete, L1 Complete , recon on xrootd (root://fermi-gpfs05//glast/Data/Flight/Level1/LPA/prod/5.6/recon/r0549415721_v000_recon.root) but not in datacat. Run was reprocessed correctly in P300x. No follow-up needed. 2018-05-31L1Proc-5.6GR-20-10-04-gr01
Image Modified
5.6 Image Modified
Image Modified
553318321ACQ Complete, L1 Failed. Repiped and reprocessed.2018-07-16L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.7 Image Modified
Image Modified
554179852ACQ Complete, L1 Failed. Repiped and reprocessed. 2018-07-25L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.7 Image Modified
Image Modified
554396875 ACQ Complete, cleanupCompleteRun.getTimes terminated by reaper. Run was reprocessed correctly in P300x. No follow-up needed. 2018-07-27L1Proc-5.6GR-20-10-04-gr01
Image Modified
5.6 Image Modified
Image Modified
555737292ACQ Complete, L1 InProgress. Finalized.2018-08-12L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.6 Image Modified
Image Modified
557026900ACQ Complete, L1 InProgress. Finalized.2018-08-27L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.6 Image Modified
Image Modified
559965007Truncated Run. Set to Incomplete and finalized. 2018-09-30L1Proc-5.6

GR-20-10-04-gr01

(error)
5.6 Image Modified
Image Modified
562253064Missing Data. Set to Incomplete and finalized. Missing data sent by MOC 9/24/19, reprocessed with 5.7 - versions numbers generated are less than the v305 version from the reprocessing so ingest is not happening at the FSSC. File version edited by hand and re-exported.
2018-10-26L1Proc-5.6

GR-20-10-04-gr01

(tick)
5.7 Image Modified
Image Modified
562258990This run was missing entirely. Retrieved by MOC 9/24/19 and reprocessed with 5.7.2018-10-26L1Proc-5.6

GR-20-10-04-gr01

(tick)5.7 (tick)
Image Modified
562264544Missing Data. Set to Incomplete and finalized. Missing data sent by MOC 9/24/19, reprocessed with 5.7 - versions numbers generated are less than the v305 version from the reprocessing so ingest is not happening at the FSSC. File version edited by hand and re-exported.
2018-10-27L1Proc-5.6

GR-20-10-04-gr01

(tick)5.7 (tick)
Image Modified

564602913

Repiped, reprocessed with 5.7, re-exported.2018-11-22L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.7 (tick)(tick)

Runs from LEO through 2009

Run Number

Reason

Date

FSW

L1Proc

GR version

rePipe

L1Proc

237911263

silent NFS read error

2008-07-16

B1-0-8

L1Proc-1.63

Version-v15r39

Image Modified

(tick)

237916725

silent NFS read error

2008-07-16

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

(tick)

238008393

silent NFS read error

2008-07-17

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

(tick)

238014123

silent NFS read error

2008-07-17

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238207772

silent NFS read error

2008-07-20

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238759073

silent NFS read error

2008-07-26

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238856333

silent NFS read error

2008-07-26

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

239125611

silent NFS read error

2008-07-30

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

239129222

silent NFS read error

2008-07-30

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

241125159

1 missing Datagram

2008-08-22

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

241491796

missing chunk in in gcr

2008-08-27

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

241501554

1 missing Datagram

2008-08-27

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

243170470

afs cache corruption?

2008-09-15

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243197713

afs cache corruption?

2008-09-15

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243307762

2 missing Datagrams

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243313738

2 missing Datagrams

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243336617

afs cache corruption?

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243491490

4 missing Datagrams

2008-09-19

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243873727

afs cache corruption?

2008-09-23

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243926731

afs cache corruption?

2008-09-24

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

244143826

afs cache corruption?

2008-09-26

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

244167563

afs cache corruption?

2008-09-27

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

245667862

irretrievable packets missing from two datagrams

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(error)

(error)

245673843

This one was complete, but a retransmission changed its status

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245685781

all packets available; repiping will produce complete data-set

2008-10-14

B1-1-3

L1Proc-1.69

Version-v15r47p1

(tick)

(tick)

245691461

all packets available; repiping will produce complete data-set

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245697061

This one was already complete

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245702790

irretrievable packet missing from one datagram

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(error)

(error)

246390295

Merging problems

2008-10-22

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

246405636

Merging problems

2008-10-22

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

246447791

Merging problems

2008-10-23

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

253539991

Delivery problems ?

2009-01-13

B1-1-3

L1Proc-1.70

Version-v15r47p1

(tick)

(tick)

260478011

1 Missing datagram ???

2009-04-03

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261148032

Insane number of chunks

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261154012

Insane number of chunks

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261406015

1 Missing datagram ???

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

269048825

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269054554

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269060283

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269066012

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269071741

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269075743

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269081879

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269087937

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269093942

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269099929

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

272627642

1 bad event (seg fault)

2009-08-22

B1-1-3

L1Proc-1.75

Version-v15r47p12

(tick)

(tick)