Versions Compared

Old Version 214

changes.mady.by.user Tom Stephens

Saved on

compared with

New Version Current

changes.mady.by.user Michael Kuss

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
maxLevel3

Runs from

...

May 31, 2021 onward (L1Proc-5.

...

9), backward sorted

Run NumberReasonDateTimeDlsL1ProcGR versionrePipeL1Proc
565721985Botched attempt at a repipe, repiped and marked as Complete2018-12-05 
747364498MergeEvt reports missing datagrams at the end of epu1. Gapfinder doesn't report any gaps.2024-09-07
24090700[234]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07(tick)(tick)
565942990stuck "in progress", repiped and marked as Complete2018-12-08 
743660123MergeEvt found a gap of 6 datagrams in epu1. Nevertheless the halfPipe labeled the run as complete.  Fortunately L1Proc.verify choked on it. Gapfinder doesn't report any gaps.2024-07-26
240716004
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07(tick)(tick)
566022500HP "in progress", repiped and marked as Complete2018-12-09 
742704925Puzzling.  MergeEvt seems to have found all datagrams, but nevertheless labels both epu's as incomplete.  L1Proc.verify claims that one datagram was dropped in epu0.2024-07-15
24071500[34]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07(tick)(tick)
566235626HP "in progress", repiped and marked as Complete 2018-12-11  
741148235MergeEvt reports 6 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2024-06-27
24062700[34]L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07(tick)(tick)
567301141stuck "in progress", repiped and marked as Complete2018-12-23 
738549526L1Proc.verify claims to have found gaps of 5 datagrams each in both epus. mergeEvt reports the gap for epu0 but claims epu1 is complete. The HalfPipe labeled the run complete. Gapfinder doesn't report a gap for the duration of this run.2024-05-31
24053100[45]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)

567598970

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27 
738549526mergeEvt reports 7 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2024-05-28
24052800[123]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
(error)
(tick)(tick)
567581889

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27 
736930436mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report a gap for the duration of this run.2024-05-09
24050900[89]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
(error)
(tick)(tick)
569069307in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-13

 010 011
730458995240224011.mergeEvt ran twice, creating an incomplete evt file on the second try.2024-02-24
24022401[01]L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
569213577

in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-15 006 007
729548905Highly fragmented run. 91, 15, and 61 chunks. This requires task RePipe.2024-02-14
24021400[345]L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
572617673HP "in progress", repiped and marked as Complete 2019-02-23 
727336262mergeEvt reports 6 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-19
240119003
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
572853518stuck "in progress". Leftover chunkToken. Fixed.2019-02-26  
727324474mergeEvt reports missing datagrams at the end of epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-19
240119002L1Proc-5.9
L1Proc-5.7
GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
574781844"in progress", incomplete data, missing datagrams2019-03-20 
726579387mergeEvt reports 5 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-10
24011000[78]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
575044322"in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized.2019-03-23 013 014
726452942mergeEvt reports missing datagrams at the end of epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-0900:29:01 - 01:41:00240109004L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(error)
(tick)
(error)
575762229"in progress", incomplete data, missing datagrams2019-03-31 
723956756mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report a gap for the duration of this run.2023-12-1103:05:55 - 04:39:3623121100[23]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
576305031Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized.2019-04-07 
723939304mergeEvt reports 5 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2023-12-1022:15:02 - 23:32:30231211001
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
(error)
(tick)(tick)
576658880"in progress", incomplete data2019-04-11 
723865764mergeEvt reports 6 missing datagrams in epu1. Gapfinder doesn't report any gaps.2023-12-10
23121000[34]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)

578530168

corrupted data in monitoring2019-05-02 
723665322mergeEvt reports 6 missing datagrams in epu1. Gapfinder found a gap for this run and sent just one retransmit request.2023-12-0718:08:41 - 19:19:2223120101[02]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
580907445in progress, mergeIndices reports 5 missing datagrams2019-05-30 
723103279mergeEvt reports about 100 missing datagrams at the end of epu1. Gapfinder doesn't report any gaps.2023-12-01
23120100[78]
 
L1Proc-5.
7
9GlastRelease-20-10-04-
gr02
gr07
Image Removed
(tick)(tick)
717719650

Dls 230929007 and 009 resulted from passes the FOT called "ratty".  All run pieces in these dls were highly fragmented, and L1 didn't process them.  Redumps were triggered, but only run 717708266 was completely filled in dl 230930008, 717713958 partially.  Of the latter it's estimated that about half of the data are missing (1.2M events), while 717702333 (2.5M events) and 717719650 (2.7M events) should be almost filled.

UPDATE: rePipe didn't result in more merged events.  The runs were finalized as is, i.e. with the gaps.

2023-09-29
23093000[123], (?230929009?)L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
7177139582023-09-29
230930008, (?230929009?)L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
7177023332023-09-29
230929008, 23093000[12], (?230929007?)L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
710506750mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report any gaps.2023-07-08
23070800[89]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
709339013somehow the xrootd scratch area vanished before cleanupCompleteRun. The run needs to be repiped from the HalfPipe.2023-06-2422:41:52 - 00:16:33 (+1)23062500[12]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
708257402mergeEvt reports 8 missing datagrams in epu1. Gapfinder doesn't report any gaps.2023-06-12
23031200[56]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
700230335mergeEvt reports 5 missing datagrams in epu0.  Gapfinder doesn't report any gaps.2023-02-11
230311009L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
697691485mergeEvt reports 7 missing datagrams in epu1.2023-02-10
23021000[45]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
694642846verify found a datagram gap.  Actually already mergeEvt found this gap of 12 datagrams in epu1, but somehow the run was labeled as complete.  A HalfPipe repipe leaves one missing datagram, decompression error. Finalized with one datagram gap.2023-01-05
230105010, 230106001L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
693360593003.mergeIndices failed and was processed last. When mergeEvt ran it didn't merge in the datagrams of 004.2022-12-22
22122200[234]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
687193623mergeReconChunks failed with a bus error but exit code 0. The merged recon file was not copied to xrootd but cleanupChunks removed all temporary files.2022-10-11
22101101[123]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
687119520mergeEvt reports 6 missing datagrams in epu0, gapfinder doesn't report any gaps.2022-10-10
221010013L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
686035262dl's 220928003 and 004 came in together. It was not possible to disentangle both. 220928003.mergeIndices fails.2022-09-28
22092800[34]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
68602????2022-09-28
220928003L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
6860226852022-09-28
22092800[23]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
685255472mergeEvt reports 7 missing datagrams in epu0, gapfinder doesn't report any gaps.2022-09-19
22091900[45]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
682569294Aug 19 (PDT) there were issues with a faulty nfs disk. This run was merged badly.2022-08-1902:34:53 - 03:45:0322081900[45]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
682556983Aug 19 (PDT) there were issues with a faulty nfs disk. This run was merged badly.2022-08-1823:09:42 - 00:29:45 (+1)22081900[34]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
680740571mergeEvt reports 5 missing datagrams in epu1, gapfinder doesn't report any gaps.2022-07-2822:36:10 - 23:58:0122072900[12]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
676577929mergeEvt reports 5 missing datagrams in epu0, gapfinder doesn't report any gaps.2022-06-10
22061001[01]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
675367448mergeEvt reports 8 missing datagram in epu0, additionally verify finds 7 missing datagrams in epu1.  However gapfinder doesn't report any gaps.2022-05-2718:04:07 - 19:28:29220527012, 220528001L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
674884484mergeEvt reports 1 missing datagram in epu1, gapfinder doesn't report any gaps.  MakeEvt reports a decompression error, finalizing with one datagram gap.2022-05-22
2205220[567]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
674878528mergeEvt reports several gaps with missing datagrams in epu1, gapfinder doesn't report any gaps.  MakeEvt reports decompression errors, finalizing with 14 scattered datagram gaps.2022-05-22
22052200[45]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
674872543mergeEvt reports several gaps with missing datagrams in epu1, gapfinder doesn't report any gaps.  MakeEvt reports decompression errors, finalizing with 16 missing datagrams scattered over 14 gaps.2022-05-22
22052200[34]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
674052578mergeEvt reports 5 missing datagrams in epu1, gapfinder doesn't report any gaps.2022-05-12
220512010L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
671827689verify reported missing data. Indeed also mergeEvt reports 5 missing datagrams in epu0, but somehow the run was labeled as complete.2022-04-16
220416014L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
671754063missing datagrams at the end of epu1. gapfinder doesn't report any gaps.2022-04-16
22041600[23]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
670363411Dls 220331005 and 006 came in within the same second, and got mangled. 006 passed, but 005 fails at HalfPipe.mergeIndices. Needs to be repiped completely.2022-03-31
22033100[25]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
670357708Dls 220331005 and 006 came in within the same second, and got mangled. 006 passed, but 005 fails at HalfPipe.mergeIndices. Needs to be repiped completely.2022-03-31
22033100[345]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
661643650Highly fragmented run (138 chunks). Needs a "real" repipe.2021-12-2021:54:09 - 23:28:5821122000[124]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
656744433Fall-out of the 211024 hypervisor failure. EPU0 is complete but EPU1 has a gap of ~5000 datagrams.  Roll back of HalfPipe didn't work!  Repipe did. But there was a gap in the magic7 file → second repipe
2021-10-24
21102400[56]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
649557766Highly fragmented run (>400 chunks). Needs a "real" repipe.2021-08-02
21080200[234]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
648918184mergeEvt reports a gap of 5 datagrams in epu0. Gapfinder doesn't report any gaps.2021-07-2515:03:03 - 16:13:0521072501[01]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
648862027mergeEvt reports a gap of 5 datagrams in epu1. Gapfinder doesn't report any gaps.2021-07-2423:27:06 - 00:59:49[+1]21072500[12]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
645896054One datagram in the inaccessible part of the SSR. The run will stay incomplete.2021-06-20
21062001[12]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
645331200mergeEvt reports a gap of 9 datagrams in epu1. Gapfinder doesn't report any gaps.2021-06-14
21061400[34]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
644150495mergeEvt reports a gap of 9 datagrams in epu0. Gapfinder doesn't report any gaps.2021-05-31
21053100[78]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)

Runs from May 25, 2020 onward (L1Proc-5.7), backward sorted

Run NumberReasonDateTimeDlsL1ProcGR versionrePipeL1Proc
640254671mergeEvt reports a gap of 8 datagrams in epu1. Gapfinder doesn't report any gaps.2021-04-1608:31:10-10:06:0021041600[78]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
637031869mergeEvt reports a gap of 5 datagrams in epu1. Gapfinder doesn't report any gaps.2021-03-10
210310003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
636598553missing datagrams at the end of epu0. Gapfinder doesn't report any gaps.2021-03-05
210305003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
635205107Verify reports a datagram gap in epu1. Also none of the mergeEvt sees the run as complete. However somehow it got labeled "Complete" in L1Proc. Gapfinder doesn't report any gaps.2021-02-16
210216015, 210217001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
633363581One chunk has one event only.  Roll-back of HalfPipe.210126012 reproduced the one-event chunk.  A "real" repipe worked.2021-01-26
21012601[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
632964783Missing data from the inaccessible part of the SSR. The run will stay incomplete.2021-01-2123:33:02-01:07:56(+1)21012200[16]L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
630962587Verify reports a datagram gap in epu0. Also none of the mergeEvt sees the run as complete. However somehow it got labeled "Complete" in L1Proc.  Gapfinder doesn't report any gaps.2020-12-29
20122901[012]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
627448410CAL FIFO overflow problem: digitization.201119003.627448410.3566554 fails with "3 CalDigis are missing readouts for Event: 4141638 Apid: 956 Event Index: 122309"2020-11-19
20111900[34]L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
626729954Highly fragmented run (>66 chunks). doRun/scanXxx processes fail. Moreover it's spread over 7 deliveries.2020-11-1019:39:13-21:14:0420111001[345], 20111100[1237]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
625160754mergeEvt: 6 missing datagrams in apid 956, gapfinder doesn't report any gaps2020-10-23
201023010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
624820000switch from hequs to the bullets2020-10-19





624516636mergeEvt: 6 missing datagrams in apid 956, gapfinder doesn't report any gaps2020-10-16
20101600[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
623917505mergeEvt: 7 missing datagrams in apid 9562020-10-0906:25:04-08:02:32201009007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
622822397mergeEvt: 5 missing datagrams in apid 9572020-09-2614:13:16-15:48:0620092601[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
622211988verify finds a gap of 3 datagrams in EPU0, and of 4 in EPU12020-09-19
20091901[02]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
622149260mergeEvt: 4 datagrams missing at the start of apid 9562020-09-1819:14:19-20:49:09200919001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621835509mergeEvt: incomplete apid 956, probably 7 datagrams missing at the end (apid 957 has 7 more)2020-09-15
200915005L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621824100mergeEvt: 8 missing datagrams in apid 9572020-09-15
200915003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621806986mergeEvt: 6 missing datagrams in apid 9572020-09-14
2009{14013,15001}L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621664373mergeEvt: 1 non closed datagram at the end of apid 957, gapfinder: no datagram gaps for 2020-09-132020-09-13
200913004L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621612709mergeEvt: 5 missing datagrams in apid 957, gapfinder: no datagram gaps for 2020-09-122020-09-1214:11:48-15:27:3220091201[012]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621600789mergeEvt: 5 missing datagrams in apid 957, gapfinder: no datagram gaps for 2020-09-122020-09-1210:53:08-12:01:40200912009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621435528mergeEvt: 5 missing datagrams in apid 956, gapfinder: no datagram gaps for 2020-09-102020-09-1012:58:47-14:10:3020091001[01]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621236535mergeEvt: 6 missing datagrams in apid 956, gapfinder: there was a gap an hour earlier, retransmitted2020-09-0805:42:14-07:17:0420090800[56]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621081007mergeEvt: 5 missing datagrams in apid 956, gapfinder: no gaps for this run2020-09-0610:30:06-11:43:23200906010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620894265Michael screwed it up by accidentally rolling back the finished doRun.200904006.6208942652020-09-04
20090400[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620460723mergeEvt: 6 missing datagrams in apid 957, gapfinder: no gaps for 2020-08-302020-08-3006:12:02-07:46:5320083000[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620375156missing datagrams2020-08-29
200829007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620369451caused 200829007.mergeIndices to fail because of an extra evt file. Repipe created identical evt files, thus L1Proc will not be rolled back, the run just needs to be finalized.2020-08-29
20082900[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
619313054mergeEvt: >1400 missing datagrams in both apids at the beginning of the run, gapfinder: no gaps around midnight of 2020-08-302020-08-1623:24:13-00:33:2020081700[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
618823233mergeEvt: 6 missing datagrams in apid 956, gapfinder: no gaps for the morning of 2020-08-112020-08-1107:20:32-08:36:4220073000[68]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
618805328mergeEvt: 8 missing datagrams in apid 957, gapfinder: no gaps for the morning of 2020-08-112020-08-1102:22:07-03:31:2620081100[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
617777656mergeEvt reports 8 missing datagrams in apid 956, but gapfinder doesn't report any gaps for 2020-07-302020-07-30
20073000[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
617190465mergeEvt reports 8 missing datagrams in apid 957, but gapfinder doesn't report any gaps for 2020-07-232020-07-23
200723007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615811484mergeEvt reports 6 missing datagrams, but gapfinder doesn't report any gaps2020-07-0710:44:43-12:19:342007070{09,10}L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615599718mergeEvt reports 6 missing datagrams, but gapfinder doesn't report any gaps2020-07-0404 23:55:17-05 01:06:3420070500[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615508121one digitization fails: "... 2 CalDigis are missing readouts for Event ...".2020-07-0322:28:40-23:37:3320070314, 200704001L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
615073887missing datagrams. Gapfinder doesn't report any gaps for the evening of 2020-06-28.2020-06-2821:51:26-23:06:4220062900[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
614967205mergeEvt reports 6 data gaps in apid 957. Gapfinder doesn't report any gaps 2020-06-27.2020-06-2716:13:24-17:48:1520062701[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
614162399mergeEvt reports 5 datagaps in apid 956. However gapfinder doesn't find any gaps.2020-06-18
20061800[89]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612445752The run had data gaps, which were supposed to be filled by a redump. Unfortunately one piece was in the inaccessible part of the SSR. The run will stay incomplete.2020-05-29
20052900[89]L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
612183340HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. There were several data gaps 2020-05-26. At least some were filled.2020-05-2610:55:39-12:30:30200526007, 20052700[125]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612171931HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. There were several data gaps 2020-05-26. At least some were filled.2020-05-2607:45:30-09:20:2120052600[567], 200527001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612166226HalfPipe labeled this run as "InComplete". MergeEvt said it's complete. However when repiping 200526005 (for 612171931) mergeIndices relabeled it, and failed with an error message hinting to this run. MK: the error was due to an evt file with slight different name. However, rolling back the HalfPipes resulted in exactly the same evt file contents. Thus, I didn't roll back L1Proc.2020-05-2606:10:25-07:14:1620052600[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
611880377HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. Gapfinder doesn't report a gap for the duration of the run.2020-05-2222:46:16-23:58:37200522015, 20052300[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)

Runs from December 2018 onward (L1Proc-5.7)

Run Number

Reason

Date

TimeDls

L1Proc

GR version

rePipe

L1Proc

565721985Botched attempt at a repipe, repiped and marked as Complete2018-12-05

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
565942990stuck "in progress", repiped and marked as Complete2018-12-08

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

(tick)

566022500HP "in progress", repiped and marked as Complete2018-12-09

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
566235626HP "in progress", repiped and marked as Complete 2018-12-11

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
567301141stuck "in progress", repiped and marked as Complete2018-12-23

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)

567598970

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
567581889

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
569069307in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-13


010 011L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
569213577

in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-15
006 007L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
572617673HP "in progress", repiped and marked as Complete 2019-02-23

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
572853518stuck "in progress". Leftover chunkToken. Fixed.2019-02-26

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
574781844"in progress", incomplete data, missing datagrams2019-03-20

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
575044322"in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized.2019-03-23
013 014L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
575762229"in progress", incomplete data, missing datagrams2019-03-31

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
576305031Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized.2019-04-07

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
576658880"in progress", incomplete data2019-04-11

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)

578530168

corrupted data in monitoring2019-05-02

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
580907445in progress, mergeIndices reports 5 missing datagrams2019-05-30

L1Proc-5.7

GlastRelease-20-10-04-gr02

Image Added
(tick)
582073028in progress, mergeIndices reports missing datagrams2019-06-13

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
584730839in progress, complete data, missing datagrams2019-07-13

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585187986

This was actually complete. A botched rollback of halfPipe (190719003) produced leftover chunkTokens. Set to Complete and finalized.

2019-07-19

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585193691Both halfPipe and RePipe failed at delivering a complete set of data (this included a botched rollback of halfPipe 190719003). Set to Incomplete (due to 11 missing datagrams) and finalized.2019-07-19

L1Proc-5.7GlastRelease-20-10-04-gr02(error)(tick)
585535991in progress, complete data, missing datagrams2019-07-23

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585690096in progress, missing datagrams2019-07-24

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586076888see thread. However it seems the recon file is complete. Reprocessing complete.  Everything seems to have worked including the cleanupCompleteRun process.2019-07-29

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586893790bunch of tasks in "Waiting" state preventing a rollback.  Once that is cleared a rollback should hopefully fix everything2019-08-07

L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
586905200see thread . Seems the run is complete and can be just finalized. After checking, I've finalized this run as it looks fine -TS. MEM: same as 588742217, run was *not* finalized. 1. remove chunktokens as needed, 2. rollback checkRun, 3. make sure that cleanupCompleteRun is executed and recon file is merged. This is fine now. 2019-08-07

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
587042253Several failed tasks with errors of the form: "IOError: Can't stage in <filename>".  Looks like it was an xrootd hiccup.  Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete2019-08-09 12:46

L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
587104373

This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success.  There is a chunk token directory but no token.  However, the verify log complains about closing reasons for both EUP0 and EPU1. I'm think that means we need to repipe everything.

2019-08-10 6:54

L1Proc-5.7GlastRelease-20-10-04-gr02

(tick)

(tick)
588742217Run is incomplete according to the halfPipe. TS: I think this is okay as I don't see any errors in the verify log. MEM: run wasn't finalized because of leftover chunktokens. Please make sure that cleanupCompleteRun is executed before claiming success. This is fine now.2019-08-29

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
590065315Run still Incomplete in L1.  Looks like cleanupCompleteRun didn't get triggered.2019-09-13

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
591931054 failure in checkRun 191005003.591931054.  2 missing datagram.2019-10-05

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
5952403499 missing datagrams2019-11-12

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595405654missing datagrams2019-11-14

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595576882missing datagrams.  In progress2019-11-16

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595963106Missing datagrams due to decompression errors- Both halfPipe and rePipe report 6789 orphans (which translate to 53 missing datagrams). Set to Incomplete and finalized.2019-11-2017:18:24 - 18:32:18

191120013 191121001

L1Proc-5.7GlastRelease-
582073028in progress, mergeIndices reports missing datagrams2019-06-13  L1Proc-5.7GlastRelease-
20-10-04-gr02
(tick)
(error) (tick)
584730839in progress, complete data, missing datagrams2019-07-13 
596299562HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-11-24 21:30-24:002019-11-2414:46:00 - 16:01:21

191124009

 

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585187986in progress, complete data, missing datagrams, problems with rolling back halfPipe.190719003
596916997missing datagrams2019-
07-19  
12-01

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)
(error)?
(tick)
585193691in progress, complete data, missing datagrams, problems with rolling back halfPipe.1907190032019-07-19 
597315084HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-12-06 05:00-14:002019-12-0608:51:23 - 10:06:10191206009 010
 
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)
(error) ?
(tick)
585535991in progress, complete data
598223809HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-162019-
07
12-
23 
1621:16:48 - 22:51:39191217001 002
 
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585690096
598581385HalfPipe reports "inProgress"
in progress
, missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-
07-24 
12-2100:36:24 - 01:53:08191221003
 
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586076888see thread. However it seems the recon file is complete. Reprocessing complete.  Everything seems to have worked including the cleanupCompleteRun process.2019-07-29 
598645966HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2118:32:45 - 20:07:36191221009
 
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586893790bunch of tasks in "Waiting" state preventing a rollback.  Once that is cleared a rollback should hopefully fix everything
598758794HalfPipe reports "inProgress"2019-
08
12-
07 
2305:39:57 - 08:44:23191223003 004
 
L1Proc-5.7GlastRelease-20-10-04-gr02
N/A
(tick)(tick)
586905200see thread . Seems the run is complete and can be just finalized. After checking, I've finalized this run as it looks fine -TS. MEM: same as 588742217, run was *not* finalized. 1. remove chunktokens as needed, 2. rollback checkRun, 3. make sure that cleanupCompleteRun is executed and recon file is merged. This is fine now. 2019-08-07 
599272563HalfPipe reports "inProgress".  Datagram errors2019-12-292:22:25 - 4:58:04191229002 003
 
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
587042253Several failed tasks with errors of the form: "IOError: Can't stage in <filename>".  Looks like it was an xrootd hiccup.  Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete2019-08-09 12:46 
599746995HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2020-01-032020-01-0312:23:14 - 13:58:05200103009 010 011
 
L1Proc-5.7GlastRelease-20-10-04-gr02
N/A
(tick)(tick)
587104373

This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success.  There is a chunk token directory but no token.  However, the verify log complains about closing reasons for both EUP0 and EPU1. I'm think that means we need to repipe everything.

2019-08-10 6:54 
599809968HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2020-01-042020-01-0405:52:47 - 07:14:52200103009 010 011L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
600328891HalfPipe reports "inProgress".  Datagram errors2020-01-1010:43:13200110005
 
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
588742217Run is incomplete according to the halfPipe. TS: I think this is okay as I don't see any errors in the verify log. MEM: run wasn't finalized because of leftover chunktokens. Please make sure that cleanupCompleteRun is executed before claiming success. This is fine now.2019-08-29 
601777922HalfPipe reports "inProgress", but mergeEvt doesn't report any datagram errors!  Gapfinder doesn't report any data gaps for 2020-01-27.2020-01-2702:07:06 - 03:41:57200127002 003 004
 
L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
590065315Run still Incomplete in L1.  Looks like cleanupCompleteRun didn't get triggered.2019-09-13 
601783627HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-27.2020-01-2700:32:01 - 02:06:55200127004 005 006
 
L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
591931054 failure in checkRun 191005003.591931054.  2 missing datagram.2019-10-05 
601890911HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for the morning of 2020-01-28.2020-01-2807:55:10 - 09:04:44200128006 008
 
L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
5952403499 missing datagrams2019-11-12 
602153762HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-31. No decompression errors in makeEvt.2020-01-3108:56:01 - 10:07:38200131007
 
L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
595405654missing datagrams2019-11-14  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)595576882missing datagrams.  In progress2019-11-16 
603637679HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-02-17. No decompression errors in makeEvt.2020-02-1713:07:57 - 14:42:49200217009 010
 
L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
595963106missing datagrams - The repiped run report more missing data than the original run. I'm at a loss on how to fix this one.2019-11-2017:18:24 - 18:32:18
603974262Run is complete.  mergeEvt.200221009.603974262 reports 6 missing datagrams. gapfinder doesn't report any data gaps for the time interval.2020-02-2110:37:41 - 12:12:31200221009
191120013 191121001
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)
(error)
(tick)
596299562
604573262

HalfPipe reports "inProgress"

, gapfinder

.  Missing datagrams. Gapfinder doesn't report any data gaps for

2019

2020-

11-24 21:30-24:00

02-28.

2020-02-2809:01:01 - 10:35:52200228008 - 009
2019-11-2414:46:00 - 16:01:21191124009
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
596916997missing datagrams2019-12-01 
605908182HalfPipe reports "inProgress".  Missing datagrams. Gapfinder doesn't report any data gaps for 2020-03-15.2020-03-1519:49:41 - 21:24:31200315002
 
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
597315084HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-12-06 05:00-14:002019-12-0608:51:23 - 10:06:10191206009 010
607059914Highly fragmented run (231 chunks). doRun/scanXxx processes fail.2020-03-28
200328005, 006, 008L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598223809
609843485HalfPipe reports "
inProgress", missing datagrams, gapfinder
InProgress". Gapfinder doesn't report any data gaps
for 2019
of 2020-
12
04-
16
29.
2019
2020-
12
04-
1621:16:48 - 22:51:39
29
200429007
191217001 002
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598581385HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2100:36:24 - 01:53:08
610323623mergeIndices labeled this run as Incomplete, but mergeEvt as Complete! MK: Repipe resulted in the same chunks. Checking the old merit file didn't reveal any gaps within the run. Thus I relabeled the run to "Complete" and rolled-back checkRun.2020-05-05
200505001, 002
191221003
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598645966
610711547
HalfPipe
mergeEvt reports
"inProgress",
missing datagrams
, gapfinder
. Gapfinder doesn't report any data gaps for
2019
2020-
12
05-
21
09.
2019
2020-
12
05-
21
09
18
10:
32
05:
45
46 -
20
11:
07
23:
36
53
191221009
200509008, 009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598758794
610791211HalfPipe
reports "inProgress"
labels the run as Incomplete, mergeEvt reports missing datagrams. Gapfinder doesn't report any data gaps for 2020-05-10.2020-05-1008:13:30 - 09:30:17200510006, 007
2019-12-2305:39:57 - 08:44:23191223003 004
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599272563
610968381
HalfPipe reports "inProgress".  Datagram errors2019-12-292:22:25 - 4:58:04
mergeEvt reports missing datagrams2020-05-12
200512007, 008
191229002 003
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599746995
610996774
HalfPipe
mergeEvt reports
"inProgress",
missing datagrams
, gapfinder doesn't report any data gaps for 2020-01-03
2020-05-12
200512014, 015
2020-01-0312:23:14 - 13:58:05200103009 010 011
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599809968
611162210
HalfPipe
mergeEvt reports
"inProgress",
missing datagrams
, gapfinder doesn't report any data gaps for 2020-01-04
2020
-01
-
04
05
:52:47
-
07:
14
:52

200514011, 012
200103009 010 011
L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
600328891
611782769HalfPipe
reports "inProgress".  Datagram errors
labels the run as "Incomplete". MergeEvt reports missing datagrams. Gapfinder only reports a gap after 21:19.2020-
01
05-
1010:43
2119:39:28 - 20:50:13
200110005
200522001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)

How to use the RePipe facility

...

Runs before December 2018 but after July 27 2011 (L1Proc-5.6 and earlier)

Run Number

Reason

Date

L1Proc

GR version

rePipe

L1(Re)Proc

Export
369152685 ACQ Complete, L1 not finalized. Fixed in P300x.2012-09-12L1Proc-4.0GR-17-35-24-gr35
Image Modified
4.0 Image Modified
Image Modified
478706331 ACQ Complete, L1 not finalized. Fixed in P300x.2016-03-03L1Proc-5.4GR-20-10-02
Image Modified
5.4 Image Modified
Image Modified
478756313ACQ Complete, L1 not finalized. Fixed in P300x.2016-03-04L1Proc-5.4

GR-20-10-02

Image Modified
5.4 Image Modified
Image Modified
504919670Leap Second Run. Set to Incomplete and finalized.2016-12-31L1Proc-5.6

GR-20-10-04-gr01

(error)
5.6 Image Modified
Image Modified
508046861ACQ InProgress. Repiped and reprocessed.2017-02-09L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.7 Image Modified
Image Modified
512115535ACQ Complete, L1 Complete/Failed. Fixed in P300x.2017-03-25L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.6 Image Modified
Image Modified
516617903ACQ Complete, L1 Complete/Failed. Fixed in P300x. 2017-05-16L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.6 Image Modified
Image Modified
531419768ACQ Complete, L1 Complete , recon on xrootd (root://fermi-gpfs05//glast/Data/Flight/Level1/LPA/prod/5.6/recon/r0531419768_v000_recon.root) but not in datacat. Run was reprocessed correctly in P300x. No follow-up needed.2017-11-03L1Proc-5.6GR-20-10-04-gr01
Image Modified
5.6 Image Modified
Image Modified
534500370ACQ Complete, L1 Complete/Failed. Fixed in P300x. 2017-12-09L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.6 Image Modified
Image Modified
549415721 ACQ Complete, L1 Complete , recon on xrootd (root://fermi-gpfs05//glast/Data/Flight/Level1/LPA/prod/5.6/recon/r0549415721_v000_recon.root) but not in datacat. Run was reprocessed correctly in P300x. No follow-up needed. 2018-05-31L1Proc-5.6GR-20-10-04-gr01
Image Modified
5.6 Image Modified
Image Modified
553318321ACQ Complete, L1 Failed. Repiped and reprocessed.2018-07-16L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.7 Image Modified
Image Modified
554179852ACQ Complete, L1 Failed. Repiped and reprocessed. 2018-07-25L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.7 Image Modified
Image Modified
554396875 ACQ Complete, cleanupCompleteRun.getTimes terminated by reaper. Run was reprocessed correctly in P300x. No follow-up needed. 2018-07-27L1Proc-5.6GR-20-10-04-gr01
Image Modified
5.6 Image Modified
Image Modified
555737292ACQ Complete, L1 InProgress. Finalized.2018-08-12L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.6 Image Modified
Image Modified
557026900ACQ Complete, L1 InProgress. Finalized.2018-08-27L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.6 Image Modified
Image Modified
559965007Truncated Run. Set to Incomplete and finalized. 2018-09-30L1Proc-5.6

GR-20-10-04-gr01

(error)
5.6 Image Modified
Image Modified
562253064Missing Data. Set to Incomplete and finalized. Missing data sent by MOC 9/24/19, reprocessed with 5.7 - versions numbers generated are less than the v305 version from the reprocessing so ingest is not happening at the FSSC. File version edited by hand and re-exported.
2018-10-26L1Proc-5.6

GR-20-10-04-gr01

(tick)
5.7 Image Modified
Image Modified
562258990This run was missing entirely. Retrieved by MOC 9/24/19 and reprocessed with 5.7.2018-10-26L1Proc-5.6

GR-20-10-04-gr01

(tick)5.7 (tick)
Image Modified
562264544Missing Data. Set to Incomplete and finalized. Missing data sent by MOC 9/24/19, reprocessed with 5.7 - versions numbers generated are less than the v305 version from the reprocessing so ingest is not happening at the FSSC. File version edited by hand and re-exported.
2018-10-27L1Proc-5.6

GR-20-10-04-gr01

(tick)5.7 (tick)
Image Modified

564602913

Repiped, reprocessed with 5.7, re-exported.2018-11-22L1Proc-5.6

GR-20-10-04-gr01

Image Modified
5.7 (tick)(tick)

Runs from LEO through 2009

Run Number

Reason

Date

FSW

L1Proc

GR version

rePipe

L1Proc

237911263

silent NFS read error

2008-07-16

B1-0-8

L1Proc-1.63

Version-v15r39

Image Modified

(tick)

237916725

silent NFS read error

2008-07-16

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

(tick)

238008393

silent NFS read error

2008-07-17

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

(tick)

238014123

silent NFS read error

2008-07-17

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238207772

silent NFS read error

2008-07-20

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238759073

silent NFS read error

2008-07-26

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238856333

silent NFS read error

2008-07-26

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

239125611

silent NFS read error

2008-07-30

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

239129222

silent NFS read error

2008-07-30

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

241125159

1 missing Datagram

2008-08-22

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

241491796

missing chunk in in gcr

2008-08-27

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

241501554

1 missing Datagram

2008-08-27

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

243170470

afs cache corruption?

2008-09-15

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243197713

afs cache corruption?

2008-09-15

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243307762

2 missing Datagrams

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243313738

2 missing Datagrams

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243336617

afs cache corruption?

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243491490

4 missing Datagrams

2008-09-19

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243873727

afs cache corruption?

2008-09-23

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243926731

afs cache corruption?

2008-09-24

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

244143826

afs cache corruption?

2008-09-26

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

244167563

afs cache corruption?

2008-09-27

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

245667862

irretrievable packets missing from two datagrams

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(error)

(error)

245673843

This one was complete, but a retransmission changed its status

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245685781

all packets available; repiping will produce complete data-set

2008-10-14

B1-1-3

L1Proc-1.69

Version-v15r47p1

(tick)

(tick)

245691461

all packets available; repiping will produce complete data-set

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245697061

This one was already complete

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245702790

irretrievable packet missing from one datagram

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(error)

(error)

246390295

Merging problems

2008-10-22

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

246405636

Merging problems

2008-10-22

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

246447791

Merging problems

2008-10-23

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

253539991

Delivery problems ?

2009-01-13

B1-1-3

L1Proc-1.70

Version-v15r47p1

(tick)

(tick)

260478011

1 Missing datagram ???

2009-04-03

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261148032

Insane number of chunks

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261154012

Insane number of chunks

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261406015

1 Missing datagram ???

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

269048825

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269054554

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269060283

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269066012

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269071741

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269075743

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269081879

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269087937

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269093942

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269099929

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

272627642

1 bad event (seg fault)

2009-08-22

B1-1-3

L1Proc-1.75

Version-v15r47p12

(tick)

(tick)