Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Run NumberReasonDateTimeDlsL1ProcGR versionrePipeL1Proc
730458995240224011.mergeEvt ran twice, creating an incomplete evt file on the second try.2024-02-24
24022401[01]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(error)
729548905Highly fragmented run. 91, 15, and 61 chunks. This requires task RePipe.2024-02-14
24021400[345]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
727336262mergeEvt reports 6 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-19
240119003L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
727324474mergeEvt reports missing datagrams at the end of epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-19
240119002L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
726579387mergeEvt reports 5 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-10
24011000[78]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
726452942mergeEvt reports missing datagrams at the end of epu1. Gapfinder doesn't report a gap for the duration of this run.2024-01-0900:29:01 - 01:41:00240109004L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
723956756mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report a gap for the duration of this run.2023-12-1103:05:55 - 04:39:3623121100[23]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
723939304mergeEvt reports 5 missing datagrams in epu1. Gapfinder doesn't report a gap for the duration of this run.2023-12-1022:15:02 - 23:32:30231211001L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
723865764mergeEvt reports 6 missing datagrams in epu1. Gapfinder doesn't report any gaps.2023-12-10
23121000[34]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
723665322mergeEvt reports 6 missing datagrams in epu1. Gapfinder found a gap for this run and sent just one retransmit request.2023-12-0718:08:41 - 19:19:2223120101[02]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
723103279mergeEvt reports about 100 missing datagrams at the end of epu1. Gapfinder doesn't report any gaps.2023-12-01
23120100[78]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
717719650

Dls 230929007 and 009 resulted from passes the FOT called "ratty".  All run pieces in these dls were highly fragmented, and L1 didn't process them.  Redumps were triggered, but only run 717708266 was completely filled in dl 230930008, 717713958 partially.  Of the latter it's estimated that about half of the data are missing (1.2M events), while 717702333 (2.5M events) and 717719650 (2.7M events) should be almost filled.

UPDATE: rePipe didn't result in more merged events.  The runs were finalized as is, i.e. with the gaps.

2023-09-29
23093000[123], (?230929009?)L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
7177139582023-09-29
230930008, (?230929009?)L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
7177023332023-09-29
230929008, 23093000[12], (?230929007?)L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
710506750mergeEvt reports 6 missing datagrams in epu0. Gapfinder doesn't report any gaps.2023-07-08
23070800[89]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
709339013somehow the xrootd scratch area vanished before cleanupCompleteRun. The run needs to be repiped from the HalfPipe.2023-06-2422:41:52 - 00:16:33 (+1)23062500[12]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
708257402mergeEvt reports 8 missing datagrams in epu1. Gapfinder doesn't report any gaps.2023-06-12
23031200[56]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
700230335mergeEvt reports 5 missing datagrams in epu0.  Gapfinder doesn't report any gaps.2023-02-11
230311009L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
697691485mergeEvt reports 7 missing datagrams in epu1.2023-02-10
23021000[45]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
694642846verify found a datagram gap.  Actually already mergeEvt found this gap of 12 datagrams in epu1, but somehow the run was labeled as complete.  A HalfPipe repipe leaves one missing datagram, decompression error. Finalized with one datagram gap.2023-01-05
230105010, 230106001L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
693360593003.mergeIndices failed and was processed last. When mergeEvt ran it didn't merge in the datagrams of 004.2022-12-22
22122200[234]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
687193623mergeReconChunks failed with a bus error but exit code 0. The merged recon file was not copied to xrootd but cleanupChunks removed all temporary files.2022-10-11
22101101[123]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
687119520mergeEvt reports 6 missing datagrams in epu0, gapfinder doesn't report any gaps.2022-10-10
221010013L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
686035262dl's 220928003 and 004 came in together. It was not possible to disentangle both. 220928003.mergeIndices fails.2022-09-28
22092800[34]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
68602????2022-09-28
220928003L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
6860226852022-09-28
22092800[23]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
685255472mergeEvt reports 7 missing datagrams in epu0, gapfinder doesn't report any gaps.2022-09-19
22091900[45]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
682569294Aug 19 (PDT) there were issues with a faulty nfs disk. This run was merged badly.2022-08-1902:34:53 - 03:45:0322081900[45]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
682556983Aug 19 (PDT) there were issues with a faulty nfs disk. This run was merged badly.2022-08-1823:09:42 - 00:29:45 (+1)22081900[34]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
680740571mergeEvt reports 5 missing datagrams in epu1, gapfinder doesn't report any gaps.2022-07-2822:36:10 - 23:58:0122072900[12]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
676577929mergeEvt reports 5 missing datagrams in epu0, gapfinder doesn't report any gaps.2022-06-10
22061001[01]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
675367448mergeEvt reports 8 missing datagram in epu0, additionally verify finds 7 missing datagrams in epu1.  However gapfinder doesn't report any gaps.2022-05-2718:04:07 - 19:28:29220527012, 220528001L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
674884484mergeEvt reports 1 missing datagram in epu1, gapfinder doesn't report any gaps.  MakeEvt reports a decompression error, finalizing with one datagram gap.2022-05-22
2205220[567]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
674878528mergeEvt reports several gaps with missing datagrams in epu1, gapfinder doesn't report any gaps.  MakeEvt reports decompression errors, finalizing with 14 scattered datagram gaps.2022-05-22
22052200[45]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
674872543mergeEvt reports several gaps with missing datagrams in epu1, gapfinder doesn't report any gaps.  MakeEvt reports decompression errors, finalizing with 16 missing datagrams scattered over 14 gaps.2022-05-22
22052200[34]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
674052578mergeEvt reports 5 missing datagrams in epu1, gapfinder doesn't report any gaps.2022-05-12
220512010L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
671827689verify reported missing data. Indeed also mergeEvt reports 5 missing datagrams in epu0, but somehow the run was labeled as complete.2022-04-16
220416014L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
671754063missing datagrams at the end of epu1. gapfinder doesn't report any gaps.2022-04-16
22041600[23]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
670363411Dls 220331005 and 006 came in within the same second, and got mangled. 006 passed, but 005 fails at HalfPipe.mergeIndices. Needs to be repiped completely.2022-03-31
22033100[25]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
670357708Dls 220331005 and 006 came in within the same second, and got mangled. 006 passed, but 005 fails at HalfPipe.mergeIndices. Needs to be repiped completely.2022-03-31
22033100[345]L1Proc-5.9GlastRelease-20-10-04-gr07(error)(error)
661643650Highly fragmented run (138 chunks). Needs a "real" repipe.2021-12-2021:54:09 - 23:28:5821122000[124]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
656744433Fall-out of the 211024 hypervisor failure. EPU0 is complete but EPU1 has a gap of ~5000 datagrams.  Roll back of HalfPipe didn't work!  Repipe did. But there was a gap in the magic7 file → second repipe
2021-10-24
21102400[56]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
649557766Highly fragmented run (>400 chunks). Needs a "real" repipe.2021-08-02
21080200[234]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
648918184mergeEvt reports a gap of 5 datagrams in epu0. Gapfinder doesn't report any gaps.2021-07-2515:03:03 - 16:13:0521072501[01]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
648862027mergeEvt reports a gap of 5 datagrams in epu1. Gapfinder doesn't report any gaps.2021-07-2423:27:06 - 00:59:49[+1]21072500[12]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
645896054One datagram in the inaccessible part of the SSR. The run will stay incomplete.2021-06-20
21062001[12]L1Proc-5.9GlastRelease-20-10-04-gr07-(tick)
645331200mergeEvt reports a gap of 9 datagrams in epu1. Gapfinder doesn't report any gaps.2021-06-14
21061400[34]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)
644150495mergeEvt reports a gap of 9 datagrams in epu0. Gapfinder doesn't report any gaps.2021-05-31
21053100[78]L1Proc-5.9GlastRelease-20-10-04-gr07(tick)(tick)

...