Versions Compared

Old Version 229

changes.mady.by.user Michael Kuss

Saved on

compared with

New Version 230

changes.mady.by.user Maria Elena Monzani

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Run Number

Reason

Date

TimeDls

L1Proc

GR version

rePipe

L1Proc

565721985Botched attempt at a repipe, repiped and marked as Complete2018-12-05  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
565942990stuck "in progress", repiped and marked as Complete2018-12-08  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

(tick)

566022500HP "in progress", repiped and marked as Complete2018-12-09  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
566235626HP "in progress", repiped and marked as Complete 2018-12-11  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
567301141stuck "in progress", repiped and marked as Complete2018-12-23  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

567598970

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27  L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
567581889

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27  L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
569069307in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-13

 010 011L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
569213577

in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-15 006 007L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572617673HP "in progress", repiped and marked as Complete 2019-02-23  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572853518stuck "in progress". Leftover chunkToken. Fixed.2019-02-26  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
574781844"in progress", incomplete data, missing datagrams2019-03-20  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575044322"in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized.2019-03-23 013 014L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575762229"in progress", incomplete data, missing datagrams2019-03-31  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
576305031Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized.2019-04-07  L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
576658880"in progress", incomplete data2019-04-11  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

578530168

corrupted data in monitoring2019-05-02  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
580907445in progress, mergeIndices reports 5 missing datagrams2019-05-30  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
582073028in progress, mergeIndices reports missing datagrams2019-06-13  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
584730839in progress, complete data, missing datagrams2019-07-13  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585187986

in progress, complete data, missing datagrams, problems with rolling back halfPipe.190719003halfPipe and RePipe failed at delivering a complete set of data. Set to Incomplete and finalized.

2019-07-19  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(error)?(tick)
585193691in progress, complete data, missing datagrams, problems with rolling back halfPipe.190719003halfPipe and RePipe failed at delivering a complete set of data. Set to Incomplete and finalized.2019-07-19  L1Proc-5.7GlastRelease-20-10-04-gr02(tick) (error) ?(tick)
585535991in progress, complete data, missing datagrams2019-07-23  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585690096in progress, missing datagrams2019-07-24  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586076888see thread. However it seems the recon file is complete. Reprocessing complete.  Everything seems to have worked including the cleanupCompleteRun process.2019-07-29  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586893790bunch of tasks in "Waiting" state preventing a rollback.  Once that is cleared a rollback should hopefully fix everything2019-08-07  L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
586905200see thread . Seems the run is complete and can be just finalized. After checking, I've finalized this run as it looks fine -TS. MEM: same as 588742217, run was *not* finalized. 1. remove chunktokens as needed, 2. rollback checkRun, 3. make sure that cleanupCompleteRun is executed and recon file is merged. This is fine now. 2019-08-07  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
587042253Several failed tasks with errors of the form: "IOError: Can't stage in <filename>".  Looks like it was an xrootd hiccup.  Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete2019-08-09 12:46  L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
587104373

This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success.  There is a chunk token directory but no token.  However, the verify log complains about closing reasons for both EUP0 and EPU1. I'm think that means we need to repipe everything.

2019-08-10 6:54  L1Proc-5.7GlastRelease-20-10-04-gr02

(tick)

(tick)
588742217Run is incomplete according to the halfPipe. TS: I think this is okay as I don't see any errors in the verify log. MEM: run wasn't finalized because of leftover chunktokens. Please make sure that cleanupCompleteRun is executed before claiming success. This is fine now.2019-08-29  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
590065315Run still Incomplete in L1.  Looks like cleanupCompleteRun didn't get triggered.2019-09-13  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
591931054 failure in checkRun 191005003.591931054.  2 missing datagram.2019-10-05  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
5952403499 missing datagrams2019-11-12  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595405654missing datagrams2019-11-14  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595576882missing datagrams.  In progress2019-11-16  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595963106missing datagrams - The repiped run report more missing data than the original run. I'm at a loss on how to fix this one.2019-11-2017:18:24 - 18:32:18

191120013 191121001

L1Proc-5.7GlastRelease-20-10-04-gr02(tick) (error)
596299562HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-11-24 21:30-24:002019-11-2414:46:00 - 16:01:21

191124009

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
596916997missing datagrams2019-12-01  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
597315084HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-12-06 05:00-14:002019-12-0608:51:23 - 10:06:10191206009 010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598223809HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-162019-12-1621:16:48 - 22:51:39191217001 002L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598581385HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2100:36:24 - 01:53:08191221003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598645966HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2118:32:45 - 20:07:36191221009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598758794HalfPipe reports "inProgress"2019-12-2305:39:57 - 08:44:23191223003 004L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599272563HalfPipe reports "inProgress".  Datagram errors2019-12-292:22:25 - 4:58:04191229002 003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599746995HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2020-01-032020-01-0312:23:14 - 13:58:05200103009 010 011L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599809968HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2020-01-042020-01-0405:52:47 - 07:14:52200103009 010 011L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
600328891HalfPipe reports "inProgress".  Datagram errors2020-01-1010:43:13200110005L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
601777922HalfPipe reports "inProgress", but mergeEvt doesn't report any datagram errors!  Gapfinder doesn't report any data gaps for 2020-01-27.2020-01-2702:07:06 - 03:41:57200127002 003 004L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
601783627HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-27.2020-01-2700:32:01 - 02:06:55200127004 005 006L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
601890911HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for the morning of 2020-01-28.2020-01-2807:55:10 - 09:04:44200128006 008L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
602153762HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-31. No decompression errors in makeEvt.2020-01-3108:56:01 - 10:07:38200131007L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
603637679HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-02-17. No decompression errors in makeEvt.2020-02-1713:07:57 - 14:42:49200217009 010L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
603974262Run is complete.  mergeEvt.200221009.603974262 reports 6 missing datagrams. gapfinder doesn't report any data gaps for the time interval.2020-02-2110:37:41 - 12:12:31200221009L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
604573262

HalfPipe reports "inProgress".  Missing datagrams. Gapfinder doesn't report any data gaps for 2020-02-28.

2020-02-2809:01:01 - 10:35:52200228008 - 009L1Proc-5.7GlastRelease-20-10-04-gr02

(tick)

(error) Michael

...