Versions Compared

Old Version 209

changes.mady.by.user Tom Stephens

Saved on

compared with

New Version 210

changes.mady.by.user Tom Stephens

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Run Number

Reason

Date

TimeDls

L1Proc

GR version

rePipe

L1Proc

565721985Botched attempt at a repipe, repiped and marked as Complete2018-12-05  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
565942990stuck "in progress", repiped and marked as Complete2018-12-08  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

(tick)

566022500HP "in progress", repiped and marked as Complete2018-12-09  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
566235626HP "in progress", repiped and marked as Complete 2018-12-11  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
567301141stuck "in progress", repiped and marked as Complete2018-12-23  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

567598970

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27  L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
567581889

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27  L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
569069307in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-13

 010 011L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
569213577

in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-15 006 007L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572617673HP "in progress", repiped and marked as Complete 2019-02-23  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572853518stuck "in progress". Leftover chunkToken. Fixed.2019-02-26  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
574781844"in progress", incomplete data, missing datagrams2019-03-20  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575044322"in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized.2019-03-23 013 014L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575762229"in progress", incomplete data, missing datagrams2019-03-31  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
576305031Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized.2019-04-07  L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
576658880"in progress", incomplete data2019-04-11  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

578530168

corrupted data in monitoring2019-05-02  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
580907445in progress, mergeIndices reports 5 missing datagrams2019-05-30  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
582073028in progress, mergeIndices reports missing datagrams2019-06-13  L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
584730839in progress, complete data, missing datagrams2019-07-13  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585187986

in progress, complete data, missing datagrams, problems with rolling back halfPipe.190719003

2019-07-19  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(error)?
585193691in progress, complete data, missing datagrams, problems with rolling back halfPipe.1907190032019-07-19  L1Proc-5.7GlastRelease-20-10-04-gr02(tick) (error) ?
585535991in progress, complete data, missing datagrams2019-07-23  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585690096in progress, missing datagrams2019-07-24  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586076888see thread. However it seems the recon file is complete. Reprocessing complete.  Everything seems to have worked including the cleanupCompleteRun process.2019-07-29  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586893790bunch of tasks in "Waiting" state preventing a rollback.  Once that is cleared a rollback should hopefully fix everything2019-08-07  L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
586905200see thread . Seems the run is complete and can be just finalized. After checking, I've finalized this run as it looks fine -TS. MEM: same as 588742217, run was *not* finalized. 1. remove chunktokens as needed, 2. rollback checkRun, 3. make sure that cleanupCompleteRun is executed and recon file is merged. This is fine now. 2019-08-07  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
587042253Several failed tasks with errors of the form: "IOError: Can't stage in <filename>".  Looks like it was an xrootd hiccup.  Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete2019-08-09 12:46  L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
587104373

This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success.  There is a chunk token directory but no token.  However, the verify log complains about closing reasons for both EUP0 and EPU1. I'm think that means we need to repipe everything.

2019-08-10 6:54  L1Proc-5.7GlastRelease-20-10-04-gr02

(tick)

(tick)
588742217Run is incomplete according to the halfPipe. TS: I think this is okay as I don't see any errors in the verify log. MEM: run wasn't finalized because of leftover chunktokens. Please make sure that cleanupCompleteRun is executed before claiming success. This is fine now.2019-08-29  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
590065315Run still Incomplete in L1.  Looks like cleanupCompleteRun didn't get triggered.2019-09-13  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
591931054 failure in checkRun 191005003.591931054.  2 missing datagram.2019-10-05  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
5952403499 missing datagrams2019-11-12  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595405654missing datagrams2019-11-14  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595576882missing datagrams.  In progress2019-11-16  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595963106missing datagrams - The repiped run report more missing data than the original run. I'm at a loss on how to fix this one.2019-11-2017:18:24 - 18:32:18

191120013 191121001

L1Proc-5.7GlastRelease-20-10-04-gr02(tick) (error)
596299562HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-11-24 21:30-24:002019-11-2414:46:00 - 16:01:21

191124009

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
596916997missing datagrams2019-12-01  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
597315084HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-12-06 05:00-14:002019-12-0608:51:23 - 10:06:10191206009 010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598223809HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-162019-12-1621:16:48 - 22:51:39191217001 002L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598581385HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2100:36:24 - 01:53:08191221003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598645966HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2118:32:45 - 20:07:36191221009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598758794HalfPipe reports "inProgress"2019-12-2305:39:57 - 08:44:23191223003 - 004L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599272563HalfPipe reports "inProgress".  Datagram errors2019-12-292:22:25 - 4:58:04191229002 - 003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
600328891HalfPipe reports "inProgress".  Datagram errors2020-01-1010:43:13200110005L1Proc-5.7GlastRelease-20-10-04-gr02Tom S. 

How to use the RePipe facility

...