Versions Compared

Old Version 172

changes.mady.by.user Tom Stephens

Saved on

compared with

New Version 173

changes.mady.by.user Tom Stephens

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Run Number

Reason

Date

Dls

L1Proc

GR version

rePipe

L1Proc

565721985Botched attempt at a repipe, repiped and marked as Complete2018-12-05 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
565942990stuck "in progress", repiped and marked as Complete2018-12-08 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

(tick)

566022500HP "in progress", repiped and marked as Complete2018-12-09 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
566235626HP "in progress", repiped and marked as Complete 2018-12-11 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
567301141stuck "in progress", repiped and marked as Complete2018-12-23 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

567598970

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27 L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
567581889

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27 L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
569069307in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-13

010 011L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
569213577

in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-15006 007L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572617673HP "in progress", repiped and marked as Complete 2019-02-23 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572853518stuck "in progress". Leftover chunkToken. Fixed.2019-02-26 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
574781844"in progress", incomplete data, missing datagrams2019-03-20 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575044322"in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized.2019-03-23013 014L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575762229"in progress", incomplete data, missing datagrams2019-03-31 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
576305031Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized.2019-04-07 L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
576658880"in progress", incomplete data2019-04-11 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

578530168

corrupted data in monitoring2019-05-02 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
580907445in progress, mergeIndices reports 5 missing datagrams2019-05-30 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
582073028in progress, mergeIndices reports missing datagrams2019-06-13 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
584730839in progress, complete data, missing datagrams2019-07-13 L1Proc-5.7GlastRelease-20-10-04-gr02?(tick)
585187986

in progress, complete data, missing datagrams, problems with rolling back halfPipe.190719003

2019-07-19 L1Proc-5.7GlastRelease-20-10-04-gr02?(error)
585193691in progress, complete data, missing datagrams, problems with rolling back halfPipe.1907190032019-07-19 L1Proc-5.7GlastRelease-20-10-04-gr02?(error)
585535991in progress, complete data, missing datagrams2019-07-23 L1Proc-5.7GlastRelease-20-10-04-gr02?(tick)
585690096in progress, missing datagrams2019-07-24 L1Proc-5.7GlastRelease-20-10-04-gr02Tom(tick)(tick)
586076888see thread. However it seems the recon file is complete. Reprocessing complete.  Everything seems to have worked including the cleanupCompleteRun process.2019-07-29 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586893790bunch of tasks in "Waiting" state preventing a rollback.  Once that is cleared a rollback should hopefully fix everything2019-08-07 L1Proc-5.7GlastRelease-20-10-04-gr02  
586905200see thread. Seems the run is complete and can be just finalized. After checking, I've finalized this run as it looks fine -TS. MEM: same as 588742217, run was *not* finalized. 1. remove chunktokens as needed, 2. rollback checkRun, 3. make sure that cleanupCompleteRun is executed and recon file is merged. This is fine now.2019-08-07 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
587042253Several failed tasks with errors of the form: "IOError: Can't stage in <filename>".  Looks like it was an xrootd hiccup.  Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete2019-08-09 12:46 L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
587104373

This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success.  There is a chunk token directory but no token.  However, the verify log has these lines (for both EPU0 and EPU1):

Warning! The closing reason for the last datagram for Epu1 was: Datagram Full -- Data might be lost!
Warning! The last datagram for Epu1 was not closed because we reached the end of run! The datagram closing action was 3

I'm think that means we need to repipe everything.

2019-08-10 6:54 L1Proc-5.7GlastRelease-20-10-04-gr02  
588742217Run is incomplete according to the halfPipe. TS: I think this is okay as I don't see any errors in the verify log. MEM: run wasn't finalized because of leftover chunktokens. Please make sure that cleanupCompleteRun is executed before claiming success. This is fine now.2019-08-29 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
590065315Run still Incomplete in L1.  Looks like cleanupCompleteRun didn't get triggered.2019-09-13 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)

...