Versions Compared

Old Version 175

changes.mady.by.user Maria Elena Monzani

Saved on

compared with

New Version 176

changes.mady.by.user Tom Stephens

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Run Number

Reason

Date

Dls

L1Proc

GR version

rePipe

L1Proc

565721985Botched attempt at a repipe, repiped and marked as Complete2018-12-05 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
565942990stuck "in progress", repiped and marked as Complete2018-12-08 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

(tick)

566022500HP "in progress", repiped and marked as Complete2018-12-09 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
566235626HP "in progress", repiped and marked as Complete 2018-12-11 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
567301141stuck "in progress", repiped and marked as Complete2018-12-23 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

567598970

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27 L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
567581889

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27 L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
569069307in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-13

010 011L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
569213577

in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-15006 007L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572617673HP "in progress", repiped and marked as Complete 2019-02-23 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572853518stuck "in progress". Leftover chunkToken. Fixed.2019-02-26 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
574781844"in progress", incomplete data, missing datagrams2019-03-20 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575044322"in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized.2019-03-23013 014L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575762229"in progress", incomplete data, missing datagrams2019-03-31 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
576305031Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized.2019-04-07 L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
576658880"in progress", incomplete data2019-04-11 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

578530168

corrupted data in monitoring2019-05-02 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
580907445in progress, mergeIndices reports 5 missing datagrams2019-05-30 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
582073028in progress, mergeIndices reports missing datagrams2019-06-13 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
584730839in progress, complete data, missing datagrams2019-07-13 L1Proc-5.7GlastRelease-20-10-04-gr02Tom(tick)(tick)?
585187986

in progress, complete data, missing datagrams, problems with rolling back halfPipe.190719003

2019-07-19 L1Proc-5.7GlastRelease-20-10-04-gr02Tom(tick)(error)?
585193691in progress, complete data, missing datagrams, problems with rolling back halfPipe.1907190032019-07-19 L1Proc-5.7GlastRelease-20-10-04-gr02Tom(tick)(error)?
585535991in progress, complete data, missing datagrams2019-07-23 L1Proc-5.7GlastRelease-20-10-04-gr02Tom(tick)(tick)?
585690096in progress, missing datagrams2019-07-24 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586076888see thread. However it seems the recon file is complete. Reprocessing complete.  Everything seems to have worked including the cleanupCompleteRun process.2019-07-29 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586893790bunch of tasks in "Waiting" state preventing a rollback.  Once that is cleared a rollback should hopefully fix everything2019-08-07 L1Proc-5.7GlastRelease-20-10-04-gr02  
586905200see thread. Seems the run is complete and can be just finalized. After checking, I've finalized this run as it looks fine -TS. MEM: same as 588742217, run was *not* finalized. 1. remove chunktokens as needed, 2. rollback checkRun, 3. make sure that cleanupCompleteRun is executed and recon file is merged. This is fine now.2019-08-07 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
587042253Several failed tasks with errors of the form: "IOError: Can't stage in <filename>".  Looks like it was an xrootd hiccup.  Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete2019-08-09 12:46 L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
587104373

This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success.  There is a chunk token directory but no token.  However, the verify log complains about closing reasons for both EUP0 and EPU1. I'm think that means we need to repipe everything.

2019-08-10 6:54 L1Proc-5.7GlastRelease-20-10-04-gr02  
588742217Run is incomplete according to the halfPipe. TS: I think this is okay as I don't see any errors in the verify log. MEM: run wasn't finalized because of leftover chunktokens. Please make sure that cleanupCompleteRun is executed before claiming success. This is fine now.2019-08-29 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
590065315Run still Incomplete in L1.  Looks like cleanupCompleteRun didn't get triggered.2019-09-13 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)

...

Run Number

Reason

Date

L1Proc

GR version

rePipe

L1(Re)Proc

Export
369152685ACQ Complete, L1 not finalized. Fixed in P300x.2012-09-12L1Proc-4.0GR-17-35-24-gr354.0
478706331ACQ Complete, L1 not finalized. Fixed in P300x.2016-03-03L1Proc-5.4GR-20-10-025.4
478756313ACQ Complete, L1 not finalized. Fixed in P300x.2016-03-04L1Proc-5.4

GR-20-10-02

5.4
504919670Leap Second Run. Set to Incomplete and finalized.2016-12-31L1Proc-5.6

GR-20-10-04-gr01

(error)5.6
508046861ACQ InProgress. Repiped and reprocessed.2017-02-09L1Proc-5.6

GR-20-10-04-gr01

5.7
512115535ACQ Complete, L1 Complete/Failed. Fixed in P300x.2017-03-25L1Proc-5.6

GR-20-10-04-gr01

5.6
516617903ACQ Complete, L1 Complete/Failed. Fixed in P300x.2017-05-16L1Proc-5.6

GR-20-10-04-gr01

5.6
531419768ACQ Complete, L1 Complete, recon on xrootd (root://fermi-gpfs05//glast/Data/Flight/Level1/LPA/prod/5.6/recon/r0531419768_v000_recon.root) but not in datacat2017-11-03L1Proc-5.6GR-20-10-04-gr01   
534500370ACQ Complete, L1 Complete/Failed. Fixed in P300x.2017-12-09L1Proc-5.6

GR-20-10-04-gr01

5.6
549415721ACQ Complete, L1 Complete, recon on xrootd (root://fermi-gpfs05//glast/Data/Flight/Level1/LPA/prod/5.6/recon/r0549415721_v000_recon.root) but not in datacat2018-05-31L1Proc-5.6GR-20-10-04-gr01   
553318321ACQ Complete, L1 Failed. Repiped and reprocessed.2018-07-16L1Proc-5.6

GR-20-10-04-gr01

5.7
554179852ACQ Complete, L1 Failed. Repiped and reprocessed.2018-07-25L1Proc-5.6

GR-20-10-04-gr01

5.7
554396875ACQ Complete, cleanupCompleteRun.getTimes terminated by reaper 2018-07-27L1Proc-5.6GR-20-10-04-gr01 (error) 
555737292ACQ Complete, L1 InProgress. Finalized.2018-08-12L1Proc-5.6

GR-20-10-04-gr01

5.6
557026900ACQ Complete, L1 InProgress. Finalized.2018-08-27L1Proc-5.6

GR-20-10-04-gr01

5.6
559965007Truncated Run. Set to Incomplete and finalized.2018-09-30L1Proc-5.6

GR-20-10-04-gr01

(error)5.6
562253064Missing Data. Set to Incomplete and finalized. Missing data sent by MOC 9/24/19, reprocessed with 5.7
2018-10-26L1Proc-5.6

GR-20-10-04-gr01

(error)(tick)Tom5.6 Image Removed
562264544Missing Data. Set to Incomplete and finalized. Missing data sent by MOC 9/24/19, reprocessed with 5.7
2018-10-27L1Proc-5.6

GR-20-10-04-gr01

(error)(tick)Tom5.6 Image Removed

564602913

Repiped, reprocessed with 5.7, re-exported.2018-11-22L1Proc-5.6

GR-20-10-04-gr01

5.7 (tick)(tick)

...