You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 188 Next »

Runs from December 2018 onward (L1Proc-5.7)

Run Number

Reason

Date

Dls

L1Proc

GR version

rePipe

L1Proc

565721985Botched attempt at a repipe, repiped and marked as Complete2018-12-05 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
565942990stuck "in progress", repiped and marked as Complete2018-12-08 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

(tick)

566022500HP "in progress", repiped and marked as Complete2018-12-09 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
566235626HP "in progress", repiped and marked as Complete 2018-12-11 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
567301141stuck "in progress", repiped and marked as Complete2018-12-23 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

567598970

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27 L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
567581889

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27 L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
569069307in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-13

010 011L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
569213577

in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-15006 007L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572617673HP "in progress", repiped and marked as Complete 2019-02-23 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572853518stuck "in progress". Leftover chunkToken. Fixed.2019-02-26 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
574781844"in progress", incomplete data, missing datagrams2019-03-20 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575044322"in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized.2019-03-23013 014L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575762229"in progress", incomplete data, missing datagrams2019-03-31 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
576305031Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized.2019-04-07 L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
576658880"in progress", incomplete data2019-04-11 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

578530168

corrupted data in monitoring2019-05-02 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
580907445in progress, mergeIndices reports 5 missing datagrams2019-05-30 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
582073028in progress, mergeIndices reports missing datagrams2019-06-13 L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
584730839in progress, complete data, missing datagrams2019-07-13 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585187986

in progress, complete data, missing datagrams, problems with rolling back halfPipe.190719003

2019-07-19 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(error)?
585193691in progress, complete data, missing datagrams, problems with rolling back halfPipe.1907190032019-07-19 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(error)?
585535991in progress, complete data, missing datagrams2019-07-23 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585690096in progress, missing datagrams2019-07-24 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586076888see thread. However it seems the recon file is complete. Reprocessing complete.  Everything seems to have worked including the cleanupCompleteRun process.2019-07-29 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586893790bunch of tasks in "Waiting" state preventing a rollback.  Once that is cleared a rollback should hopefully fix everything2019-08-07 L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
586905200see thread. Seems the run is complete and can be just finalized. After checking, I've finalized this run as it looks fine -TS. MEM: same as 588742217, run was *not* finalized. 1. remove chunktokens as needed, 2. rollback checkRun, 3. make sure that cleanupCompleteRun is executed and recon file is merged. This is fine now.2019-08-07 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
587042253Several failed tasks with errors of the form: "IOError: Can't stage in <filename>".  Looks like it was an xrootd hiccup.  Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete2019-08-09 12:46 L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
587104373

This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success.  There is a chunk token directory but no token.  However, the verify log complains about closing reasons for both EUP0 and EPU1. I'm think that means we need to repipe everything.

2019-08-10 6:54 L1Proc-5.7GlastRelease-20-10-04-gr02

(tick)

 
588742217Run is incomplete according to the halfPipe. TS: I think this is okay as I don't see any errors in the verify log. MEM: run wasn't finalized because of leftover chunktokens. Please make sure that cleanupCompleteRun is executed before claiming success. This is fine now.2019-08-29 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
590065315Run still Incomplete in L1.  Looks like cleanupCompleteRun didn't get triggered.2019-09-13 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
591931054 failure in checkRun 191005003.591931054.  2 missing datagram.  L1Proc-5.7GlastRelease-20-10-04-gr02(tick)Tom S
5952403499 missing datagrams2019-11-12 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595405654missing datagrams2019-11-14 L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)

How to use the RePipe facility

/afs/slac/g/glast/ground/bin/pipeline createStream --define "downlinkID=990101001,runID=237063408" RePipe
/nfs/farm/g/glast/u28/RePipe/990101001 will contain the RePiped output (should this have a --stream <run number> flag in the command for clarity in the monitoring pages?)

if you set an environment variable, rn, to the run number (i.e. setenv rn <run number>) you can do the following command to start the repipe and have the output in the format expected for the instruction linked to below.

/afs/slac/g/glast/ground/bin/pipeline createStream --stream $rn --define "downlinkID=$rn,runID=$rn" RePipe

More details: Instructions for rolling back a halfPipe stream and susbsequent L1 stream and Reprocessing runs from a RePiped job

Runs before December 2018 but after July 27 2011 (L1Proc-5.6 and earlier)

Run Number

Reason

Date

L1Proc

GR version

rePipe

L1(Re)Proc

Export
369152685ACQ Complete, L1 not finalized. Fixed in P300x.2012-09-12L1Proc-4.0GR-17-35-24-gr354.0
478706331ACQ Complete, L1 not finalized. Fixed in P300x.2016-03-03L1Proc-5.4GR-20-10-025.4
478756313ACQ Complete, L1 not finalized. Fixed in P300x.2016-03-04L1Proc-5.4

GR-20-10-02

5.4
504919670Leap Second Run. Set to Incomplete and finalized.2016-12-31L1Proc-5.6

GR-20-10-04-gr01

(error)5.6
508046861ACQ InProgress. Repiped and reprocessed.2017-02-09L1Proc-5.6

GR-20-10-04-gr01

5.7
512115535ACQ Complete, L1 Complete/Failed. Fixed in P300x.2017-03-25L1Proc-5.6

GR-20-10-04-gr01

5.6
516617903ACQ Complete, L1 Complete/Failed. Fixed in P300x.2017-05-16L1Proc-5.6

GR-20-10-04-gr01

5.6
531419768ACQ Complete, L1 Complete, recon on xrootd (root://fermi-gpfs05//glast/Data/Flight/Level1/LPA/prod/5.6/recon/r0531419768_v000_recon.root) but not in datacat. Run was reprocessed correctly in P300x. No follow-up needed.2017-11-03L1Proc-5.6GR-20-10-04-gr015.6
534500370ACQ Complete, L1 Complete/Failed. Fixed in P300x.2017-12-09L1Proc-5.6

GR-20-10-04-gr01

5.6
549415721ACQ Complete, L1 Complete, recon on xrootd (root://fermi-gpfs05//glast/Data/Flight/Level1/LPA/prod/5.6/recon/r0549415721_v000_recon.root) but not in datacat. Run was reprocessed correctly in P300x. No follow-up needed.2018-05-31L1Proc-5.6GR-20-10-04-gr015.6
553318321ACQ Complete, L1 Failed. Repiped and reprocessed.2018-07-16L1Proc-5.6

GR-20-10-04-gr01

5.7
554179852ACQ Complete, L1 Failed. Repiped and reprocessed.2018-07-25L1Proc-5.6

GR-20-10-04-gr01

5.7
554396875ACQ Complete, cleanupCompleteRun.getTimes terminated by reaper. Run was reprocessed correctly in P300x. No follow-up needed.2018-07-27L1Proc-5.6GR-20-10-04-gr015.6
555737292ACQ Complete, L1 InProgress. Finalized.2018-08-12L1Proc-5.6

GR-20-10-04-gr01

5.6
557026900ACQ Complete, L1 InProgress. Finalized.2018-08-27L1Proc-5.6

GR-20-10-04-gr01

5.6
559965007Truncated Run. Set to Incomplete and finalized.2018-09-30L1Proc-5.6

GR-20-10-04-gr01

(error)5.6
562253064Missing Data. Set to Incomplete and finalized. Missing data sent by MOC 9/24/19, reprocessed with 5.7 - versions numbers generated are less than the v305 version from the reprocessing so ingest is not happening at the FSSC. File version edited by hand and re-exported.
2018-10-26L1Proc-5.6

GR-20-10-04-gr01

(tick)5.7
562258990This run was missing entirely. Retrieved by MOC 9/24/19 and reprocessed with 5.7.2018-10-26L1Proc-5.6

GR-20-10-04-gr01

(tick)5.7 (tick)
562264544Missing Data. Set to Incomplete and finalized. Missing data sent by MOC 9/24/19, reprocessed with 5.7 - versions numbers generated are less than the v305 version from the reprocessing so ingest is not happening at the FSSC. File version edited by hand and re-exported.
2018-10-27L1Proc-5.6

GR-20-10-04-gr01

(tick)5.7 (tick)

564602913

Repiped, reprocessed with 5.7, re-exported.2018-11-22L1Proc-5.6

GR-20-10-04-gr01

5.7 (tick)(tick)

Runs from LEO through 2009

Run Number

Reason

Date

FSW

L1Proc

GR version

rePipe

L1Proc

237911263

silent NFS read error

2008-07-16

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

237916725

silent NFS read error

2008-07-16

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

(tick)

238008393

silent NFS read error

2008-07-17

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

(tick)

238014123

silent NFS read error

2008-07-17

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238207772

silent NFS read error

2008-07-20

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238759073

silent NFS read error

2008-07-26

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238856333

silent NFS read error

2008-07-26

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

239125611

silent NFS read error

2008-07-30

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

239129222

silent NFS read error

2008-07-30

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

241125159

1 missing Datagram

2008-08-22

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

241491796

missing chunk in in gcr

2008-08-27

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

241501554

1 missing Datagram

2008-08-27

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

243170470

afs cache corruption?

2008-09-15

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243197713

afs cache corruption?

2008-09-15

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243307762

2 missing Datagrams

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243313738

2 missing Datagrams

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243336617

afs cache corruption?

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243491490

4 missing Datagrams

2008-09-19

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243873727

afs cache corruption?

2008-09-23

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243926731

afs cache corruption?

2008-09-24

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

244143826

afs cache corruption?

2008-09-26

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

244167563

afs cache corruption?

2008-09-27

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

245667862

irretrievable packets missing from two datagrams

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(error)

(error)

245673843

This one was complete, but a retransmission changed its status

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245685781

all packets available; repiping will produce complete data-set

2008-10-14

B1-1-3

L1Proc-1.69

Version-v15r47p1

(tick)

(tick)

245691461

all packets available; repiping will produce complete data-set

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245697061

This one was already complete

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245702790

irretrievable packet missing from one datagram

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(error)

(error)

246390295

Merging problems

2008-10-22

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

246405636

Merging problems

2008-10-22

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

246447791

Merging problems

2008-10-23

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

253539991

Delivery problems ?

2009-01-13

B1-1-3

L1Proc-1.70

Version-v15r47p1

(tick)

(tick)

260478011

1 Missing datagram ???

2009-04-03

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261148032

Insane number of chunks

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261154012

Insane number of chunks

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261406015

1 Missing datagram ???

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

269048825

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269054554

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269060283

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269066012

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269071741

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269075743

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269081879

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269087937

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269093942

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269099929

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

272627642

1 bad event (seg fault)

2009-08-22

B1-1-3

L1Proc-1.75

Version-v15r47p12

(tick)

(tick)

  • No labels