You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 350 Next »

Runs from May 25, 2020 onward (L1Proc-5.7), backward sorted

Run NumberReasonDateTimeDlsL1ProcGR versionrePipeL1Proc
626729954Highly fragmented run (>66 chunks). doRun/scanXxx processes fail. Moreover it's spread over 7 deliveries.2020-11-1019:39:13-21:14:0420111001[345], 20111100[1237]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
625160754mergeEvt: 6 missing datagrams in apid 956, gapfinder doesn't report any gaps2020-10-23
201023010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
624820000switch from hequs to the bullets2020-10-19





624516636mergeEvt: 6 missing datagrams in apid 956, gapfinder doesn't report any gaps2020-10-16
20101600[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
623917505mergeEvt: 7 missing datagrams in apid 9562020-10-0906:25:04-08:02:32201009007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
622822397mergeEvt: 5 missing datagrams in apid 9572020-09-2614:13:16-15:48:0620092601[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
622211988verify finds a gap of 3 datagrams in EPU0, and of 4 in EPU12020-09-19
20091901[02]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
622149260mergeEvt: 4 datagrams missing at the start of apid 9562020-09-1819:14:19-20:49:09200919001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621835509mergeEvt: incomplete apid 956, probably 7 datagrams missing at the end (apid 957 has 7 more)2020-09-15
200915005L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621824100mergeEvt: 8 missing datagrams in apid 9572020-09-15
200915003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621806986mergeEvt: 6 missing datagrams in apid 9572020-09-14
2009{14013,15001}L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621664373mergeEvt: 1 non closed datagram at the end of apid 957, gapfinder: no datagram gaps for 2020-09-132020-09-13
200913004L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621612709mergeEvt: 5 missing datagrams in apid 957, gapfinder: no datagram gaps for 2020-09-122020-09-1214:11:48-15:27:3220091201[012]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621600789mergeEvt: 5 missing datagrams in apid 957, gapfinder: no datagram gaps for 2020-09-122020-09-1210:53:08-12:01:40200912009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621435528mergeEvt: 5 missing datagrams in apid 956, gapfinder: no datagram gaps for 2020-09-102020-09-1012:58:47-14:10:3020091001[01]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621236535mergeEvt: 6 missing datagrams in apid 956, gapfinder: there was a gap an hour earlier, retransmitted2020-09-0805:42:14-07:17:0420090800[56]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
621081007mergeEvt: 5 missing datagrams in apid 956, gapfinder: no gaps for this run2020-09-0610:30:06-11:43:23200906010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620894265Michael screwed it up by accidentally rolling back the finished doRun.200904006.6208942652020-09-04
20090400[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620460723mergeEvt: 6 missing datagrams in apid 957, gapfinder: no gaps for 2020-08-302020-08-3006:12:02-07:46:5320083000[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620375156missing datagrams2020-08-29
200829007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
620369451caused 200829007.mergeIndices to fail because of an extra evt file. Repipe created identical evt files, thus L1Proc will not be rolled back, the run just needs to be finalized.2020-08-29
20082900[67]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
619313054mergeEvt: >1400 missing datagrams in both apids at the beginning of the run, gapfinder: no gaps around midnight of 2020-08-302020-08-1623:24:13-00:33:2020081700[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
618823233mergeEvt: 6 missing datagrams in apid 956, gapfinder: no gaps for the morning of 2020-08-112020-08-1107:20:32-08:36:4220073000[68]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
618805328mergeEvt: 8 missing datagrams in apid 957, gapfinder: no gaps for the morning of 2020-08-112020-08-1102:22:07-03:31:2620081100[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
617777656mergeEvt reports 8 missing datagrams in apid 956, but gapfinder doesn't report any gaps for 2020-07-302020-07-30
20073000[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
617190465mergeEvt reports 8 missing datagrams in apid 957, but gapfinder doesn't report any gaps for 2020-07-232020-07-23
200723007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615811484mergeEvt reports 6 missing datagrams, but gapfinder doesn't report any gaps2020-07-0710:44:43-12:19:342007070{09,10}L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615599718mergeEvt reports 6 missing datagrams, but gapfinder doesn't report any gaps2020-07-0404 23:55:17-05 01:06:3420070500[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
615508121one digitization fails: "... 2 CalDigis are missing readouts for Event ...".2020-07-0322:28:40-23:37:3320070314, 200704001L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
615073887missing datagrams. Gapfinder doesn't report any gaps for the evening of 2020-06-28.2020-06-2821:51:26-23:06:4220062900[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
614967205mergeEvt reports 6 data gaps in apid 957. Gapfinder doesn't report any gaps 2020-06-27.2020-06-2716:13:24-17:48:1520062701[23]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
614162399mergeEvt reports 5 datagaps in apid 956. However gapfinder doesn't find any gaps.2020-06-18
20061800[89]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612445752The run had data gaps, which were supposed to be filled by a redump. Unfortunately one piece was in the inaccessible part of the SSR. The run will stay incomplete.2020-05-29
20052900[89]L1Proc-5.7GlastRelease-20-10-04-gr02-(tick)
612183340HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. There were several data gaps 2020-05-26. At least some were filled.2020-05-2610:55:39-12:30:30200526007, 20052700[125]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612171931HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. There were several data gaps 2020-05-26. At least some were filled.2020-05-2607:45:30-09:20:2120052600[567], 200527001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
612166226HalfPipe labeled this run as "InComplete". MergeEvt said it's complete. However when repiping 200526005 (for 612171931) mergeIndices relabeled it, and failed with an error message hinting to this run. MK: the error was due to an evt file with slight different name. However, rolling back the HalfPipes resulted in exactly the same evt file contents. Thus, I didn't roll back L1Proc.2020-05-2606:10:25-07:14:1620052600[45]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
611880377HalfPipe labels the run as "InComplete". MergeEvt reports missing datagrams. Gapfinder doesn't report a gap for the duration of the run.2020-05-2222:46:16-23:58:37200522015, 20052300[12]L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)

Runs from December 2018 onward (L1Proc-5.7)

Run Number

Reason

Date

TimeDls

L1Proc

GR version

rePipe

L1Proc

565721985Botched attempt at a repipe, repiped and marked as Complete2018-12-05

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
565942990stuck "in progress", repiped and marked as Complete2018-12-08

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

(tick)

566022500HP "in progress", repiped and marked as Complete2018-12-09

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
566235626HP "in progress", repiped and marked as Complete 2018-12-11

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
567301141stuck "in progress", repiped and marked as Complete2018-12-23

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

567598970

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
567581889

Missing Data - Delivery 181227007 couldn't be dispatched - Marked as Incomplete and finalized.

2018-12-27

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
569069307in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-13


010 011L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
569213577

in progress and errors about missing datagrams (MK: update: data complete but no recon). MEM: rolled back checkRun and finalized.

2019-01-15
006 007L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572617673HP "in progress", repiped and marked as Complete 2019-02-23

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
572853518stuck "in progress". Leftover chunkToken. Fixed.2019-02-26

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
574781844"in progress", incomplete data, missing datagrams2019-03-20

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575044322"in progress", incomplete data, missing datagrams (MK: update: stat is InProgress, no recon file in db). MEM: apparently the halfPipe was rolled back after run was already reprocessed. Re-set as Complete and finalized.2019-03-23
013 014L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
575762229"in progress", incomplete data, missing datagrams2019-03-31

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
576305031Run was terminated by the "LAT configuration corruption" event. Marked as Incomplete and finalized.2019-04-07

L1Proc-5.7

GlastRelease-20-10-04-gr02

(error)(tick)
576658880"in progress", incomplete data2019-04-11

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)

578530168

corrupted data in monitoring2019-05-02

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
580907445in progress, mergeIndices reports 5 missing datagrams2019-05-30

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)
582073028in progress, mergeIndices reports missing datagrams2019-06-13

L1Proc-5.7

GlastRelease-20-10-04-gr02

(tick)(tick)
584730839in progress, complete data, missing datagrams2019-07-13

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585187986

This was actually complete. A botched rollback of halfPipe (190719003) produced leftover chunkTokens. Set to Complete and finalized.

2019-07-19

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585193691Both halfPipe and RePipe failed at delivering a complete set of data (this included a botched rollback of halfPipe 190719003). Set to Incomplete (due to 11 missing datagrams) and finalized.2019-07-19

L1Proc-5.7GlastRelease-20-10-04-gr02(error)(tick)
585535991in progress, complete data, missing datagrams2019-07-23

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
585690096in progress, missing datagrams2019-07-24

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586076888see thread. However it seems the recon file is complete. Reprocessing complete.  Everything seems to have worked including the cleanupCompleteRun process.2019-07-29

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
586893790bunch of tasks in "Waiting" state preventing a rollback.  Once that is cleared a rollback should hopefully fix everything2019-08-07

L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
586905200see thread . Seems the run is complete and can be just finalized. After checking, I've finalized this run as it looks fine -TS. MEM: same as 588742217, run was *not* finalized. 1. remove chunktokens as needed, 2. rollback checkRun, 3. make sure that cleanupCompleteRun is executed and recon file is merged. This is fine now. 2019-08-07

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
587042253Several failed tasks with errors of the form: "IOError: Can't stage in <filename>".  Looks like it was an xrootd hiccup.  Will try rolling back but since it's so old, it will probably require a repipe. Rollback worked now complete2019-08-09 12:46

L1Proc-5.7GlastRelease-20-10-04-gr02N/A(tick)
587104373

This one just looks like cleanupCompleteRun didn't run properly as everything else in both parts of the runs was listed as success.  There is a chunk token directory but no token.  However, the verify log complains about closing reasons for both EUP0 and EPU1. I'm think that means we need to repipe everything.

2019-08-10 6:54

L1Proc-5.7GlastRelease-20-10-04-gr02

(tick)

(tick)
588742217Run is incomplete according to the halfPipe. TS: I think this is okay as I don't see any errors in the verify log. MEM: run wasn't finalized because of leftover chunktokens. Please make sure that cleanupCompleteRun is executed before claiming success. This is fine now.2019-08-29

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
590065315Run still Incomplete in L1.  Looks like cleanupCompleteRun didn't get triggered.2019-09-13

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
591931054 failure in checkRun 191005003.591931054.  2 missing datagram.2019-10-05

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
5952403499 missing datagrams2019-11-12

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595405654missing datagrams2019-11-14

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595576882missing datagrams.  In progress2019-11-16

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
595963106Missing datagrams due to decompression errors- Both halfPipe and rePipe report 6789 orphans (which translate to 53 missing datagrams). Set to Incomplete and finalized.2019-11-2017:18:24 - 18:32:18

191120013 191121001

L1Proc-5.7GlastRelease-20-10-04-gr02 (error) (tick)
596299562HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-11-24 21:30-24:002019-11-2414:46:00 - 16:01:21

191124009

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
596916997missing datagrams2019-12-01

L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
597315084HalfPipe reports "inProgress", gapfinder doesn't report any data gaps for 2019-12-06 05:00-14:002019-12-0608:51:23 - 10:06:10191206009 010L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598223809HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-162019-12-1621:16:48 - 22:51:39191217001 002L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598581385HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2100:36:24 - 01:53:08191221003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598645966HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2019-12-212019-12-2118:32:45 - 20:07:36191221009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
598758794HalfPipe reports "inProgress"2019-12-2305:39:57 - 08:44:23191223003 004L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599272563HalfPipe reports "inProgress".  Datagram errors2019-12-292:22:25 - 4:58:04191229002 003L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599746995HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2020-01-032020-01-0312:23:14 - 13:58:05200103009 010 011L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
599809968HalfPipe reports "inProgress", missing datagrams, gapfinder doesn't report any data gaps for 2020-01-042020-01-0405:52:47 - 07:14:52200103009 010 011L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
600328891HalfPipe reports "inProgress".  Datagram errors2020-01-1010:43:13200110005L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
601777922HalfPipe reports "inProgress", but mergeEvt doesn't report any datagram errors!  Gapfinder doesn't report any data gaps for 2020-01-27.2020-01-2702:07:06 - 03:41:57200127002 003 004L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
601783627HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-27.2020-01-2700:32:01 - 02:06:55200127004 005 006L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
601890911HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for the morning of 2020-01-28.2020-01-2807:55:10 - 09:04:44200128006 008L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
602153762HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-01-31. No decompression errors in makeEvt.2020-01-3108:56:01 - 10:07:38200131007L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
603637679HalfPipe reports "inProgress".  Missing datagrams.  Gapfinder doesn't report any data gaps for 2020-02-17. No decompression errors in makeEvt.2020-02-1713:07:57 - 14:42:49200217009 010L1Proc-5.7GlastRelease-20-10-04-gr02 (tick) (tick)
603974262Run is complete.  mergeEvt.200221009.603974262 reports 6 missing datagrams. gapfinder doesn't report any data gaps for the time interval.2020-02-2110:37:41 - 12:12:31200221009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
604573262

HalfPipe reports "inProgress".  Missing datagrams. Gapfinder doesn't report any data gaps for 2020-02-28.

2020-02-2809:01:01 - 10:35:52200228008 - 009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
605908182HalfPipe reports "inProgress".  Missing datagrams. Gapfinder doesn't report any data gaps for 2020-03-15.2020-03-1519:49:41 - 21:24:31200315002L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
607059914Highly fragmented run (231 chunks). doRun/scanXxx processes fail.2020-03-28
200328005, 006, 008L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
609843485HalfPipe reports "InProgress". GapFinder doesn't report any data gaps of 2020-04-29.2020-04-29
200429007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610323623mergeIndices labeled this run as Incomplete, but mergeEvt as Complete! MK: Repipe resulted in the same chunks. Checking the old merit file didn't reveal any gaps within the run. Thus I relabeled the run to "Complete" and rolled-back checkRun.2020-05-05
200505001, 002L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610711547mergeEvt reports missing datagrams. Gapfinder doesn't report any data gaps for 2020-05-09.2020-05-0910:05:46 - 11:23:53200509008, 009L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610791211HalfPipe labels the run as Incomplete, mergeEvt reports missing datagrams. Gapfinder doesn't report any data gaps for 2020-05-10.2020-05-1008:13:30 - 09:30:17200510006, 007L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610968381mergeEvt reports missing datagrams2020-05-12
200512007, 008L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
610996774mergeEvt reports missing datagrams2020-05-12
200512014, 015L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
611162210mergeEvt reports missing datagrams2020-05-14
200514011, 012L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)
611782769HalfPipe labels the run as "Incomplete". MergeEvt reports missing datagrams. Gapfinder only reports a gap after 21:19.2020-05-2119:39:28 - 20:50:13200522001L1Proc-5.7GlastRelease-20-10-04-gr02(tick)(tick)

How to use the RePipe facility

/afs/slac/g/glast/ground/bin/pipeline createStream --define "downlinkID=990101001,runID=237063408" RePipe
/nfs/farm/g/glast/u28/RePipe/990101001 will contain the RePiped output (should this have a --stream <run number> flag in the command for clarity in the monitoring pages?)

if you set an environment variable, rn, to the run number (i.e. setenv rn <run number>) you can do the following command to start the repipe and have the output in the format expected for the instruction linked to below.

/afs/slac/g/glast/ground/bin/pipeline createStream --stream $rn --define "downlinkID=$rn,runID=$rn" RePipe

More details: Instructions for rolling back a halfPipe stream and susbsequent L1 stream and Reprocessing runs from a RePiped job

Runs before December 2018 but after July 27 2011 (L1Proc-5.6 and earlier)

Run Number

Reason

Date

L1Proc

GR version

rePipe

L1(Re)Proc

Export
369152685 ACQ Complete, L1 not finalized. Fixed in P300x.2012-09-12L1Proc-4.0GR-17-35-24-gr35
4.0
478706331 ACQ Complete, L1 not finalized. Fixed in P300x.2016-03-03L1Proc-5.4GR-20-10-02
5.4
478756313ACQ Complete, L1 not finalized. Fixed in P300x.2016-03-04L1Proc-5.4

GR-20-10-02

5.4
504919670Leap Second Run. Set to Incomplete and finalized.2016-12-31L1Proc-5.6

GR-20-10-04-gr01

(error)
5.6
508046861ACQ InProgress. Repiped and reprocessed.2017-02-09L1Proc-5.6

GR-20-10-04-gr01

5.7
512115535ACQ Complete, L1 Complete/Failed. Fixed in P300x.2017-03-25L1Proc-5.6

GR-20-10-04-gr01

5.6
516617903ACQ Complete, L1 Complete/Failed. Fixed in P300x. 2017-05-16L1Proc-5.6

GR-20-10-04-gr01

5.6
531419768ACQ Complete, L1 Complete , recon on xrootd (root://fermi-gpfs05//glast/Data/Flight/Level1/LPA/prod/5.6/recon/r0531419768_v000_recon.root) but not in datacat. Run was reprocessed correctly in P300x. No follow-up needed.2017-11-03L1Proc-5.6GR-20-10-04-gr01
5.6
534500370ACQ Complete, L1 Complete/Failed. Fixed in P300x. 2017-12-09L1Proc-5.6

GR-20-10-04-gr01

5.6
549415721 ACQ Complete, L1 Complete , recon on xrootd (root://fermi-gpfs05//glast/Data/Flight/Level1/LPA/prod/5.6/recon/r0549415721_v000_recon.root) but not in datacat. Run was reprocessed correctly in P300x. No follow-up needed. 2018-05-31L1Proc-5.6GR-20-10-04-gr01
5.6
553318321ACQ Complete, L1 Failed. Repiped and reprocessed.2018-07-16L1Proc-5.6

GR-20-10-04-gr01

5.7
554179852ACQ Complete, L1 Failed. Repiped and reprocessed. 2018-07-25L1Proc-5.6

GR-20-10-04-gr01

5.7
554396875 ACQ Complete, cleanupCompleteRun.getTimes terminated by reaper. Run was reprocessed correctly in P300x. No follow-up needed. 2018-07-27L1Proc-5.6GR-20-10-04-gr01
5.6
555737292ACQ Complete, L1 InProgress. Finalized.2018-08-12L1Proc-5.6

GR-20-10-04-gr01

5.6
557026900ACQ Complete, L1 InProgress. Finalized.2018-08-27L1Proc-5.6

GR-20-10-04-gr01

5.6
559965007Truncated Run. Set to Incomplete and finalized. 2018-09-30L1Proc-5.6

GR-20-10-04-gr01

(error)
5.6
562253064Missing Data. Set to Incomplete and finalized. Missing data sent by MOC 9/24/19, reprocessed with 5.7 - versions numbers generated are less than the v305 version from the reprocessing so ingest is not happening at the FSSC. File version edited by hand and re-exported.
2018-10-26L1Proc-5.6

GR-20-10-04-gr01

(tick)
5.7
562258990This run was missing entirely. Retrieved by MOC 9/24/19 and reprocessed with 5.7.2018-10-26L1Proc-5.6

GR-20-10-04-gr01

(tick)5.7 (tick)
562264544Missing Data. Set to Incomplete and finalized. Missing data sent by MOC 9/24/19, reprocessed with 5.7 - versions numbers generated are less than the v305 version from the reprocessing so ingest is not happening at the FSSC. File version edited by hand and re-exported.
2018-10-27L1Proc-5.6

GR-20-10-04-gr01

(tick)5.7 (tick)

564602913

Repiped, reprocessed with 5.7, re-exported.2018-11-22L1Proc-5.6

GR-20-10-04-gr01

5.7 (tick)(tick)

Runs from LEO through 2009

Run Number

Reason

Date

FSW

L1Proc

GR version

rePipe

L1Proc

237911263

silent NFS read error

2008-07-16

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

237916725

silent NFS read error

2008-07-16

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

(tick)

238008393

silent NFS read error

2008-07-17

B1-0-8

L1Proc-1.63

Version-v15r39

(tick)

(tick)

238014123

silent NFS read error

2008-07-17

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238207772

silent NFS read error

2008-07-20

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238759073

silent NFS read error

2008-07-26

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

238856333

silent NFS read error

2008-07-26

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

239125611

silent NFS read error

2008-07-30

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

239129222

silent NFS read error

2008-07-30

B1-0-8

L1Proc-1.61

Version-v15r24

(tick)

(error)

241125159

1 missing Datagram

2008-08-22

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

241491796

missing chunk in in gcr

2008-08-27

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

241501554

1 missing Datagram

2008-08-27

B1-1-0

L1Proc-1.63

Version-v15r39

(error)

(error)

243170470

afs cache corruption?

2008-09-15

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243197713

afs cache corruption?

2008-09-15

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243307762

2 missing Datagrams

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243313738

2 missing Datagrams

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243336617

afs cache corruption?

2008-09-17

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243491490

4 missing Datagrams

2008-09-19

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243873727

afs cache corruption?

2008-09-23

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

243926731

afs cache corruption?

2008-09-24

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

244143826

afs cache corruption?

2008-09-26

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

244167563

afs cache corruption?

2008-09-27

B1-1-0

L1Proc-1.65

Version-v15r40

(error)

(error)

245667862

irretrievable packets missing from two datagrams

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(error)

(error)

245673843

This one was complete, but a retransmission changed its status

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245685781

all packets available; repiping will produce complete data-set

2008-10-14

B1-1-3

L1Proc-1.69

Version-v15r47p1

(tick)

(tick)

245691461

all packets available; repiping will produce complete data-set

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245697061

This one was already complete

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(tick)

(tick)

245702790

irretrievable packet missing from one datagram

2008-10-14

B1-1-3

L1Proc-1.67

Version-v15r47p1

(error)

(error)

246390295

Merging problems

2008-10-22

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

246405636

Merging problems

2008-10-22

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

246447791

Merging problems

2008-10-23

B1-1-3

L1Proc-1.68

Version-v15r47p1

(tick)

(tick)

253539991

Delivery problems ?

2009-01-13

B1-1-3

L1Proc-1.70

Version-v15r47p1

(tick)

(tick)

260478011

1 Missing datagram ???

2009-04-03

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261148032

Insane number of chunks

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261154012

Insane number of chunks

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

261406015

1 Missing datagram ???

2009-04-11

B1-1-3

L1Proc-1.72

Version-v15r47p7

(tick)

(tick)

269048825

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269054554

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269060283

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269066012

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269071741

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269075743

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269081879

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269087937

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269093942

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

269099929

Silent staging failures

2009-07-12

B1-1-3

L1Proc-1.73

Version-v15r47p7

(tick)

(tick)

272627642

1 bad event (seg fault)

2009-08-22

B1-1-3

L1Proc-1.75

Version-v15r47p12

(tick)

(tick)

  • No labels