Page History

status: In Progress Complete
last update: 05 07 Nov 2013

This page is a record of the configuration and execution of the P202 reprocessing project, full reprocessing from DIGIs using Pass7 analysis code. This project involves reprocessing with Pass7 classification trees and up-to-date alignment/calibration data. This task will read DIGI files and emit RECON, MERIT, GCR and CAL ROOT files, and the standard array of FITS files. It will be a CPU-intensive and storage-intensive enterprise requiring months of elapsed time and of order 0.7 Pbyte of storage. At the time of this task beginning, there will be about 20,000 science runs in Fermi (3.5 years accumulation).

...

9/17/2013
- 13:33 Begin block 11 backfill, 539 runs in the range 398087830 through 401106634 (2013-09-17 11:48:16). Once complete, 28412 runs will have been reprocessed.
9/19/2013
- 08:30 block 11 complete. The bulk of runs completed within 14 hours, but stragglers, failed/terminated jobs, due to transient problems, bad batch machines, etc., delayed completion until this morning. DataCatalog reports a total of 61,827,628,706 events in {CAL,MERIT,RECON,GCR} and 8,991,163,530 events in FILTEREDMERIT.
- xroot scratch space cleaned up.
10/4/2013
- Prepare for block 12 backfill. Warren suggests ending with run 402560477 (2013-10-04 06:21:14 UTC). First run of block 12 is 401112810. There are 263 runs in this backfill block which will bring the reprocessed run total to 28675.
10/15/2013
- Prepare for block 13 backfill. Warren suggests ending with run 403510814, 170 new runs.
  
  Run
  Run start time (UTC)
  Task Stream
  Block 13 start
  402566464
  2013-10-04 08:01:01
  28675
  Block 13 end
  403510814
  2013-10-15 06:20:11
  28844
11/05/2013
- Today P7REP goes public.
- Prepare for block 14 backfill – the ultimate backfill for P7REP. Last run = 405329691 (2013-11-05 07:34:48)
- 313 new runs to reprocess...
  
  Run
  Run start time (UTC)
  Task Stream
  Block 14 start
  403516762
  2013-10-15 07:59:19
  28845
  Block 14 end
  405329691
  2013-11-05 07:34:48
  29157

Configuration

8/19/2014
- User reported problem with an event inside a ~195s interval with LIVETIME=0 in run 395891323 (2013-07-19 01:48:40 UTC). Problem traced to a bad FT2SECONDS file. Reprocessing took place 14 Aug 2013 using v002 FT2 file. Then on 19 Aug 2013, the FT2 file was updated to v003. M.E. has rebuilt a proper FT2 file in the Reprocess/P202 area, v203 and will roll back entire stream 27487 to rebuild ROOT files. To get this rollback to run, the following steps were necessary:
  1. config.py - change ft2Selection from 'P105' to 'P202' (commonTools/repTools.py:findFT2() already knows about P202, so no changes needed there.)
  2. commonTools/setupGR.sh - change svsopts from 'redhat5' back to 'redhat4' (TEMPORARILY) for processClump step
  3. commonTools/setupSkimmer.py - a series of reversions:
    - revert to skimmer 08-02-01
    - revert to rootVersion = 'v5.26.00a-gl6'
    - revert glastBuild from redhat5-i686-32bit-gcc41 to
    redhat4-i686-32bit-gcc34 to get libraries for mergeClumps step.
    - revert to old ROOTSYS definition that includes compiler version as last path item
- The mods to setupGR.sh and setupSkimmer.py have been preserved as OLD-setupGR.sh and OLD-setupSkimmer.py in the commonTools directory

Configuration

Task Location	/nfs/farm/g/glast/u38/Reprocess-tasks/P202-ROOT
Task Status	http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp
GlastRelease	17-35-24-gr17 and 17-35-24-rp04 (SCons RHEL4-32 build)
Run Selection	based on a modified "standard" selection, see https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets (((sIntent=="nomSciOps" \|\| sIntent=="nomSO
Task Location	/nfs/farm/g/glast/u38/Reprocess-tasks/P202-ROOT
Task Status	http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp
GlastRelease	17-35-24-gr17 and 17-35-24-rp04 (SCons RHEL4-32 build)
Run Selection	based on a modified "standard" selection, see https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets (((sIntent=="nomSciOps" \|\| sIntent=="nomSO_noSk_noCno_optGccc_allEna" \|\| sIntent=="nomSciOps_diagEna" \|\| (sIntent=="nomSciOps_Emin5MeV"&&RunMin>242070455) \|\| nRun==242429468 ) && (RunQuality != "Bad" \|\| is_null ( RunQuality ) ) ) \|\| sIntent=="nadirOps" )
s/c data	"standard" Public Release 2 https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets
Input Run List	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-ROOT/config/runList.txt
photonFilter	CTBParticleType==1 && ((FT1EventClass & 0x00003EFF)!=0) pass7.6_Extended_cuts_L1 in evtClassDefs
electronFilter	CTBParticleType==1
Code Variants	redhat4-i686-32bit-gcc34 (Optimized)
jobOpts	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-ROOT/config/doRecon.txt
Output Data Products	RECON, GCR, CAL, MERIT, FILTEREDMERIT, ELECTRONMERIT

...

3/2/2012 - Define block 1 as the 776 runs in P202-ROOT block 1. Configure trickleStream and begin (14:08)
3/31/2012 - Define block 2 as 5600 runs. Reconfig trickleStream and begin (18:05)
4/01/2012 - Block 2 complete (most of the 4824 jobs completed in about six hours w/1000 job limit).
5/31/2012
- Discover stream 5599 (run 271999199) requires rollback - new MERIT file version (v203->v205).
- Regenerate runlist with 20,229 input MERIT files.
- Minor config change: twoClumpMin=False (formerly True, but caused unnecessary extra processClump dummy jobs)

6/5/2012 - Final cleanup

Five runs are responsible for discrepant event tallies:

Run	Stream	Reason	Action
239557414	0	bad TCut	rollback ROOT + FITS
241599746	352	bad processing order	rollback FITS
245403855	1019	bad start time	change start time in runList and rollback
332661583	16244	silent root errors	rollback ROOT
339081502	17416	silent root errors	rollback ROOT

Code changes:

directory	script	modification
commonTools/00-01-00	repTools.py	added new getKey() function to extract #events from FITS event files
P202-FITS/config	config.py	added os.environ['HEADASNOQUERY']='true' to enable 'ftlist' to run in batch
P202-FITS/config	makeFITS.py	add diagnostic print of #evts in FITS files for each processing sub-step

After these actions, the dataCatalog tallies are now consistent:

Name	Type	Files	Events	Size	Created (UTC)	Links
ELECTRONFT1	Group	20229	0	8.5 GB	02-Mar-2012 00:06:07	Files
ELECTRONMERIT	Group	20229	90,904,582	205.7 GB	25-Jan-2012 00:53:32	Files
FT1	Group	20229	189,323,074	17.8 GB	02-Mar-2012 00:06:06	Files
LS1	Group	20229	1,325,204,821	215.3 GB	02-Mar-2012 00:06:08	Files
EXTENDEDFT1	Group	20229	6,291,424,926	574.7 GB	02-Mar-2012 00:06:09	Files
EXTENDEDLS1	Group	20229	6,291,424,926	1,020.1 GB	02-Mar-2012 00:06:09	Files
FILTEREDMERIT	Group	20229	6,291,424,926	5.3 TB	25-Jan-2012 00:53:29	Files
MERIT	Group	20229	44,125,679,961	35.4 TB	25-Jan-2012 00:53:30	Files
RECON	Group	20229	44,125,679,961	590.0 TB	25-Jan-2012 00:53:33	Files
GCR	Group	20229	44,125,679,961	942.7 GB	25-Jan-2012 00:53:31	Files
CAL	Group	20229	44,125,679,961	128.7 TB	25-Jan-2012 00:53:31	Files

Note that the number of events in ELECTRONFT1 files are not currently tallied by the dataCatalog.

8/24/2012 - Configure and run backfill through 31 July 2012

10/7/2012 - Rollback the following seven streams to fix corrupt FITS files. The makeFT1 app received a large number of ROOT error, but terminated normally. The resulting FT1/LS1 files had multiple symptoms, including one discovered by the FSSC during transfer to them: one EVENT_ID was repeated ~1000 times. Scan of log files uncovered an additional five mergeClumps with these errors.

Code Block

%INFO: 20120331:20:34:32 - makeFITS(run)/line-127 - Running: makeFT1
---------------- start commentary ----------------
About to run [time makeFT1 rootFile=/scratch/glastmp/P202-FITS/3345/r0259101994_v202_merit.root fitsFile=/scratch/glastmp/P202-FITS/3345/gll_xp_p202_r0259101994_v202.fit TCuts=/afs/slac.stanford.edu/g/glast/ground/releases//volume04/evtClassDefs/00-19-04/data/pass7.6_Extended_cuts_L1 dict_file=/afs/slac.stanford.edu/g/glast/ground/releases//volume04/evtClassDefs/00-19-04/data/FT1variables tstart=259101996.933000 tstop=259106272.085000 file_version=202 tempRootFile=/scratch/glastmp/P202-FITS/3345/dummy.root xml_classifier=/afs/slac.stanford.edu/g/glast/ground/releases//volume04/evtClassDefs/00-19-04/xml/EvtClassDefs_P7V6.xml evtclsmap=FT1EventClass chatter=4 debug=yes] at Sat Mar 31 20:34:32 2012
---------------- start log ----------------
This is makeFT1 version ScienceTools-09-27-01
applying TCut: ((FT1EventClass & 0x00003EFF)!=0) && (EvtElapsedTime >= 259101996)  && (EvtElapsedTime <= 259106273))
Warning in <TClass::TClass>: no dictionary for class FileHeader is available
Warning in <TClass::TClass>: no dictionary for class FileHeaderRootObj<int> is available
Warning in <TClass::TClass>: no dictionary for class RootObj<int> is available
R__unzip: error in header
Error in <TBasket::ReadBasketBuffers>: fNbytes = 15550, fKeylen = 86, fObjlen = 31912, noutot = 0, nout=0, nin=11010118, nbuf=7864444
Error in <TBranch::GetBasket>: File: /scratch/glastmp/P202-FITS/3345/r0259101994_v202_merit.root at byte:459100837, branch:Vtx2LongDoca, entry:510338, badread=0, nerrors=1, basketnumber=1
[...]

Stream	Run
3345	259101994	<- found by FSSC
4122	263571912
4707	266893978
13927	319436826
16181	332306548	<- found by FSSC
17430	339161346
17479	339408141

10/15/2012 - Reconfigure for block 4 backfill and begin trickleStream.
- Discover run 22240 makeFT1 contains error messages as the mentioned last week (above). Rolling back the entire chain of processing, starting with P202-ROOT seemed to do the trick.
10/17/2012 - block4 backfill complete.
12/18/2012 - Prepare for block 5. Before:
Name
Type
Files
Events
Size
EXTENDEDFT1
Group
23141
7,252,996,601
662.6 GB
EXTENDEDLS1
Group
23141
7,252,996,601
1.1 TB
FT1
Group
23141
229,126,932
21.5 GB
LS1
Group
23141
1,549,133,207
251.7 GB
12/19/2012 - block5 backfill complete. Final DataCatalog numbers:
Name
Type
Files
Events
Size
EXTENDEDFT1
Group
24142
7,578,468,156
692.3 GB
EXTENDEDLS1
Group
24142
7,578,468,156
1.2 TB
FT1
Group
24142
240,949,332
22.6 GB
LS1
Group
24142
1,621,773,636
263.5 GB

3/6/2013 - Many changes!

New task created (version 0.9 -> 1.0)
Update to run native on RHEL5-64 and RHEL6-64
New Interstellar Emission Model, v2r0 -> v3r0, provided by Luigi Tibaldo.
Update ScienceTools from 09-27-01 to 09-31-01
Update IRFS from P7*V6 to P7REP_V10 ( = CLEAN or SOURCE)
This means that rather than rolling-back, the task starts from scratch.
New evtClassDiffs version: 00-19-05, which changes P7V6 -> P7REP, the new designation.

Reference list of all existing FITS files (with old diffuse model) generated and stored here: /nfs/farm/g/glast/u38/Reprocess-tasks/P202-FITS/config/task-v0.9

Code Block

diffmodel_p7rep_clean_v10_reduced.xml
diffmodel_p7rep_clean_v10.xml
diffmodel_p7rep_source_v10_reduced.xml
diffmodel_p7rep_source_v10.xml
gal_p7rep_v10_v1.fits
gal_p7rep_v10_v1_reduced.fits
iso_p7rep_clean_v10_back_v1.txt
iso_p7rep_clean_v10_front_v1.txt
iso_p7rep_clean_v10_v1.txt
iso_p7rep_source_v10_back_v1.txt
iso_p7rep_source_v10_front_v1.txt
iso_p7rep_source_v10_v1.txt

3/7/2013 - Run off 10 test runs (starting at beginning of mission with task version 1.0)

5/30/2013 - Green light given on new IRFs and diffuse model.
- Changes to config.py:
  - New diffuse model: /afs/slac/g/glast/ground/GLAST_EXT/diffuseModels/v4r0
  - New IRFs: P7REP_SOURCE_V15,P7REP_CLEAN_V15
- Awaiting a new ScienceTools release...
6/10/2013 - Several ST updates, and renaming of the diffuse model files. All appears good to go.
6/11/2013 - final tweaks to diffuse model v4r0 files; bump task version 1.0 -> 1.1 (delete version 1.0); and start...
6/14/2013 - block 1 complete (26,263 runs: 239557414 through 389089696; MET 2008-08-04 15:43:33 through 2013-05-01 08:28:13)

Name	Files	Events	Size
CAL	26263	57,227,320,767	166.0 TB
DIGIGAP	24200	0	19.0 kB
ELECTRONFT1	26263	0	10.9 GB
ELECTRONMERIT	26263	116,799,950	263.6 GB
EXTENDEDFT1	26263	8,284,002,713	725.9 GB
EXTENDEDLS1	26263	8,284,002,713	1.3 TB
FILTEREDMERIT	26263	8,284,037,323	7.0 TB
FT1	26263	268,810,274	24.2 GB
GCR	26263	57,227,320,767	1.2 TB
LS1	26263	1,782,493,106	289.6 GB
MERIT	26263	57,227,320,767	45.8 TB
RECON	26263	57,227,320,767	763.1 TB

Discrepancy between FILTEREDMERIT and EXTENDED{LS1,FT1}. This turns out to be an issue with tstart/tstop for run 383219654.


R__unzip: error in header
Error in <TBasket::ReadBasketBuffers>: fNbytes = 15550, fKeylen = 86, fObjlen = 31912, noutot = 0, nout=0, nin=11010118, nbuf=7864444
Error in <TBranch::GetBasket>: File: /scratch/glastmp/P202-FITS/3345/r0259101994_v202_merit.root at byte:459100837, branch:Vtx2LongDoca, entry:510338, badread=0, nerrors=1, basketnumber=1
[...]

Stream	Run
3345	259101994	<- found by FSSC
4122	263571912
4707	266893978
13927	319436826
16181	332306548	<- found by FSSC
17430	339161346
17479	339408141

10/15/2012 - Reconfigure for block 4 backfill and begin trickleStream.
- Discover run 22240 makeFT1 contains error messages as the mentioned last week (above). Rolling back the entire chain of processing, starting with P202-ROOT seemed to do the trick.
10/17/2012 - block4 backfill complete.
12/18/2012 - Prepare for block 5. Before:
Name
Type
Files
Events
Size
EXTENDEDFT1
Group
23141
7,252,996,601
662.6 GB
EXTENDEDLS1
Group
23141
7,252,996,601
1.1 TB
FT1
Group
23141
229,126,932
21.5 GB
LS1
Group
23141
1,549,133,207
251.7 GB
12/19/2012 - block5 backfill complete. Final DataCatalog numbers:
Name
Type
Files
Events
Size
EXTENDEDFT1
Group
24142
7,578,468,156
692.3 GB
EXTENDEDLS1
Group
24142
7,578,468,156
1.2 TB
FT1
Group
24142
240,949,332
22.6 GB
LS1
Group
24142
1,621,773,636
263.5 GB

3/6/2013 - Many changes!

New task created (version 0.9 -> 1.0)
Update to run native on RHEL5-64 and RHEL6-64
New Interstellar Emission Model, v2r0 -> v3r0, provided by Luigi Tibaldo.
Update ScienceTools from 09-27-01 to 09-31-01
Update IRFS from P7*V6 to P7REP_V10 ( = CLEAN or SOURCE)
This means that rather than rolling-back, the task starts from scratch.
New evtClassDiffs version: 00-19-05, which changes P7V6 -> P7REP, the new designation.

Reference list of all existing FITS files (with old diffuse model) generated and stored here: /nfs/farm/g/glast/u38/Reprocess-tasks/P202-FITS/config/task-v0.9

Code Block

diffmodel_p7rep_clean_v10_reduced.xml
diffmodel_p7rep_clean_v10.xml
diffmodel_p7rep_source_v10_reduced.xml
diffmodel_p7rep_source_v10.xml
gal_p7rep_v10_v1.fits
gal_p7rep_v10_v1_reduced.fits
iso_p7rep_clean_v10_back_v1.txt
iso_p7rep_clean_v10_front_v1.txt
iso_p7rep_clean_v10_v1.txt
iso_p7rep_source_v10_back_v1.txt
iso_p7rep_source_v10_front_v1.txt
iso_p7rep_source_v10_v1.txt

3/7/2013 - Run off 10 test runs (starting at beginning of mission with task version 1.0)

5/30/2013 - Green light given on new IRFs and diffuse model.
- Changes to config.py:
  - New diffuse model: /afs/slac/g/glast/ground/GLAST_EXT/diffuseModels/v4r0
  - New IRFs: P7REP_SOURCE_V15,P7REP_CLEAN_V15
- Awaiting a new ScienceTools release...
6/10/2013 - Several ST updates, and renaming of the diffuse model files. All appears good to go.
6/11/2013 - final tweaks to diffuse model v4r0 files; bump task version 1.0 -> 1.1 (delete version 1.0); and start...
6/14/2013 - block 1 complete (26,263 runs: 239557414 through 389089696; MET 2008-08-04 15:43:33 through 2013-05-01 08:28:13)

Name	Files	Events	Size
CAL	26263	57,227,320,767	166.0 TB
DIGIGAP	24200	0	19.0 kB
ELECTRONFT1	26263	0	10.9 GB
ELECTRONMERIT	26263	116,799,950	263.6 GB
EXTENDEDFT1	26263	8,284,002,713	725.9 GB
EXTENDEDLS1	26263	8,284,002,713	1.3 TB
FILTEREDMERIT	26263	8,284,037,323	7.0 TB
FT1	26263	268,810,274	24.2 GB
GCR	26263	57,227,320,767	1.2 TB
LS1	26263	1,782,493,106	289.6 GB
MERIT	26263	57,227,320,767	45.8 TB
RECON	26263	57,227,320,767	763.1 TB

Discrepancy between FILTEREDMERIT and EXTENDED{LS1,FT1}. This turns out to be an issue with tstart/tstop for run 383219654.

8/16/2013 - configure block2 backfill: 1610 new runs to reprocess
- 27873 total runs
- 60,682,674,790 total events
- Start run: 239557417 2008-08-04 15:43:37
- Last run: 398086126 2013-08-13 11:28:46
8/21/2013 - block2 complete (after redoing the entire block due to bizarre effects of last Friday's fermi-xrd005 crash). DataCatalog contents as of this morning:

Name	Files	Events	Size	Created (UTC)
CAL	27873	60,682,674,790	175.8 TB	25-Jan-2012 00:53:31
ELECTRONFT1	27873	0	11.5 GB	02-Mar-2012 00:06:07
ELECTRONMERIT	27873	123,494,286	278.5 GB	25-Jan-2012 00:53:32
EXTENDEDFT1	27873	8,811,129,094	772.1 GB	02-Mar-2012 00:06:09
EXTENDEDLS1	27873	8,811,129,094	1.4 TB	02-Mar-2012 00:06:09
FILTEREDMERIT	27873	8,811,129,090	7.5 TB	25-Jan-2012 00:53:29
FT1	27873	289,969,364	26.1 GB	02-Mar-2012 00:06:06
GCR	27873	60,682,674,790	1.3 TB	25-Jan-2012 00:53:31
LS1	27873	1,903,568,484	309.3 GB	02-Mar-2012 00:06:08
MERIT	27873	60,682,674,790	48.6 TB	25-Jan-2012 00:53:30
RECON	27873	60,682,674,790	808.5 TB	25-Jan-2012 00:53:33

9/19/2013 - prepare block3 (corresponding to P202-ROOT block11).
9/20/2013 - block3 complete. One run from block2 was rolledback due to not using the latest MERIT file (run 383219654). Statistics are now consistent.

Image Added

10/7/2013 - prepare block4 (corresponding to P202-ROOT block12).
10/8/2013 - block 4 complete
8/16/2013 - configure block2 backfill: 1610 new runs to reprocess
- 27873 total runs
- 60,682,674,790 total events
- Start run: 239557417 2008-08-04 15:43:37
- Last run: 398086126 2013-08-13 11:28:46
8/21/2013 - block2 complete (after redoing the entire block due to bizarre effects of last Friday's fermi-xrd005 crash). DataCatalog contents as of this morning:
Name
Files
Events
Size
Created (UTC)

...

ELECTRONMERIT

...

28675

...

126,

...

895,

...

805

...

286.

...

1 GB
25-Jan-2012 00:53:

...

32

...

FT1

...

27873

...

0

...

11.5 GB

...

02-Mar-2012 00:06:07

...

ELECTRONMERIT

...

27873

...

123,494,286

...

278.5 GB

28675
301,166,782
27.1 GB
02-Mar

...

-2012 00:

...

06:

...

06

...

LS1

...

28675

...

1,

...

964,

...

328,

...

038

...

319.1 GB
02-Mar-2012 00:06:

...

08
EXTENDEDLS1

...

28675

...

9,

...

072,

...

793,

...

026
1.4 TB
02-Mar-2012 00:06:09
FILTEREDMERIT

...

28675

...

9,

...

072,

...

793,

...

026
7.

...

7 TB
25-Jan-2012 00:53:29

...

EXTENDEDFT1

...

28675

...

9,072,

...

793,

...

026

...

795.

...

0 GB
02-Mar-2012 00:06:

...

09

...

MERIT

...

28675

...

62,

...

383,

...

754,

...

1.3 TB

...

25-Jan-2012 00:53:31

997
49.9 TB
25-Jan

...

LS1

...

27873

...

1,903,568,484

...

309.3 GB

...

-2012 00:

...

53:

...

30

...

CAL

...

28675

...

62,

...

383,

...

754,

...

997

...

180.6 TB
25-Jan-2012 00:53:

...

31

...

GCR

...

28675

...

62,

...

383,

...

754,

...

997

...

1.

...

3 TB
25-Jan-2012 00:53:

...

31
RECON
28675
62,383,754,997
831.0 TB
25-Jan-2012 00:53:33
10/17/2013 - prepare block5 backfill (P202-ROOT block13

9/19/2013 - prepare block3 (corresponding to P202-ROOT block11).
9/20/2013 - block3 complete. One run from block2 was rolledback due to not using the latest MERIT file (run 383219654). Statistics are now consistent.

Image Removed

10/7/2013 - prepare block4 (corresponding to P202-ROOT block12).

10/824/2013 - block 4 complete5 complete (as of 10/18/2013)

Name	Files	Events	SizeCreated (UTC)
ELECTRONMERIT	2867528844	126127,895613,805745	286287.1 GB25-Jan-2012 00:53:327 GB
FT1	2867528844	301303,166135,782818	27.1 3 GB	02-Mar-2012 00:06:06
LS1	2867528844	1,964976,328418,038512	319321.1 GB02-Mar-2012 00:06:08
EXTENDEDLS1	2867528844	9,072127,793005,026732	1.4 TB	02-Mar-2012 00:06:09
FILTEREDMERIT	2867528844	9,072127,793005,026732	7.7 TB	25-Jan-2012 00:53:29
EXTENDEDFT1	2867528844	9,072127,793005,026732	795799.0 GB02-Mar-2012 00:06:098 GB
MERIT	2867528844	62,383747,754048,997217	4950.9 2 TB	25-Jan-2012 00:53:30
CAL	2867528844	62,383747,754048,997217	180181.6 TB	25-Jan-2012 00:53:31
GCR	2867528844	62,383747,754048,997217	1.3 TB	25-Jan-2012 00:53:31
RECON	2867528844	62,383747,754048,997217	831835.0 8 TB	25-Jan-2012 00:53:33

10/17/2013 - prepare block5 backfill (P202-ROOT block13).

11/7/2013 - block 6 complete and P7REP was made public two days ago. END OF PROJECT

Final tally by the dataCatalog:

Name	Files	Events	Size	Created (UTC)
CAL	29158	63,449,261,789	183.6 TB	25-Jan-2012 00:53:31
ELECTRONMERIT	29158	128,971,706	290.7 GB	25-Jan-2012 00:53:32
EXTENDEDFT1	29158	9,243,063,549	809.9 GB	02-Mar-2012 00:06:09
EXTENDEDLS1	29158	9,243,063,549	1.5 TB	02-Mar-2012 00:06:09
FILTEREDMERIT	29158	9,243,063,549	7.8 TB	25-Jan-2012 00:53:29
FT1	29158	310,326,817	27.9 GB	02-Mar-2012 00:06:06
GCR	29158	63,449,261,789	1.3 TB	25-Jan-2012 00:53:31
LS1	29158	2,007,123,864	326.1 GB	02-Mar-2012 00:06:08
MERIT	29158	63,449,261,789	50.8 TB	25-Jan-2012 00:53:30
RECON	29158	63,449,261,789	845.0 TB	25-Jan-2012 00:53:33

8/20/2014 - Using new MERIT file from P202-ROOT (see log entry above), update fullList.txt and then rollback/regenerate FITS files for run 395891323, stream 27487.
10/24/2013 - block 5 complete (as of 10/18/2013)
Name
Files
Events
Size
ELECTRONMERIT
28844
127,613,745
287.7 GB
FT1
28844
303,135,818
27.3 GB
LS1
28844
1,976,418,512
321.1 GB
EXTENDEDLS1
28844
9,127,005,732
1.4 TB
FILTEREDMERIT
28844
9,127,005,732
7.7 TB
EXTENDEDFT1
28844
9,127,005,732
799.8 GB
MERIT
28844
62,747,048,217
50.2 TB
CAL
28844
62,747,048,217
181.6 TB
GCR
28844
62,747,048,217
1.3 TB
RECON
28844
62,747,048,217
835.8 TB

Configuration

Task Location	/nfs/farm/g/glast/u38/Reprocess-tasks/P202-FITS
Task Status	http://glast-ground.slac.stanford.edu/Pipeline-II/task.jsp?task=107152539
Input Data	MERIT (direct from P202-ROOT)
spacecraft data	same as P202-ROOT
Input Run List	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-FITS/config/runList.txt
evtClassDefs	00-19-05 (March 2013, changed pass_ver to P7REP)
eventClassMap	EvtClassDefs_P7V6.xml
ScienceTools	09-32-03 (6/7/2013) (but ST may report themselves as 09-32-02 due to RM snafu)
Code Variants	redhat5-x86_64-64bit-gcc41, redhat6-x86_64-64bit-gcc44 (Optimized)
Diffuse Model	based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v4r0 (see https://confluence.slac.stanford.edu/display/SCIGRPS/Quick+Start+with+Pass+7)
Diffuse Response	'source' using P7REP_SOURCE_V15 IRF 'clean' using P7REP_CLEAN_V15 IRF
IRFs	P7REP_*_V15, contained within ScienceTools release
Output Data Products	FT1, LS1, EXTENDEDFT1, EXTENDEDLS1, ELECTRONFT1

...

Space shortcuts

Child pages

Versions Compared

Old Version 136

New Version Current

Key

Configuration

Configuration

Configuration

	Run	Run start time (UTC)	Task Stream
Block 13 start	402566464	2013-10-04 08:01:01	28675
Block 13 end	403510814	2013-10-15 06:20:11	28844

	Run	Run start time (UTC)	Task Stream
Block 14 start	403516762	2013-10-15 07:59:19	28845
Block 14 end	405329691	2013-11-05 07:34:48	29157

Name	Type	Files	Events	Size
EXTENDEDFT1	Group	23141	7,252,996,601	662.6 GB
EXTENDEDLS1	Group	23141	7,252,996,601	1.1 TB
FT1	Group	23141	229,126,932	21.5 GB
LS1	Group	23141	1,549,133,207	251.7 GB

Name	Type	Files	Events	Size
EXTENDEDFT1	Group	24142	7,578,468,156	692.3 GB
EXTENDEDLS1	Group	24142	7,578,468,156	1.2 TB
FT1	Group	24142	240,949,332	22.6 GB
LS1	Group	24142	1,621,773,636	263.5 GB