Versions Compared

Old Version 22

changes.mady.by.user Tom Glanzman

Saved on

compared with

New Version Current

changes.mady.by.user Tom Glanzman

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

P118 Reprocessing

status: In progress Complete
last update: 20 Apr 25 May 2011

This page is a record of the configuration for the P118 reprocessing project, recalculating diffuse response with new P6_V11 IRFs and adding to existing FT1 files. This processing will be applied to all science data, be incorporated into Level 1 processing, and results sent to the FSSC.

...

Block

First run
MET

First run
(UTC)

Last run
MET

Last run
(UTC)

#Runs

#Evts

DataCatalog Source

Note

1

239557414

2008-08-04 15:43:34

311669624

2010-11-17 06:53:42

12557

435,741,248

/Data/Flight/Reprocess/P116/FT1

entire P116 dataset

1

311675447

2010-11-17 08:30:45

319652280

2011-02-17 16:17:58

1410

55,911,339

/Data/Flight/Level1/LPA/

Standard Level 1 output

2

319658687

2011-02-17 18:04:45

324849509

2011-04-18 19:58:27

908

45,004,526

/Data/Flight/Level1/LPA/

Standard Level 1 output

3

 

324855475

2011-04-18 21:37:53  

327999656

2011-05-25 07:00:54

 

552

22,469,614 

/Data/Flight/Level1/LPA/

Standard Level 1 output

TOTAL

 

 

 

 

15427

559,126,727

 

 

Final number of selected photon events (in the FT1 and LS1 files) = TBD

...

  1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
  2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P118Image Removed
    1. List of all reprocessings
    2. List of all data runs reprocessed
    3. Pointers to all input data files (-> dataCatalog)
    4. Pointers to associated task processes (-> Pipeline II status)
  3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jspImage Removed
    1. Lists of and pointers to all output data files
    2. Meta data associated with each output data product

...

(This task is, in configuration, equivalent to the P116-FT1 task, bypassing all but the gtdiffrsp step.)

Status chronology

5/25/2011 - Begin and complete reprocessing block3, 552 runs (324855475 through 327999656)
start 09:30, complete 11:40
4/20/2011 - Begin and complete reprocessing block2, 908 runs (319658690 through 324849509)
start 11:30, complete 14:00
3/24/2011 - Eight runs re-reprocessed due to updated FT1 files from Level 1:

...

Task Location

/nfs/farm/g/glast/u38/Reprocess-tasks/P118-FT1

Task Status

P118-FT1

Input Data Selection

FT1

Input Run List

http://glast-ground.slac.stanford.edu/Decorator/Decorate/u38/Reprocess-tasks/P118-FT1/config/runFile.txtImage Removed

Reprocessing Mode

doDiffRsp

evtClassDefs

n/a (see P116)

photon cut

n/a (see P116)

eventClassifier

n/a (see P116)

s/c data

FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production

ScienceTools

09-21-00 (SCons build)

Allowed Code Variants

redhat4-i686-32bit-gcc34, redhat4-x86_64-64bit-gcc34,
redhat5-i686-32bit-gcc41, redhat5-x86_64-64bit-gcc41 (Optimized)

Diffuse Model

/afs/slac/g/glast/groups/diffuse/mapcubes/source_model_v02_P6_V11_DIFFUSE.xml
(

https://confluence.slac.stanford.edu/display/SCIGRPS/Diffuse+Model+for+Analysis+of+LAT+DataImage Removed

)

Diffuse Response IRFs

P6_V11_DIFFUSE

Output Data Products

FT1

...

The execution of gtdiffrsp happens in the mergeClumps step and takes ~5 hequ-minutes per run (on average) to complete. Some individual runs (few percent) take considerably longer...up to 2 hours.
(update 5/25/2011) Statistics for mergeClump now show mean time of 14.4 CPU minutes (averaged across all CPU types, but mostly hequ and boer)