P118 Reprocessing

status: Complete
last update: 25 May 2011

This page is a record of the configuration for the P118 reprocessing project, recalculating diffuse response with new P6_V11 IRFs and adding to existing FT1 files. This processing will be applied to all science data, be incorporated into Level 1 processing, and results sent to the FSSC.

Datafile names, versions and locations

Data file version numbers for this reprocessing will begin with v118.

XROOT location and file naming

Location template:

/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>

Locations for P118:

/glast/Data/Flight/Reprocess/P118/ft1

File naming:

Data Type

aka

Send to FSSC

Naming template

FT1

LS-002

Yes

gll_ph_p<procVer>_r<run#>_<version>.fit

Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD (.doc, or PDF) on 5/12/2010.

Example:

/glast/Data/Flight/Reprocess/P118/ft1/gll_ph_p118_r0239559565_v118.fit
/glast/Data/Flight/Reprocess/P118/ls1/gll_ev_p118_r0239559565_v118.fit
DataCatalog location and naming

Logical directory and group template:

Data/Flight/Reprocess/<reprocessName>:<dataType>

Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.

Naming examples:

Data/Flight/Reprocess/P118:FT1 r0239557414
Data/Flight/Reprocess/P118:LS1 r0239557414

Data Sample

The data sample for P118 reprocessing includes the current definition of the "Public_v2" dataset.

Block

First run
MET

First run (UTC)

Last run
MET

Last run (UTC)

#Runs

#Evts

DataCatalog Source

Note

1

239557414

2008-08-04 15:43:34

311669624

2010-11-17 06:53:42

12557

435,741,248

/Data/Flight/Reprocess/P116/FT1

entire P116 dataset

1

311675447

2010-11-17 08:30:45

319652280

2011-02-17 16:17:58

1410

55,911,339

/Data/Flight/Level1/LPA/

Standard Level 1 output

2

319658687

2011-02-17 18:04:45

324849509

2011-04-18 19:58:27

908

45,004,526

/Data/Flight/Level1/LPA/

Standard Level 1 output

3

324855475

2011-04-18 21:37:53

327999656

2011-05-25 07:00:54

552

22,469,614

/Data/Flight/Level1/LPA/

Standard Level 1 output

TOTAL

 

 

 

 

15427

559,126,727

 

 

Final number of selected photon events (in the FT1 and LS1 files) = TBD

These data (along with subsequent Level 1 data) will be in the SLAC Astroserver as P6_public_v3.



Bookkeeping

  1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
  2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P118
    1. List of all reprocessings
    2. List of all data runs reprocessed
    3. Pointers to all input data files (-> dataCatalog)
    4. Pointers to associated task processes (-> Pipeline II status)
  3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
    1. Lists of and pointers to all output data files
    2. Meta data associated with each output data product

P118-FT1

(This task is, in configuration, equivalent to the P116-FT1 task, bypassing all but the gtdiffrsp step.)

Status chronology

5/25/2011 - Begin and complete reprocessing block3, 552 runs (324855475 through 327999656)
start 09:30, complete 11:40
4/20/2011 - Begin and complete reprocessing block2, 908 runs (319658690 through 324849509)
start 11:30, complete 14:00
3/24/2011 - Eight runs re-reprocessed due to updated FT1 files from Level 1: 

316703389
316739248
316744661
316761944
316767672
316773400
316776949
318795366

2/23/2011 - Block 1 complete
2/22/2011 - Begin trickleStream of block1
2/15/2011 - Begin configuration for this task

Configuration

Task Location

/nfs/farm/g/glast/u38/Reprocess-tasks/P118-FT1

Task Status

P118-FT1

Input Data Selection

FT1

Input Run List

 http://glast-ground.slac.stanford.edu/Decorator/Decorate/u38/Reprocess-tasks/P118-FT1/config/runFile.txt

Reprocessing Mode

doDiffRsp

evtClassDefs

n/a (see P116)

photon cut

n/a (see P116)

eventClassifier

n/a (see P116)

s/c data

FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production

ScienceTools

09-21-00 (SCons build)

Allowed Code Variants

redhat4-i686-32bit-gcc34, redhat4-x86_64-64bit-gcc34,
redhat5-i686-32bit-gcc41, redhat5-x86_64-64bit-gcc41 (Optimized)

Diffuse Model

/afs/slac/g/glast/groups/diffuse/mapcubes/source_model_v02_P6_V11_DIFFUSE.xml
(

https://confluence.slac.stanford.edu/display/SCIGRPS/Diffuse+Model+for+Analysis+of+LAT+Data

)

Diffuse Response IRFs

P6_V11_DIFFUSE

Output Data Products

FT1

Processing chain for FITS data products

Data Product

makeFT1

gtdiffrsp

gtmktime

gtltcube

FT1

false

true for
evclsmin==3

false

false

Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RedHat Enterprise Linux 4, and 5, gcc 3.4, 4.1, etc.). Currently, all batch machines have 64-bit hardware and (most, if not all) run with the 64-bit operating systems.

Timing

The execution of gtdiffrsp happens in the mergeClumps step and takes ~5 hequ-minutes per run (on average) to complete. Some individual runs (few percent) take considerably longer...up to 2 hours.
(update 5/25/2011) Statistics for mergeClump now show mean time of 14.4 CPU minutes (averaged across all CPU types, but mostly hequ and boer)