P130 Reprocessing

status: In Development Complete
last update: 25 January 6 Aug 2012

This page is a record of the configuration for the P130 reprocessing project, recalculating the diffuse response (see here for a description of the change). This task is identical with P120-FT1, v2.1 [please see for details], run in July 2011, except for updating the following details.

1. New version of ScienceTools

...

=============== WHAT FOLLOWS IS TO BE UPDATED ====================

The name "P120" derives from the word "processing" and the initial file version to be used for the output data products, e.g., r0123456789_v120_merit.root.

• P120-MERIT - this task reads DIGI+RECON+MERIT and produces reprocessed MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT

• P120-FT1
  • V1.0 of this task reads MERIT and produces FT1 (photons) + LS1 (merit-like FITS file for photons) + electron FITS file
  • V2.1 of this task is modified such that FT1 and LS1 files are filtered for FSSC, while new EXTENDEDFT1 and EXTENDEDLS1 files are produced containing all photon event classes.

• P120-LEO-MERIT - this task reads DIGI+RECON+MERIT and produces reprocessed MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT for 200 runs of earth limb (L&EO) data

Datafile names, versions and locations

Data file version numbers for this reprocessing will begin with v120.

XROOT location and file naming

Location template:

/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>

Locations for P120:

/glast/Data/Flight/Reprocess/P120/merit
/glast/Data/Flight/Reprocess/P120/filteredmerit
/glast/Data/Flight/Reprocess/P120/electronmerit
/glast/Data/Flight/Reprocess/P120/ft1
/glast/Data/Flight/Reprocess/P120/extendedft1
/glast/Data/Flight/Reprocess/P120/electronft1
/glast/Data/Flight/Reprocess/P120/ls1
/glast/Data/Flight/Reprocess/P120/extendedls1

File naming:

Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf^{Image Removed}

Example:

/glast/Data/Flight/Reprocess/P120/merit/r0239557414_v120_merit.root
/glast/Data/Flight/Reprocess/P120/filteredmerit/r0239557414_v120_filteredmerit.root
/glast/Data/Flight/Reprocess/P120/electronmerit/r0239557414_v120_electronmerit.root
/glast/Data/Flight/Reprocess/P120/extendedft1/gll_xp_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/ft1/gll_ph_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/electronft1/gll_el_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/extendedls1/gll_xe_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/ls1/gll_ev_p120_r0239559565_v120.fit

DataCatalog location and naming

Logical directory and group template:

Data/Flight/Reprocess/<reprocessName>:<dataType>

Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.

Naming examples:

Data/Flight/Reprocess/P120:MERIT r0239557414
Data/Flight/Reprocess/P120:FILTEREDMERIT r0239557414
Data/Flight/Reprocess/P120:EXTENDEDFT1 r0239557414
Data/Flight/Reprocess/P120:FT1 r0239557414
Data/Flight/Reprocess/P120:ELECTRONFT1 r0239557414
Data/Flight/Reprocess/P120:EXTENDEDLS1 r0239557414
Data/Flight/Reprocess/P120:LS1 r0239557414

Data Sample

1. (09-26-02) which includes new integration scheme for gtdiffrsp
2. New run list (updated for data processed since the end of P120)
3. Name of Pipeline task (P120->P130) as well as in file naming

Data Sample

The data sample for P130 The currently defined data sample for P120 reprocessing includes:

Summary from DataCatalog as of 4/17/2012 and updated 8/2/20116/2012 (after new evt selection filter).

NOTE: One run, 242429468, of type TrigTest was declared 'good for science' but long after this task got started, so it has been intentionally omitted.

8/17/2011 update: Four orphaned runs are being reprocessed, including one TrigTest run and four nadir-pointed runs.

Bookkeeping

1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P120^{Image Removed}P130
   1. List of all reprocessings
   2. List of all data runs reprocessed
   3. Pointers to all input data files (-> dataCatalog)
   4. Pointers to associated task processes (-> Pipeline II status)
3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp^{Image Removed}
   1. Lists of and pointers to all output data files
   2. Meta data associated with each output data product

...

P130-FT1

Anchor

...

P130-FT1

...

P130-FT1

This task generates all desired FITS data products. An example of the code processing chain appears on a child page.

Status chronology

Configuration (version 2)

• 1/26/2012 - Initial task configured, 1000 runs reprocessed. On hold for validation.
• 2/1/2012 - Okay to continue. Begin block 1 reprocessing, containing 19133 runs:

• 2/3/2012 - Block 1 reprocessing complete
• 4/2/2012 - Begin block 2, raising the total # runs to 20126, through run 354926369 2012-03-31 22:39:29 (UTC)
• 4/18/2012 - Final reprocessing block: additional 259 runs through 356429251, 2012-04-18 08:07:29 (UTC)
• 4/19/2012 - Complete.
• 5/1/2012 - Reconfigure start time - reducing by 1.9 seconds - for run 245403855 due to old leap second problem - to regain 27 events at the start of the run. This is stream 1019.
• 8/3/2012 - A request to update the event selection filter on data in the period 1 Aug 2011 and 19 Apr 2012 was made to bring EXTENDED FITS and FILTEREDMERIT files into synch. See this page for details. The P130-FT1 task has been modified to run a new event filter; mergeClumps job steps for the following runs rolled back:

  	Run	Task Stream
  Start	333886531	16460
  End	356429251	20384

  This action will recreate only the EXTENDEDFT1 and EXTENDEDLS1 data products. These will automatically be ingested by the astroserver. Note that the file original-config.py contains the former configuration for this task, while config.py contains the changes discussed above.
• 8/6/2012 - Re-reprocessing complete

Configuration

Task Location	/nfs/farm/g/glast/u38/Reprocess-tasks/P120P130-FT1
Task Status	http://glast-ground.slac.stanford.edu/Pipeline-II/task.jsp?task=63390236Image Removed74765664
Input Data	MERIT ( from P120-MERIT (runs 239557414 - 333880535) and Level 1 (runs 333886531 - present)
spacecraft data	FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production
Input Run List	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P120P130-FT1/config/runFile.txtImage Removed
Reprocessing Mode	reFT1
meritFilter	FT1EventClass!=0 for P120-reprocessing, and (FT1EventClass& 0x00003EFF)!=0 for subsequent data
evtClassDefs	00-19-01
eventClassMap	EvtClassDefs_P7V6.xml
ScienceTools	09-2426-00 02
Code Variants	redhat5-i686-32bit-gcc41 (Optimized)
Diffuse Model	based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 (see https://confluence.slac.stanford.edu/display/SCIGRPS/Quick+Start+with+Pass+7Image Removed )
Diffuse Response	'source' using P7SOURCE_V6 IRF 'clean' using P7CLEAN_V6 IRF
IRFs	P6V7P7V6, contained within ScienceTools release
Output Data Products	FT1, LS1, EXTENDEDFT1, EXTENDEDLS1, ELECTRONFT1

...

Note that diffuse response is calculated for 'source' and 'clean' event classes only.

Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.).

Timing and Performance

With a sample of the first 8128 runs, the CPU-intensive part of the task (mergeClumps) is consuming ~42 CPU-min/job (differences in machine class are small). TrickleStream configured to allow 1500 simultaneous mergeClumps jobs, and up to 150 job starts every 200 second cycle.