status: In Progress
last update: 21 Feb 2014

*** PAGE IN PROGRESS: Not yet ready for public consumption ***

"New generation" tasks (using SCons builds, rewritten task scripts, common python scripts, etc.)

• P203-FITS - This task reads MERIT and produces FT1 (photons) + EXTENDEDFT1

This task is identical with P202-FITS with the following exceptions:

1. The dataset reprocessed is extended to include all current (Level 1) data since the end of the P202 task.
2. New diffuse response
3. Only FT1 and EXTENDEDFT1 data products are produced
4. File naming and other bookkeeping uses "P203" rather than "P202", where appropriate

Bookkeeping

1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P203
   1. List of all reprocessings
   2. List of all data runs reprocessed
   3. Pointers to all input data files (-> dataCatalog)
   4. Pointers to associated task processes (-> Pipeline II status)
3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
   1. Lists of and pointers to all output data files
   2. Meta data associated with each output data product

P203-FITS

This task generates FT1 and EXTENDEDFT1 data products.

Status chronology

• 2/20/2014 - Initial set-up of task
• 2/25/2014 - Begin trickleStream on Block 1 – entire P202 run list
• 2/28/2014 - Many problems: LSF seems to bunch up a large number of jobs before dispatching.  This causes a shock wave of jobs causing a problem for AFS (reading the ~500 MB diffuse response file), and for the dataCatalog (the queries for FT2 file, and to determine the next file version for each data product).  Therefore, reconfigure trickleStream to a mere dribble – no more than 300-400 jobs typically running simultaneously.  This means a 1-day task has become a week-long effort.
• 3/3/2014 - Block 1 complete.  Last week developed new scheme to move all dataCatalog queries from batch jobs into jython scriptlets.  These changes will be integrated prior to processing any more data.  Validity check: There are the same number of runs and the same number of events in the reprocessed P203 data as in the P202 data.

Configuration

 Identical with P202-FITS except:

Generation of output data products:

Note that diffuse response is calculated for 'source' and 'clean' event classes only.

Timing