status: Running
last update: 01 Aug 2013

This page is a record of the configuration and execution of the P300 reprocessing project, full reprocessing from DIGIs using Pass8 analysis code.

Pipeline tasks

• P300-ROOT - This task reads DIGI and produces reprocessed RECON + GCR + MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT

• P300-FITS - This task (once it has been created) reads MERIT and produces FT1 (photons) + EXTENDEDFT1 + LS1 (merit-like FITS file for photons) + EXTENDEDLS1 + ELECTRONFITS file

• An experimental P300x-FITS task is being used for early validation studies. Data will not survive in the long run...

Datafile names, versions and locations

Data file version numbers for this reprocessing will begin with v300.

XROOT location and file naming

Location template:

/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>

Locations for P300:

/glast/Data/Flight/Reprocess/P300/recon
/glast/Data/Flight/Reprocess/P300/gcr
/glast/Data/Flight/Reprocess/P300/relation/glast/Data/Flight/Reprocess/P300/merit
/glast/Data/Flight/Reprocess/P300/filteredmerit
/glast/Data/Flight/Reprocess/P300/electronmerit
/glast/Data/Flight/Reprocess/P300/ft1
/glast/Data/Flight/Reprocess/P300/extendedft1
/glast/Data/Flight/Reprocess/P300/electronft1
/glast/Data/Flight/Reprocess/P300/ls1
/glast/Data/Flight/Reprocess/P300/extendedls1

File naming:

Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf

Example:

/glast/Data/Flight/Reprocess/P300/recon/r0239557414_v202_recon.root

DataCatalog location and naming

Logical directory and group template:

Data/Flight/Reprocess/<reprocessName>:<dataType>

Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.

Naming examples:

Data/Flight/Reprocess/P300:RECON r0239557414

Data Sample

The currently defined data sample (as of 7/2/2013) for P300 reprocessing includes:

Bookkeeping

1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P300
   1. List of all reprocessings
   2. List of all data runs reprocessed
   3. Pointers to all input data files (-> dataCatalog)
   4. Pointers to associated task processes (-> Pipeline II status)
3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
   1. Lists of and pointers to all output data files
   2. Meta data associated with each output data product

P300-ROOT

Status chronology

• 4/23/2013 - Begin setting up P300 task for Pass8 reprocessing.
• 5/10/2013 - Initial version of task running.  See this page for infomation about the task creation, and this page for performance comparison with P202
• 7/2/2013 - after weeks of testing, final GR, jobOpts, etc. come together and production tentatively starts.
  • block 1 is defined with this run range: 239557414 through 392842073 (2008-08-04 15:43:37 through 2013-06-13 18:47:50 UTC)
  • Update task to use the P202 generation of FT2 files (adds columns LAMBDA, RA_SUN, DEC_SUN, as well as meaningful BTIs)
  • First 10 trial jobs launched. Too many ACD INFO messages 'Missed Poca for ID 602 at...' so modify jobOpts to disable.
  • xroot disk space at the start of Pass8 = 508 TB (just after commissioning of fermi-xrd11)
• 7/29/2013 - a new GR is prepared (20-09-01) which fixes some problems in ACD tracking issues affecting ~32 runs so far (out of 5283). trickleStream disabled, wait for pending jobs to start, rollback other failed jobs, switch to new GR, rollback of ACD-failed jobs...
  • This new GR continues to seg fault, so revert back to GR 20-09-00 and restart trickleStream.
• 7/31/2013 - A new GR (20-09-02) with patch for ACD tracking installed after stream 5970. At the time, there were 33 failed processClumps jobs. All rolled back.
  • Last stream with GR 20-09-00 = 5970, run 274118943
  • First stream with GR 20-09-02 = 5971, run 274124672
  • Full list of rolled back substreams is here

Configuration

Timing and Scaling

• Current performance results for the Pass8 code are on this page .
• For 20,000 event clumps:

  job step	average CPU time
  processClump	183 min
  mergeClumps	10 min

• Plot of job throughput as of 20130813
  
  From the plot, one can see a processing rate of ~208 runs/day. Assuming 28500 total runs, this would mean a repro time of 137 days or 4.5 months.
• Another plot showing the number of run scratch directories cleaned up per day:
  

P300x-FITS

This task generates all desired FITS data products.

NOTE: a temporary task called P300x-FITS has been created. It generates only EXTENDEDFT1 files and nothing else. There is no diffuse calculation performed. An untagged version of evtClassDefs is used which contains Matthew Wood's initial event classification and selections. This will likely be a template for a future production P300-FITS task. All data produced by this task are "throw-away" and not expected to survive beyond initial validation studies.

Status chronology

• 8/1/2013 - Initial P300x-FITS task created. 10 trial runs processed. (See caveats above.)
• 8/7/2013 - First year of data run through the task, amounting to 5537 runs and 11790391606 events:

  First run	239557417	2008-08-04 15:43:37
  Last run	271850279	2009-08-13 09:57:59

• 8/28/2013 - Second year of data ready and processed -> extendedFT1. Note that 36 troublesome runs omitted from 1st year data were included in this sample. There should be no missing runs in this sample.

  First run	239557417	2008-04-2008 15:43:37
  Last run

  10,976 runs, 2,677,747,829 events, 234.7 GB

Configuration

CAUTION: The following data describes the experimental P300x-FITS task, not the production task

WARNING: THIS NEXT SECTION IS OBSOLETE

Generation of output data products:

[1] /afs/slac/g/glast/ground/releases/volume04/evtClassDefs/00-19-04/data

Note that diffuse response is calculated for 'source' and 'clean' event classes only.

Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL5-32 (RHEL6-64 are built but not yet validated), while ScienceTools builds for RHEL5-64, RHEL5-64.

Timing and Scaling