P130 Reprocessing

status: In Progress
last update: 3 February 2012

This page is a record of the configuration for the P130 reprocessing project, recalculating the diffuse response (see here for a description of the change). This task is identical with P120-FT1 [[please see] for details], run in July 2011, except for the following details.

1. New version of ScienceTools (09-26-92) which includes new integration scheme for gtdiffrsp
2. New run list (updated for data processed since the end of P120)
3. Name of Pipeline task (P120->P130) as well as in file naming

Data Sample

=============== THIS SECTION NEEDS TO BE UPDATED FOR P130 ====================

The currently defined data sample for P120 reprocessing includes:

|First run|239557414 (MET), 2008-08-04 15:43:34 (UTC)|beginning of Science |
|Last run| 333880535 (MET), 2011-08-01 08:35:33 (UTC) | Official Pass7 release|
|Total runs | 16,459| |
|Total MERIT events|35,921,666,747|all "events"|
|Total FILTEREDMERIT/EXTENDEDFT1/LS1 events| 5,035,929,409|all photon event classes|
|Total ELECTRONMERIT/ELECTRONFT1 events| 68,055,849 | |
|Total LS1 (FSSC selection) events |1,025,359,231 |event classes (bits) 0,2,3,4 (transient, source, clean, ultraclean)|
|Total FT1 (FSSC selection) events | 142,042,060|event classes (bits) 2,3,4 (source, clean, ultraclean)|
|Total disk space used| 33.9 TB| |

Summary from DataCatalog as of 8/2/2011.

||Name || Files || Events || Size ||
|ELECTRONFT1 | 16459 | 68,055,849 | 6.4 GB |
|ELECTRONMERIT | 16459 | 68,055,849 | 147.4 GB |
|EXTENDEDFT1 | 16459 | 5,035,929,409 | 441.3 GB |
|EXTENDEDLS1 | 16459 | 5,035,929,409 | 816.5 GB |
|FILTEREDMERIT| 16459 | 5,035,929,409 | 4.0 TB |
|FT1 | 16459 |142,042,060 | 12.9 GB |
|LS1 | 16459 |1,025,359,231 | 166.6 GB |
|MERIT| 16459 |35,921,666,747 | 28.3 TB |

Bookkeeping

1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P130
   1. List of all reprocessings
   2. List of all data runs reprocessed
   3. Pointers to all input data files (-> dataCatalog)
   4. Pointers to associated task processes (-> Pipeline II status)
3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
   1. Lists of and pointers to all output data files
   2. Meta data associated with each output data product

P130-FT1

This task generates all desired FITS data products. An example of the code processing chain appears on a child page.

Status chronology

• 1/26/2012 - Initial task configured, 1000 runs reprocessed. On hold for validation.
• 2/1/2012 - Okay to continue. Begin block 1 reprocessing, containing 19133 runs:

• 2/3/2012 - Block 1 reprocessing complete

Configuration

Task Location	/nfs/farm/g/glast/u38/Reprocess-tasks/P130-FT1
Task Status	http://glast-ground.slac.stanford.edu/Pipeline-II/task.jsp?task=74765664
Input Data	MERIT (from P120-MERIT and Level 1)
spacecraft data	FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production
Input Run List	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P130-FT1/config/runFile.txt
Reprocessing Mode	reFT1
meritFilter	FT1EventClass!=0
evtClassDefs	00-19-01
eventClassMap	EvtClassDefs_P7V6.xml
ScienceTools	09-26-02
Code Variants	redhat5-i686-32bit-gcc41 (Optimized)
Diffuse Model	based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 (see https://confluence.slac.stanford.edu/display/SCIGRPS/Quick+Start+with+Pass+7 )
Diffuse Response	'source' using P7SOURCE_V6 IRF 'clean' using P7CLEAN_V6 IRF
IRFs	P6V7, contained within ScienceTools release
Output Data Products	FT1, LS1, EXTENDEDFT1, EXTENDEDLS1, ELECTRONFT1

Processing chain for FITS data products

Note that diffuse response is calculated for 'source' and 'clean' event classes only.

Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.).

Timing and Performance

With a sample of the first 8128 runs, the CPU-intensive part of the task (mergeClumps) is consuming ~42 CPU-min/job (differences in machine class are small). TrickleStream configured to allow 1500 simultaneous mergeClumps jobs, and up to 150 job starts every 200 second cycle.