P115 Reprocessing

status: Under Construction Running
last update: 8 July 2010

This page is a record of the configuration for the P115 reprocessing project, motivated by the Pass 7.3 event classification. This project involves reprocessing with Pass7 classification trees and (ultimately) new IRFs. The name "P115" derives from the word "processing" and the initial file version to be used for the output data products, e.g., r0123456789_v115_merit.root.

• P115-LEO-MERIT - like P110-LEO-MERIT but reprocess Reclassify data from the P106 (full reprocessing of earth limb data from L&EO period) rather than Level 1 (See P106 page for details), producing MERIT, FILTEREDMERIT and ELECTRONMERIT

• P115-LEO-FT1 - From P115-LEO-MERIT, generate FT1, LS1, LS3, ELECTRONFT1 products, including the calculation of the diffuse response.

Datafile names, versions and locations

...

Note that the L&EO data represent a discontiguous set of runs.

Bookkeeping

1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P115-LEO
   1. List of all reprocessings
   2. List of all data runs reprocessed
   3. Pointers to all input data files (-> dataCatalog)
   4. Pointers to associated task processes (-> Pipeline II status)
3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
   1. Lists of and pointers to all output data files
   2. Meta data associated with each output data product

...

• 8 Jul 2010 - Construct task
• 9 Jul 2010 - Task complete

Configuration

Task Location	/nfs/farm/g/glast/u38/Reprocess-tasks/P115-LEO-MERIT
Task Status	http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp
GlastRelease	v17r35p1gr02
Skimmer	07-07-00
Input Data Selection	Set of 200 runs selected by Anders Borgland
Input Run List	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P115-LEO-MERIT/config/runFile.txt
photonFilter	evtClassDefs v0r6p1 CTBParticleType==0 && CTBClassLevel>0
electronFilter	CTBParticleType==1
jobOpts	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P115-LEO-MERIT/config/reClassify.txt
Output Data Products	MERIT, FILTEREDMERIT, ELECTRONMERIT

Timing

...

and Scaling

The processClump job step averaged 20-30 hequ-min or 70-80 fell-min.
The mergeClumps job step averaged 5-15 hequ-min or 8-20 fell-min.

It was observed that xroot became very stressed when the number of processClump jobs reached ~1000. The stress increased when mergeClumps jobs became part of the mix. Pipeline throttle needed.

P115-LEO

Not yet

...

-FT1

Status chronology

...

• 9 Jul 2010 - Assemble task

Configuration

NOT YET UPDATED FOR P115

Task Location	/nfs/farm/g/glast/u38/Reprocess-tasks/P110P115-LEO-FT1
Task Status	http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp
Input Data Selection	MERIT (from P110P106-LEO-MERIT), FT2 (from P100P110-FT2 and Level1)
Input Run List	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P110P115-LEO-FT1/config/runFile.txtImage Modified
evtClassDefs	00-16-00
meritFilter	pass7_FSW_cuts, (FswGamState==0||FswGamState==3) && (CTBCORE>0) && (CTBBestEnergyProb>0) && (CTBBestEnergy>10) && (CTBBestEnergyRatio<5) && (CTBClassLevel>0)
eventClassifier	Pass7_Classifier.py
ScienceTools	09-15-05 (SCons build)
Code Variant	forced to redhat4-i686-32bit-gcc34 or redhat5-i686-32bit-gcc41
Diffuse Model	/afs/slac.stanford.edu/g/glast/ground/releases/analysisFiles/diffuse/v2/source_model_v02.xml ( https://confluence.slac.stanford.edu/display/SCIGRPS/Diffuse+Model+for+Analysis+of+LAT+Data )
Diffuse Response IRFs	P7_v2_diff, P7_v2_extrad, P7_v2_datac
IRFs	implemented as 'custom irf', files in /afs/slac.stanford.edu/g/glast/ground/PipelineConfig/IRFS/Pass7.2
Output Data Products	FT1, LS1, LS3, ELECTRONFT1

...

Note on diffuse response calculation: gtdiffrsp is called three times in succession. The first time with IRF P7_v2_diff and evclsmin==8, followed by IRF P7_v2_extrad and evclsmin==9, and finally IRF P7_v2_datac and evclsmin==10. The resulting FT1 file has six columns of diffuse response, two columns (galactic and extragalactic response) for each of the three IRFs. This creates a non-standard FT1 file by FSSC standards as they expect only five diffuse response columns.

Timing

Not yet