P110 Reprocessing

WORK IN PROGRESS

This page is a record of the configuration for the P110 reprocessing project, motivated by the Pass 7.2 event classification. This project involves reprocessing with Pass7 classification trees and (ultimately) new IRFs. The name "P110" derives from the word "processing" and the initial file version to be used for the output data products, e.g., r0123456789_v110_merit.root.

• P110-MERIT - this task reads DIGI+RECON+MERIT and produces reprocessed MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT

• P110-FITS (eventually) - this task will read FILTEREDMERIT and produce FT1 (photons) + LS1 (merit-like FITS file for photons) + electron FITS file

Datafile names, versions and locations

Data file version numbers for this reprocessing will begin with v110.

XROOT location and file naming

Location template:

/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>

Locations for P110:

/glast/Data/Flight/Reprocess/P110/merit
/glast/Data/Flight/Reprocess/P110/filteredmerit
/glast/Data/Flight/Reprocess/P110/electronmerit
/glast/Data/Flight/Reprocess/P110/ft1
/glast/Data/Flight/Reprocess/P110/electronft1
/glast/Data/Flight/Reprocess/P110/ls1
/glast/Data/Flight/Reprocess/P110/<prodORdev>/<taskVersion>/ft2
/glast/Data/Flight/Reprocess/P110/<prodORdev>/<taskVersion>/ft2Seconds

File naming:

Example:

/glast/Data/Flight/Reprocess/P100/merit/r0239557414_v110_merit.root
/glast/Data/Flight/Reprocess/P100/filteredmerit/r0239557414_v110_filteredmerit.root
/glast/Data/Flight/Reprocess/P100/electronmerit/r0239557414_v110_electronmerit.root
/glast/Data/Flight/Reprocess/P100/ft1/gll_ph_r0239559565_v110.fit
/glast/Data/Flight/Reprocess/P100/electronfit/r0239557414_v110_electronft1.fit
/glast/Data/Flight/Reprocess/P100/ls1/gll_ev_r0239559565_v110.fit
/glast/Data/Flight/Reprocess/P100/dev/1.73/ft2/gll_pt_r0254754872_v110.fit
/glast/Data/Flight/Reprocess/P100/dev/1.73/ft2Seconds/r0254754872_v110_ft2Seconds.fit

DataCatalog location and naming

Logical directory and group template:

Data/Flight/Reprocess/<reprocessName>:<dataType>

Note that the <dataType> field (following the colon) is a DataCatalog 'group' name.

Logical directories for P110:

Data/Flight/Reprocess/P110:MERIT
Data/Flight/Reprocess/P110:FILTEREDMERIT
Data/Flight/Reprocess/P110:ELECTRONMERIT
Data/Flight/Reprocess/P110:FT1
Data/Flight/Reprocess/P110:ELECTRONFT1
Data/Flight/Reprocess/P110:LS1
Data/Flight/Reprocess/P110:FT2
Data/Flight/Reprocess/P110:FT2SECONDS

In the DataCatalog, all file names are of the form r<run#>.

Naming examples:

Data/Flight/Reprocess/P110:MERIT r0239557414
Data/Flight/Reprocess/P110:FILTEREDMERIT r0239557414
Data/Flight/Reprocess/P110:FT1 r0239557414
Data/Flight/Reprocess/P110:LS1 r0239557414
Data/Flight/Reprocess/P110:FT2 r0239557414
Data/Flight/Reprocess/P110:FT2SECONDS r0239557414

Data Sample

The currently defined data sample under consideration consists of the following period.

Bookkeeping

1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P110
   1. List of all reprocessings
   2. List of all data runs reprocessed
   3. Pointers to all input data files (-> dataCatalog)
   4. Pointers to associated task processes (-> Pipeline II status)
3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
   1. Lists of and pointers to all output data files
   2. Meta data associated with each output data product

P110-MERIT

Status chronology

• 20 Oct 2009 - 650 early runs reprocessed (about 6 weeks, including two significant GRBs) with P110-MERIT

  	MET(sec)	UTC
  first run	239557414	2008-08-04 15:43:34
  last run	243289793	2008-09-16 20:29:53

• 17 Oct 2009 - Single run reprocessed for validation

Configuration

[Current production version of task]

Task Location	/nfs/farm/g/glast/u38/Reprocess-tasks/P110-MERIT
Task Status	http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp
GlastRelease	v17r31p1
Input Data Selection	"standard" from https://confluence.slac.stanford.edu/display/SCIGRPS/LAT+Dataset+Definitions along with "&& (RunQuality != "Bad"	is_null ( RunQuality )"
Input Run List	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P110-MERIT/config/runFile.txt
photonFilter	evtClassDefs v0r6p1 CTBParticleType==0 && CTBClassLevel>0
electronFilter	CTBParticleType==1
jobOpts	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P110-MERIT/config/reClassify.txt
Output Data Products	MERIT, FILTEREDMERIT, ELECTRONMERIT

Timing

P110-MERIT

The 650 runs in the six-week sample completed in about 20 hours elapsed time. Each run produces, on average, 7.5 1-hour "processClumps" jobs. Hence, the total CPU time to reprocess 650 runs is about 650 x 7.5 x 1 CPU-hour (fell-class machine) = 4875 CPU hours or 203 CPU-days.

The entire dataset (through 18 October 2009) consists of 6581 runs, which would be 49k CPU-hours or 2056 CPU-days. With 500 cores, this could take (with no operational problems) as little as 4.1 days.