P110 Reprocessing
WORK IN PROGRESS
This page is a record of the configuration for the P110 reprocessing project, motivated by the Pass 7.2 event classification. This project involves reprocessing with Pass7 classification trees and (ultimately) new IRFs. The name "P110" derives from the word "processing" and the initial file version to be used for the output data products, e.g., r0123456789_v110_merit.root.
- P110-MERIT - this task reads DIGI+RECON+MERIT and produces reprocessed MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT
- P110-FITS (eventually) - this task will read FILTEREDMERIT and produce FT1 (photons) + LS1 (merit-like FITS file for photons) + electron FITS file
Datafile names, versions and locations
Data file version numbers for this reprocessing will begin with v110.
XROOT location and file naming
Location template:
/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>
Locations for P110:
/glast/Data/Flight/Reprocess/P110/merit /glast/Data/Flight/Reprocess/P110/filteredmerit /glast/Data/Flight/Reprocess/P110/electronmerit /glast/Data/Flight/Reprocess/P110/ft1 /glast/Data/Flight/Reprocess/P110/electronft1 /glast/Data/Flight/Reprocess/P110/ls1 /glast/Data/Flight/Reprocess/P110/<prodORdev>/<taskVersion>/ft2 /glast/Data/Flight/Reprocess/P110/<prodORdev>/<taskVersion>/ft2Seconds
File naming:
Data Type |
Send to FSSC |
Naming template |
---|---|---|
MERIT |
No |
r<run#>_<version>_<dataType>.root |
FILTEREDMERIT |
No |
r<run#>_<version>_<dataType>.root |
ELECTRONMERIT |
No |
r<run#>_<version>_<dataType>.root |
ELECTRONFIT |
No |
r<run#>_<version>_<dataType>.fit |
FT2SECONDS |
No |
r<run#><version><dataType>.fit |
FT1 |
Yes |
gll_ph_r<run#>_<version>.fit |
LS1 |
Yes |
gll_ev_r<run#>_<version>.fit |
FT2 |
Yes |
gll_pt_r<run#>_<version>.fit |
Example:
/glast/Data/Flight/Reprocess/P100/merit/r0239557414_v110_merit.root /glast/Data/Flight/Reprocess/P100/filteredmerit/r0239557414_v110_filteredmerit.root /glast/Data/Flight/Reprocess/P100/electronmerit/r0239557414_v110_electronmerit.root /glast/Data/Flight/Reprocess/P100/ft1/gll_ph_r0239559565_v110.fit /glast/Data/Flight/Reprocess/P100/electronfit/r0239557414_v110_electronft1.fit /glast/Data/Flight/Reprocess/P100/ls1/gll_ev_r0239559565_v110.fit /glast/Data/Flight/Reprocess/P100/dev/1.73/ft2/gll_pt_r0254754872_v110.fit /glast/Data/Flight/Reprocess/P100/dev/1.73/ft2Seconds/r0254754872_v110_ft2Seconds.fit
DataCatalog location and naming
Logical directory and group template:
Data/Flight/Reprocess/<reprocessName>:<dataType>
Note that the <dataType> field (following the colon) is a DataCatalog 'group' name.
Logical directories for P110:
Data/Flight/Reprocess/P110:MERIT Data/Flight/Reprocess/P110:FILTEREDMERIT Data/Flight/Reprocess/P110:ELECTRONMERIT Data/Flight/Reprocess/P110:FT1 Data/Flight/Reprocess/P110:ELECTRONFT1 Data/Flight/Reprocess/P110:LS1 Data/Flight/Reprocess/P110:FT2 Data/Flight/Reprocess/P110:FT2SECONDS
In the DataCatalog, all file names are of the form r<run#>.
Naming examples:
Data/Flight/Reprocess/P110:MERIT r0239557414 Data/Flight/Reprocess/P110:FILTEREDMERIT r0239557414 Data/Flight/Reprocess/P110:FT1 r0239557414 Data/Flight/Reprocess/P110:LS1 r0239557414 Data/Flight/Reprocess/P110:FT2 r0239557414 Data/Flight/Reprocess/P110:FT2SECONDS r0239557414
Data Sample
The currently defined data sample under consideration consists of the following period.
First run |
239557414 |
2008-08-04 15:43:34 UT |
Last run |
277596392 |
2009-10-18 22:06:32 UT |
Total runs |
6581 |
|
Total events |
14112958893 |
|
Bookkeeping
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P110
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
P110-MERIT
Status chronology
- 20 Oct 2009 - 650 early runs reprocessed (about 6 weeks, including two significant GRBs) with P110-MERIT
MET(sec)
UTC
first run
239557414
2008-08-04 15:43:34
last run
243289793
2008-09-16 20:29:53
- 17 Oct 2009 - Single run reprocessed for validation
Configuration
[Current production version of task]
Task Location |
/nfs/farm/g/glast/u38/Reprocess-tasks/P110-MERIT |
|
Task Status |
http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp | |
GlastRelease |
v17r31p1 |
|
Input Data Selection |
"standard" from https://confluence.slac.stanford.edu/display/SCIGRPS/LAT+Dataset+Definitionsalong with "&& (RunQuality != "Bad" |
is_null ( RunQuality )" |
---|---|---|
Input Run List |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P110-MERIT/config/runFile.txt | |
photonFilter |
evtClassDefs v0r6p1 CTBParticleType==0 && CTBClassLevel>0 |
|
electronFilter |
CTBParticleType==1 |
|
jobOpts |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P110-MERIT/config/reClassify.txt | |
Output Data Products |
Timing
P110-MERIT
The 650 runs in the six-week sample completed in about 20 hours elapsed time. Each run produces, on average, 7.5 1-hour "processClumps" jobs. Hence, the total CPU time to reprocess 650 runs is about 650 x 7.5 x 1 CPU-hour (fell-class machine) = 4875 CPU hours or 203 CPU-days.
The entire dataset (through 18 October 2009) consists of 6581 runs, which would be 49k CPU-hours or 2056 CPU-days. With 500 cores, this could take (with no operational problems) as little as 4.1 days.