...
P118 Reprocessing
status: In preparation Complete
last update: 15 Feb 25 May 2011
This page is a record of the configuration for the P1xx P118 reprocessing project, recalculating diffuse response with new P6_V11 IRFs and adding to existing FT1 files. This processing will be applied to all science data, be incorporated into Level 1 processing, and results sent to the FSSC.
Datafile names, versions and locations
Data file version numbers for this reprocessing will begin with v116v118.
XROOT location and file naming
Location template:
Code Block |
---|
/glast/Data/Flight/Reprocess/<reprocessName>/<dataType> |
Locations for P1xxP118:
Code Block |
---|
/glast/Data/Flight/Reprocess/P1xxP118/ft1 |
File naming:
Data Type | aka | Send to FSSC | Naming template | ||||
---|---|---|---|---|---|---|---|
FT1 | LS-002 | Yes | gll_ph_p<procVer>_r<run#>_<version>.fit | LS1 | LS-001 | Yes | gll_ev_p<procVer>_r<run#>_<version>.fit |
Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD (.doc, or PDF) on 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_ICD_RevA.pdf
Example:
Code Block |
---|
/glast/Data/Flight/Reprocess/P116P118/ft1/gll_ph_p116p118_r0239559565_v116v118.fit /glast/Data/Flight/Reprocess/P116P118/ls1/gll_ev_p116p118_r0239559565_v116v118.fit |
DataCatalog location and naming
Logical directory and group template:
...
Code Block |
---|
Data/Flight/Reprocess/P116P118:FT1 r0239557414 Data/Flight/Reprocess/P116P118:LS1 r0239557414 |
Data Sample
The data sample for P116 P118 reprocessing includes the following runsthe current definition of the "Public_v2" dataset.
Block | First run | First run | Last run | Last run | #Runs | #Merit Evts#Evts | DataCatalog Source | Note | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
1 | 239557414 | 2008-08-04 15:43:34 | 242047683311669624 | 2008 2010-0911-02 1117 06:2853:02 42 | 43112557 | 909 435,050741,672248 | /Data/TestFlight/FlightReprocess/ReproP116/ReproTest7FT1 | entire P116 datasetreprocessed for alignment fix | |||||||||
1 | 242053458311675447 | 20082010-0911-02 1317 08:0430:1745 | 307579060 319652280 | 20102011-0902-30 2217 16:3717:3858 | 114131410 | 2455,906911,606,023339 | /Data/Flight/Level1/LPA/ | Standard Level 1 output (ignore StdIntent) | 1 Subtotal |
|
|
|
| 11844 | 25,815,656,695 |
|
|
2 | 307585048 319658687 | 2010 2011-1002-01 0017 18:1704:26 45 | 311108362 324849509 | 2010 2011-1104-10 18 19:5958:2027 | 617 908 | 145,368004,042,493526 | /Data/Flight/Level1/LPA/ | Standard Level 1 output | |||||||||
3 | 311112219 324855475 | 2010 2011-1104-10 2018 21:0337:3753 | 311669624 327999656 | 2010 2011-1105-17 0625 07:5300:4254 | 98 552 | 215 22,382469,591614 | /Data/Flight/Level1/LPA/ | Standard Level 1 output changed to correct gtdiffrsp parameters | |||||||||
TOTAL |
|
|
|
| 15427 | 559,126,727 | Grand Total | 239557414 | 2008-08-04 15:43:33 | 311669624 | 2010-11-17 06:53:42 | 12559 | 27,399,081,779 |
|
|
Final number of selected photon events (in the FT1 and LS1 files) = 435,801,176These data are destined for the FSSC and were made public as of 17 November 2010TBD
These data (along with subsequent Level 1 data) are will be in the SLAC Astroserver as P6_public_v2.
Three extra runs were unintentionally processed as part of P116:
Run | Task Stream | Trigger | LAT configuration flag | Disposition |
---|---|---|---|---|
242429468 | 499 | nomSciOps_trigTest | LAT_CONFIG = 1 | retained |
250687192 | 1948 | hldVetoCalib_Hi | LAT_CONFIG = 0 | removed |
250692922 | 1949 | hldVetoCalib_Lo | LAT_CONFIG = 0 | removed |
The first of these was deemed good for science, while the last two were not. Therefore, runs 250687192 and 250692922 have been retroactively removed from the datacatlog, astroserver and FSSCv3.
...
Bookkeeping
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P116
P118
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
...
P118-FT1
Anchor |
---|
...
|
...
|
(This task is roughly , in configuration, equivalent to the P105 P116-FT1 task but with updates and modernization, bypassing all but the gtdiffrsp step.)
Status chronology
...
5/
...
25/
...
2011 - Begin and complete reprocessing block3, 552 runs (324855475 through 327999656)
start 09:30, complete 11:40
4/20/2011 - Begin and complete reprocessing block2, 908 runs (319658690 through 324849509)
start 11:30, complete 14:00
3/24/2011 - Eight runs re-reprocessed due to updated FT1 files from Level 1:
Code Block |
---|
316703389
316739248
316744661
316761944
316767672
316773400
316776949
318795366
|
2/23/2011 - Block 1 complete
2/22/2011 - Begin trickleStream of block1
2/15/2011 - Begin configuration for this task
...
Configuration
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P116P118-FT1 |
Task Status | P116P118-FT1 |
Input Data Selection | MERITFT1 |
Input Run List | http://glast-ground.slac.stanford.edu/Decorator/Decorate/u38/Reprocess-tasks/P116P118-FT1/config/runFile.txt![]() |
Reprocessing Mode | reFT1 doDiffRsp |
evtClassDefs 00-18-00 | n/a (see P116) |
photon cut | pass6_FSW_cuts |
eventClassifier | dataclean_classifier.py |
n/a (see P116) | |
eventClassifiereventClassMap | n/a (see P116) |
s/c data | FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production |
ScienceTools | 09-1821-05 00 (SCons build) |
Allowed Code Variants | redhat4-i686-32bit-gcc34, redhat4-x86_64-64bit-gcc34, |
Diffuse Model | /afs/slac.stanford.edu/g/glast/ground/releases/analysisFilesgroups/diffuse/v2mapcubes/source_model_v02_P6_V11_DIFFUSE.xml ![]() ) |
Diffuse Response IRFs | P6_V3V11_DIFFUSE , P6_V3_DATACLEAN |
Output Data Products |
Processing chain for FITS data products
Data Product | makeFT1 | gtdiffrsp | gtmktime | gtltcube | ||||
---|---|---|---|---|---|---|---|---|
FT1 | truefalse | true for | true | false | LS1 | true | falsetrue | false |
Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RedHat Enterprise Linux 4, and 5, gcc 3.4, 4.1, etc.). Currently, all batch machines have 64-bit hardware and (most, if not all) run with the 64-bit operating systems.
Note on diffuse response calculation: gtdiffrsp is called two times in succession. For the bulk of this processing, the incorrect parameter was used for gtdiffrsp (IRF P6_V3_DIFFUSE and evclass==3, and IRF P6_V3_dataclean and evclass==4). After block2 this was corrected to IRF P6_V3_DIFFUSE and evclsmin==3, and IRF P6_V3_DATACLEAN and evclsmin==4. The resulting FT1 files have five columns for diffuse response of which two pairs are filled in for DATACLEAN, one pair for DIFFUSE and zero for all other events. Each pair contains galactic and extragalactic response.
...
Timing
The execution of gtdiffrsp happens in the mergeClumps step and takes ~5 hequ-minutes per run (on average) to complete. Some individual runs (few percent) take considerably longer...up to 2 hours.
(update 5/25/2011) Statistics for mergeClump now show mean time of 14.4 CPU minutes (averaged across all CPU types, but mostly hequ and boer)