status: In Development
last update: 10 May 2013
This page is a record of the configuration and execution of the P300 reprocessing project, full reprocessing from DIGIs using Pass8 analysis code.
Pipeline tasks
- P300-ROOT - This task reads DIGI and produces reprocessed RECON + GCR + MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT
- P300-FITS - This task reads MERIT and produces FT1 (photons) + EXTENDEDFT1 + LS1 (merit-like FITS file for photons) + EXTENDEDLS1 + ELECTRONFITS file
Datafile names, versions and locations
Data file version numbers for this reprocessing will begin with v300.
XROOT location and file naming
Location template:
/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>
Locations for P300:
/glast/Data/Flight/Reprocess/P300/recon /glast/Data/Flight/Reprocess/P300/gcr /glast/Data/Flight/Reprocess/P300/relation/glast/Data/Flight/Reprocess/P300/merit /glast/Data/Flight/Reprocess/P300/filteredmerit /glast/Data/Flight/Reprocess/P300/electronmerit /glast/Data/Flight/Reprocess/P300/ft1 /glast/Data/Flight/Reprocess/P300/extendedft1 /glast/Data/Flight/Reprocess/P300/electronft1 /glast/Data/Flight/Reprocess/P300/ls1 /glast/Data/Flight/Reprocess/P300/extendedls1
File naming:
Data Type |
aka |
Send to FSSC |
Naming template |
---|---|---|---|
RECON |
|
No |
r<run#>_<version>_<dataType>.root |
RELATION |
|
No |
r<run#>_<version>_<dataType>.root |
GCR |
|
No |
r<run#>_<version>_<dataType>.root |
MERIT |
|
No |
r<run#>_<version>_<dataType>.root |
FILTEREDMERIT |
|
No |
r<run#>_<version>_<dataType>.root |
ELECTRONMERIT |
|
No |
r<run#>_<version>_<dataType>.root |
ELECTRONFT1 |
|
No |
gll_el_p<procVer>_r<run#>_<version>.fit |
EXTENDEDFT1 |
|
No |
gll_xp_p<procVer>_r<run#>_<version>.fit |
FT1 |
LS-002 |
Yes |
gll_ph_p<procVer>_r<run#>_<version>.fit |
EXTENDEDLS1 |
|
No |
gll_xe_p<procVer>_r<run#>_<version>.fit |
LS1 |
LS-001 |
Yes |
gll_ev_p<procVer>_r<run#>_<version>.fit |
Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf
Example:
/glast/Data/Flight/Reprocess/P300/recon/r0239557414_v202_recon.root
DataCatalog location and naming
Logical directory and group template:
Data/Flight/Reprocess/<reprocessName>:<dataType>
Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.
Naming examples:
Data/Flight/Reprocess/P300:RECON r0239557414
Data Sample
The currently defined data sample (as of XXXXXX) for P300 reprocessing includes:
First run |
239557414 (MET), 2008-08-04 15:43:34 (UTC) |
Last run |
354923690 (MET), 2012-03-31 21:54:48 (UTC) |
Bookkeeping
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P300
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
P300-ROOT
Status chronology
- 4/23/2013 - Begin setting up P300 task for Pass8 reprocessing.
- 5/10/2013 - Initial version of task running. See this page for infomation about the task creation, and this page for performance comparison with P202
Configuration
NOT YET UPDATED
Task Location |
/nfs/farm/g/glast/u38/Reprocess-tasks/P300-ROOT |
Task Status |
|
GlastRelease |
20-08-12 (SCons RHEL5-32 build) |
Run Selection |
based on a modified "standard" selection, see https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
s/c data |
"standard" Public Release 2 https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
Input Run List |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300-ROOT/config/runList.txt |
photonFilter |
CTBParticleType==1 && ((FT1EventClass & 0x00003EFF)!=0) |
electronFilter |
CTBParticleType==1 |
Code Variants |
redhat5-i686-32bit-gcc41 (Optimized) |
jobOpts |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300-ROOT/config/Pass8Recon.txt |
Output Data Products |
Timing and Scaling
- Current performance results for the Pass8 code are on this page .
- No information about scaling limits on the batch farm is currently known.
P300-FITS
This task generates all desired FITS data products. This task does not yet exist.
Status chronology
Configuration
NOT YET UPDATED
Task Location |
/nfs/farm/g/glast/u38/Reprocess-tasks/P300-FITS |
Task Status |
http://glast-ground.slac.stanford.edu/Pipeline-II/task.jsp?task=75031156 |
Input Data |
MERIT (direct from P300-ROOT) |
spacecraft data |
same as P300-ROOT |
Input Run List |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-FITS/config/runList.txt |
evtClassDefs |
00-19-04 (6/23/2011) |
eventClassMap |
EvtClassDefs_P7V6.xml |
ScienceTools |
09-27-01 (2/15/2012) 09-31-01 (3/5/2013) |
Code Variants |
redhat5-i686-32bit-gcc41 (Optimized) |
Diffuse Model |
based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 /v3r0 |
Diffuse Response |
'source' using P7SOURCE_V6 P7REP_SOURCE_V10 IRF |
IRFs |
P7REP_*_V10, contained within ScienceTools release |
Output Data Products |
Generation of output data products:
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="bbd0bad7-afbd-4570-b598-159c44b85194"><ac:plain-text-body><![CDATA[ |
Data Product |
destination |
data content [1] |
event selection [1] |
makeFT1 |
gtselect |
gtdiffrsp |
gtmktime |
]]></ac:plain-text-body></ac:structured-macro> |
---|---|---|---|---|---|---|---|---|---|
EXTENDEDFT1 |
SLAC |
FT1variables |
((FT1EventClass & 0x00003EFF)!=0) |
|
|
|
|
||
FT1 |
FSSC+SLAC |
FT1variables |
'source' and above |
|
|
(inherited) |
|
||
EXTENDEDLS1 |
SLAC |
LS1variables |
((FT1EventClass & 0x00003EFF)!=0) |
|
|
|
|
||
LS1 |
FSSC+SLAC |
LS1variables |
'transient' and above |
|
|
(inherited) |
|
||
ELECTRONFT1 |
SLAC |
FT1variables |
CTBParticleType==1 |
|
|
|
|
[1] /afs/slac/g/glast/ground/releases/volume04/evtClassDefs/00-19-04/data
Note that diffuse response is calculated for 'source' and 'clean' event classes only.
Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL5-32 (RHEL6-64 are built but not yet validated), while ScienceTools builds for RHEL5-32, RHEL5-64.