status: Running
last update: 2 July 2013
This page is a record of the configuration and execution of the P300 reprocessing project, full reprocessing from DIGIs using Pass8 analysis code.
Pipeline tasks
- P300-ROOT - This task reads DIGI and produces reprocessed RECON + GCR + MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT
- P300-FITS - This task (once it has been created) reads MERIT and produces FT1 (photons) + EXTENDEDFT1 + LS1 (merit-like FITS file for photons) + EXTENDEDLS1 + ELECTRONFITS file
Datafile names, versions and locations
Data file version numbers for this reprocessing will begin with v300.
XROOT location and file naming
Location template:
/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>
Locations for P300:
/glast/Data/Flight/Reprocess/P300/recon /glast/Data/Flight/Reprocess/P300/gcr /glast/Data/Flight/Reprocess/P300/relation/glast/Data/Flight/Reprocess/P300/merit /glast/Data/Flight/Reprocess/P300/filteredmerit /glast/Data/Flight/Reprocess/P300/electronmerit /glast/Data/Flight/Reprocess/P300/ft1 /glast/Data/Flight/Reprocess/P300/extendedft1 /glast/Data/Flight/Reprocess/P300/electronft1 /glast/Data/Flight/Reprocess/P300/ls1 /glast/Data/Flight/Reprocess/P300/extendedls1
File naming:
Data Type |
aka |
Send to FSSC |
Naming template |
---|---|---|---|
RECON |
|
No |
r<run#>_<version>_<dataType>.root |
RELATION |
|
No |
r<run#>_<version>_<dataType>.root |
GCR |
|
No |
r<run#>_<version>_<dataType>.root |
MERIT |
|
No |
r<run#>_<version>_<dataType>.root |
FILTEREDMERIT |
|
No |
r<run#>_<version>_<dataType>.root |
ELECTRONMERIT |
|
No |
r<run#>_<version>_<dataType>.root |
ELECTRONFT1 |
|
No |
gll_el_p<procVer>_r<run#>_<version>.fit |
EXTENDEDFT1 |
|
No |
gll_xp_p<procVer>_r<run#>_<version>.fit |
FT1 |
LS-002 |
Yes |
gll_ph_p<procVer>_r<run#>_<version>.fit |
EXTENDEDLS1 |
|
No |
gll_xe_p<procVer>_r<run#>_<version>.fit |
LS1 |
LS-001 |
Yes |
gll_ev_p<procVer>_r<run#>_<version>.fit |
Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf
Example:
/glast/Data/Flight/Reprocess/P300/recon/r0239557414_v202_recon.root
DataCatalog location and naming
Logical directory and group template:
Data/Flight/Reprocess/<reprocessName>:<dataType>
Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.
Naming examples:
Data/Flight/Reprocess/P300:RECON r0239557414
Data Sample
The currently defined data sample (as of 7/2/2013) for P300 reprocessing includes:
First run |
239557414 (MET), 2008-08-04 15:43:34 (UTC) |
Last run |
present |
Bookkeeping
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P300
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
P300-ROOT
Status chronology
- 4/23/2013 - Begin setting up P300 task for Pass8 reprocessing.
- 5/10/2013 - Initial version of task running. See this page for infomation about the task creation, and this page for performance comparison with P202
- 7/2/2013 - after weeks of testing, final GR, jobOpts, etc. come together and production tentatively starts.
- block 1 is defined with this run range: 239557414 through 392842073 (2008-08-04 15:43:37 through 2013-06-13 18:47:50 UTC)
- Update task to use the P202 generation of FT2 files (adds columns LAMBDA, RA_SUN, DEC_SUN, as well as meaningful BTIs)
- First 10 trial jobs launched. Too many ACD INFO messages 'Missed Poca for ID 602 at...' so modify jobOpts to disable.
- xroot disk space at the start of Pass8 = 508 TB (just after commissioning of fermi-xrd11)
- 7/29/2013 - a new GR is prepared (20-09-01) which fixes some problems in ACD tracking issues affecting ~32 runs so far (out of 5283). trickleStream disabled, wait for pending jobs to start, rollback other failed jobs, switch to new GR, rollback of ACD-failed jobs...
Configuration
Task Location |
/nfs/farm/g/glast/u38/Reprocess-tasks/P300-ROOT |
Task Status |
|
GlastRelease |
20-09-00 streams 0 - 5292 (02 Jul 2013) |
Run Selection |
based on a modified "standard" selection, see https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
s/c data |
P202 FT2SECONDS which will eventually become a "standard" Public Release https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
Input Run List |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300-ROOT/config/runList.txt |
photonFilter |
Not yet applicable, although this is defined by default: CTBParticleType==1 && ((FT1EventClass & 0x00003EFF)!=0) |
electronFilter |
Not yet applicable, although this is defined by default: CTBParticleType==1 |
Code Variants |
redhat5-i686-32bit-gcc41 (Optimized), note that rhel5-64 and rhel6-64 GR builds are not yet available |
jobOpts |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300-ROOT/config/Pass8Recon.txt |
Output Data Products |
RECON, GCR, RELATION, MERIT, FILTEREDMERIT, ELECTRONMERIT (but note that FILTEREDMERIT and ELECTRONMERIT are empty or contain junk) |
Timing and Scaling
- Current performance results for the Pass8 code are on this page .
- For 20,000 event clumps:
job step
average CPU time
processClump
183 min
mergeClumps
10 min
P300-FITS
This task generates all desired FITS data products. This task does not yet exist.
Status chronology
Configuration
NOT YET UPDATED
Task Location |
/nfs/farm/g/glast/u38/Reprocess-tasks/P300-FITS |
Task Status |
http://glast-ground.slac.stanford.edu/Pipeline-II/task.jsp?task=75031156 |
Input Data |
MERIT (direct from P300-ROOT) |
spacecraft data |
same as P300-ROOT |
Input Run List |
ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-FITS/config/runList.txt |
evtClassDefs |
00-19-04 (6/23/2011) |
eventClassMap |
EvtClassDefs_P7V6.xml |
ScienceTools |
09-27-01 (2/15/2012) 09-31-01 (3/5/2013) |
Code Variants |
redhat5-i686-32bit-gcc41 (Optimized) |
Diffuse Model |
based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 /v3r0 |
Diffuse Response |
'source' using P7SOURCE_V6 P7REP_SOURCE_V10 IRF |
IRFs |
P7REP_*_V10, contained within ScienceTools release |
Output Data Products |
Generation of output data products:
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="2d6ac3ab-0b21-43eb-9abe-228e5a475d14"><ac:plain-text-body><![CDATA[ |
Data Product |
destination |
data content [1] |
event selection [1] |
makeFT1 |
gtselect |
gtdiffrsp |
gtmktime |
]]></ac:plain-text-body></ac:structured-macro> |
---|---|---|---|---|---|---|---|---|---|
EXTENDEDFT1 |
SLAC |
FT1variables |
((FT1EventClass & 0x00003EFF)!=0) |
|
|
|
|
||
FT1 |
FSSC+SLAC |
FT1variables |
'source' and above |
|
|
(inherited) |
|
||
EXTENDEDLS1 |
SLAC |
LS1variables |
((FT1EventClass & 0x00003EFF)!=0) |
|
|
|
|
||
LS1 |
FSSC+SLAC |
LS1variables |
'transient' and above |
|
|
(inherited) |
|
||
ELECTRONFT1 |
SLAC |
FT1variables |
CTBParticleType==1 |
|
|
|
|
[1] /afs/slac/g/glast/ground/releases/volume04/evtClassDefs/00-19-04/data
Note that diffuse response is calculated for 'source' and 'clean' event classes only.
Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL5-32 (RHEL6-64 are built but not yet validated), while ScienceTools builds for RHEL5-32, RHEL5-64.