status: Runable
last update: 5 May 2014
This page is a record of the configuration and execution of the P301 reprocessing project, regeneration of MERIT and FITS files from the P300 MERIT using TMine.
Notice: there is now a JIRA for this task
Pipeline tasks
- P301-MERIT - This task reads MERIT and produces reprocessed MERIT. This is the first time TMineExt has been used to perform this function.
- P301-FITS - This task (once it has been created) reads MERIT and produces FT1 (photons) + EXTENDEDFT1 + LS1 (merit-like FITS file for photons) + EXTENDEDLS1 + ELECTRONFITS file
Datafile names, versions and locations
Data file version numbers for this reprocessing will begin with v300.
XROOT location and file naming
Location template:
/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>
Locations for P301:
/glast/Data/Flight/Reprocess/P301/merit [soon] /glast/Data/Flight/Reprocess/P301/ft1 /glast/Data/Flight/Reprocess/P301/extendedft1 /glast/Data/Flight/Reprocess/P301/electronft1 /glast/Data/Flight/Reprocess/P301/ls1 /glast/Data/Flight/Reprocess/P301/extendedls1
File naming:
Data Type | aka | Send to FSSC | Naming template |
|---|---|---|---|
MERIT |
| No | r<run#>_<version>_<dataType>.root |
FILTEREDMERIT |
| No | r<run#>_<version>_<dataType>.root |
ELECTRONMERIT |
| No | r<run#>_<version>_<dataType>.root |
ELECTRONFT1 |
| No | gll_el_p<procVer>_r<run#>_<version>.fit |
EXTENDEDFT1 |
| No | gll_xp_p<procVer>_r<run#>_<version>.fit |
FT1 | LS-002 | Yes | gll_ph_p<procVer>_r<run#>_<version>.fit |
EXTENDEDLS1 |
| No | gll_xe_p<procVer>_r<run#>_<version>.fit |
LS1 | LS-001 | Yes | gll_ev_p<procVer>_r<run#>_<version>.fit |
Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf
Example:
/glast/Data/Flight/Reprocess/P301/recon/r0239557414_v301_merit.root
DataCatalog location and naming
Logical directory and group template:
Data/Flight/Reprocess/<reprocessName>:<dataType>
Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.
Naming examples:
Data/Flight/Reprocess/P301:MERIT r0239557414
Data Sample
The currently defined data sample (as of 4/18/2014) for P301 reprocessing includes:
First run | 239557414 (MET), 2008-08-04 15:43:34 (UTC) |
Last run | present |
Bookkeeping
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P301
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
P301-MERIT
Status chronology
- 3/27/2014 - Begin setting up P301 task for Pass8 re-MERIT.
- 5/5/2014 - Please see the P301 JIRA for future chronology updates
Configuration
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P301-MERIT |
Task Status | |
TMineExt | 3.3.8 (17 Apr 2014) as used from $GLAST_EXT/TMineExt |
| Classification worksheet | v0r16 from $GLAST_EXT/extFiles |
Run Selection | All MERIT files produced by P300-ROOT task |
s/c data | P202 FT2SECONDS which will eventually become a "standard" Public Release https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P301-MERIT/config/runList.txt |
Code Variants used | redhat5-i686-32bit-gcc41, redhat5-x86_64-64bit-gcc41, redhat6-x86_64-64bit-gcc44 (Optimized) |
Output Data Products | MERIT |
Timing and Scaling
job step
average CPU time
TMine
~0.5-1M evts/hour
P301-FITS
This task generates all desired FITS data products.
** THE FOLLOWING INFORMATION IS INCORRECT, DO NOT USE **
Status chronology
Configuration
CAUTION: The following data describes the experimental P300x-FITS task, not the production task
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P301-FITS |
Task Status | http://glast-ground.slac.stanford.edu/Pipeline-II/exp/Fermi/task.jsp?task=112641219 |
Input Data | MERIT (direct from P300-ROOT) |
spacecraft data | same as P300-ROOT |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300x-FITS/config/runList.txt |
evtClassDefs | untagged version from 20130801 |
eventClassMap | EvtClassDefs_P8.xml |
ScienceTools | 09-32-05 (7/30/2013) |
Code Variants | redhat5-x86_64-64bit-gcc41 & redhat6-x86_64-64bit-gcc44 (Optimized) |
Diffuse Model | based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 /v3r0 |
Diffuse Response | N/A |
IRFs | N/A |
Output Data Products |
WARNING: THIS NEXT SECTION IS OBSOLETE
Generation of output data products:
Data Product | destination | data content [1] | event selection [1] | makeFT1 | gtselect | gtdiffrsp | gtmktime |
|---|---|---|---|---|---|---|---|
EXTENDEDFT1 | SLAC | FT1variables | ((FT1EventClass & 0x00003EFF)!=0) |
|
|
|
|
FT1 | FSSC+SLAC | FT1variables | 'source' and above |
|
| (inherited) |
|
EXTENDEDLS1 | SLAC | LS1variables | ((FT1EventClass & 0x00003EFF)!=0) |
|
|
|
|
LS1 | FSSC+SLAC | LS1variables | 'transient' and above |
|
| (inherited) |
|
ELECTRONFT1 | SLAC | FT1variables | CTBParticleType==1 |
|
|
|
|
[1] /afs/slac/g/glast/ground/releases/volume04/evtClassDefs/00-19-04/data
Note that diffuse response is calculated for 'source' and 'clean' event classes only.
Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL5-32 (RHEL6-64 are built but not yet validated), while ScienceTools builds for RHEL5-64, RHEL5-64.
Timing and Scaling
(no data)