status: In development
last update: 16 January 2015
This page is a record of the configuration and execution of the P302 reprocessing project, generation FITS files from the P301 MERIT files.
Notice: there is now a JIRA for this task. Please check here for operational notes.
Pipeline tasks
- P302-FITS - This task reads new P301 MERIT and produces fits files, FT1 + LS1 + EXTENDEDFT1 + EXTENDEDLS1
Datafile names, versions and locations
Data file version numbers for this reprocessing will begin with v302.
XROOT location and file naming
Location template:
/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>
Locations for P302:
/glast/Data/Flight/Reprocess/P302/ft1 /glast/Data/Flight/Reprocess/P302/extendedft1 /glast/Data/Flight/Reprocess/P302/ls1 /glast/Data/Flight/Reprocess/P302/extendedls1
File naming:
Data Type | aka | Send to FSSC | Naming template |
---|---|---|---|
EXTENDEDFT1 |
| No | gll_xp_p<procVer>_r<run#>_<version>.fit |
FT1 | LS-002 | Yes | gll_ph_p<procVer>_r<run#>_<version>.fit |
EXTENDEDLS1 |
| No | gll_xe_p<procVer>_r<run#>_<version>.fit |
LS1 | LS-001 | Yes | gll_ev_p<procVer>_r<run#>_<version>.fit |
Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf
Example:
/glast/Data/Flight/Reprocess/P302/recon/r0239557414_v302_merit.root
DataCatalog location and naming
Logical directory and group template:
Data/Flight/Reprocess/<reprocessName>:<dataType>
Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.
Naming examples:
Data/Flight/Reprocess/P302:MERIT r0239557414
Data Sample
The currently defined data sample (as of 1/16/2015) for P302 reprocessing includes:
First run | 239557414 (MET), 2008-08-04 15:43:34 (UTC) |
Last run | present |
Bookkeeping
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P302
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
P302-FITS
This task generates all desired FITS data products.
- A slightly updated task structure has been implemented for this task: the mergeClumps step is now a separate subTask which allows overriding the batch queue parameter for extra long runs. This restructuring required quite a lot of modification related to data passed between job steps (as now certain job steps are in the new subTask).
- Task stored in my github as P302-FITS
Status chronology
This task basically mirrors the P301-FITS task but updating certain packages and expanding the generated data products to include FT1 and LS1.
Please see the associated JIRA for operational notes.
Configuration
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P302-FITS |
Task Status | |
Input Data | MERIT (direct from P301-MERIT task) |
spacecraft data | same as P301-MERIT |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P302-FITS/config/runList.txt |
evtClassDefs | 01-01-00 |
eventClassMap | EvtClassDefs_P8R2.xml |
ScienceTools | 09-33-01 (5/15/2014) |
Code Variants | redhat5-x86_64-64bit-gcc41 & redhat6-x86_64-64bit-gcc44 (Optimized) |
Diffuse Model | N/A |
Diffuse Response | N/A |
IRFs | N/A |
Output Data Products | |
commonTools | 00-02-01 |
Generation of output data products:
Data Product | destination | data content [1] | event selection [1] | makeFT1 | gtselect | gtdiffrsp | gtmktime |
---|---|---|---|---|---|---|---|
FT1 | FSSC | ||||||
EXTENDEDFT1 | SLAC | pass8_FT1variables | pass8.2_Transient_cuts_L1 | ||||
LS1 | FSSC | ||||||
EXTENDEDLS1 | SLAC | pass8_LS1variables | pass8.2_Transient_cuts_L1 |
[1] /afs/slac/g/glast/ground/releases/volume04/evtClassDefs/01-01-00/data
Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL5-32 (RHEL6-64 are built but not yet validated), while ScienceTools builds for RHEL5-64, RHEL6-64.
Timing and Scaling
Approx 20 CPU minutes and 30 wall clock minutes per job (run).