...
This page is a record of the configuration and execution of the P310 reprocessing project, generation FT2 files.
...
Adding velocity columns to FT2 Files
Pipeline tasks
- P310-FT2 - Create FT2 files with velocity columns
- flagFT2-P310 - Flag BTIs in newly created FT2 files
...
| Code Block |
|---|
/Data/Flight/Reprocess/P305P310 Group FT1FT2 |
Data Sample
The currently defined data sample (as of ) for P305 reprocessing includes:
First run | 239557414 (MET), 2008-08-04 15:43:34 (UTC) |
Last run | 561751200 (MET), 2018-10-20 20:31:47 (UTC)TBD |
| Backfill | 564939451 (MET), 2018-11-26 16:46:00 (UTC) |
...
| TBD |
Bookkeeping (Update this once the tasks are in PROD)
- (This page): Define ingredients of reprocessing (processing code/configuration changes)
- Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P305 (TBD)
- List of all reprocessings
- List of all data runs reprocessed
- Pointers to all input data files (-> dataCatalog)
- Pointers to associated task processes (-> Pipeline II status)
- Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
- Lists of and pointers to all output data files
- Meta data associated with each output data product
...
P310-FITS
...
, flagFT2-P310
...
This task generates all desired FITS data products.
- A slightly updated task structure has been implemented for this task: the mergeClumps step is now a separate subTask which allows overriding the batch queue parameter for extra long runs. This restructuring required quite a lot of modification related to data passed between job steps (as now certain job steps are in the new subTask).
- Task stored in my github as P305-FITS
Status chronology
This task basically mirrors the P301-FITS task but updating certain packages and expanding the generated data products to include FT1 and LS1.
Please see the associated JIRA for operational notes.
- 2017-09-18: start year 1 production
- 2018-10-18: update the scripts to not fail on new "centos7" systems.
Configuration
Configuration (To Be Updated)
Task Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P302-FITS |
Task Status | complete |
Input Data | MERIT (direct from P301-MERIT task) |
spacecraft data | P203 (released 3 June 2014) |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P305-FITS/config/runList.txt |
evtClassDefs | N/A |
eventClassMap | /nfs/farm/g/glast/u38/Reprocess-tasks/P305-FITS/config/EvtClassDefs_P8R2_HeavyIons_Ribbon.xml |
ScienceTools | 09-35-02 (1/26/2015) |
Code Variants | redhat6-x86_64-64bit-gcc44 (Optimized) |
Diffuse Model | N/A |
Diffuse Response | N/A |
IRFs | N/A |
Output Data Products | |
| commonTools | 00-02-01 |
Generation of output data products:
Data Product | destination | data content [1] | event selection [1] | makeFT1 | gtselect | gtdiffrsp | gtmktime | Notes |
|---|---|---|---|---|---|---|---|---|
EXTENDEDFT1 | SLAC | pass8.2_Extended_cuts_L1 | pass8.2_Transient_cuts_L1 |
|
|
|
| |
| FT1 | FSSC | pass8_FT1variables | evclass=128 |  |  | | | produced from extendedFT1 |
EXTENDEDLS1 | SLAC | pass8.2_Extended_cuts_L1 | pass8.2_Transient_cuts_L1 | | | | | |
| LS1 | FSSC | pass8_LS1variables | evclass=65544 | | | | | produced from extendedLS1 |
...
| -01 |
...
Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL5-32 (RHEL6-64 are built but not yet validated), while ScienceTools builds for RHEL5-64, RHEL6-64.
Timing and Scaling
...