status: Runable Complete
last update: 5 May 201427 June 2015
This page is a record of the configuration and execution of the P301 reprocessing project, regeneration of MERIT and FITS files from the P300 MERIT using TMine.
Notice: there is now a JIRA for this task. Please check here for operational notes.
Pipeline tasks
- P301-MERIT - This task reads P300 MERIT and produces reprocessed MERIT. This is the first time TMineExt has been used to perform this function.
- P301-FITS - This task (once it has been created) reads task reads new P301 MERIT and produces FT1 (photons) + EXTENDEDFT1 + LS1 (merit-like FITS file for photons) + EXTENDEDLS1 + ELECTRONFITS filefits files, EXTENDEDFT1 + EXTENDEDLS1 ** THIS TASK IS DEFUNCT, see P302-FITS **
Datafile names, versions and locations
...
- 3/27/2014 - Begin setting up P301 task for Pass8 re-MERIT.
- 5/5/2014 - Please see the P301 JIRA for future chronology updates
- 6/9/2014 - Begin block 2, through end of April 2014 (see JIRA)
- 6/10/2014 - Block 2 complete
- 10/21/2014 - Begin block 3, through end of September 2014 (see JIRA)
- 10/22/2014 - Block 3 complete
- 02/17/2014 - Begin block 4, through end of February 2015
- 02/18/2015 - Block 4 complete
- 04/16/2015 - Begin block 5, through 7 Apr 2015
- 04/18/2015 - Block 5 complete
- 06/11/2015 - Block 6 through 3 Jun 2015 started and completed.
- 06/26/2015 - Block 7 - the final block - through 24 June 2015 started.
- 06/27/2015 - Finished with Engines.
- 08/05/2015 - Emergency reprocessing of 40 runs that suffered bad CAL calibrations. (See P300-ROOT for details.)
Configuration
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P301-MERIT |
Task Status | |
TMineExt | 3.3.8 (17 Apr 2014) as used from $GLAST_EXT/TMineExt |
| Classification worksheet | v0r16 from $GLAST_EXT/extFiles |
Run Selection | All MERIT files produced by P300-ROOT task |
s/c data | P202 FT2SECONDS which will eventually become a "standard" Public Release https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P301-MERIT/config/runList.txt |
Code Variants used | redhat5-i686-32bit-gcc41, redhat5-x86_64-64bit-gcc41, redhat6-x86_64-64bit-gcc44 (Optimized) |
Output Data Products | MERIT |
...
This task generates all desired FITS data products.
** THE FOLLOWING INFORMATION IS INCORRECT, DO NOT USE **
Status chronology
Configuration
...
- A slightly updated task structure has been implemented for this task: the mergeClumps step is now a separate subTask which allows overriding the batch queue parameter for extra long runs. This restructuring required quite a lot of modification related to data passed between job steps (as now certain job steps are in the new subTask).
- Task stored in my github as P301-FITS
Status chronology
This task basically mirrors the P301-MERIT task, running after the aforementioned completes
Please see the associated JIRA for operational notes.
Configuration
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P301-FITS |
Task Status | http://glast-ground.slac.stanford.edu/Pipeline-II/exp/Fermi/task.jsp?refreshRate=60&task=112641219137594093 |
Input Data | MERIT (direct from P300P301-ROOTMERIT task) |
spacecraft data | same as P300P301-ROOTMERIT |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300xP301-FITS/config/runList.txt |
evtClassDefsuntagged version from 20130801 | 01-00-01 |
eventClassMap | EvtClassDefs_P8P8V1.xml |
ScienceTools | 09-3233-05 01 (75/3015/20132014) |
Code Variants | redhat5-x86_64-64bit-gcc41 & redhat6-x86_64-64bit-gcc44 (Optimized) |
Diffuse Model | based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 /v3r0 |
Diffuse Response | N/A |
IRFs | N/A |
Output Data Products |
...
, EXTENDEDLS1 | |
| commonTools | 00-02-01 |
Generation of output data products:
Data Product | destination | data content [1] | event selection [1] | makeFT1 | gtselect | gtdiffrsp | gtmktime | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
EXTENDEDFT1 | SLAC | pass8_FT1variables | ((FT1EventClass & 0x00003EFF)!=0) |
|
|
|
| FT1 | FSSC+SLAC | FT1variables | 'source' and above EVENT_CLASS bits 2,3,4 |
|
| (inherited) |
|
EXTENDEDLS1 | SLAC | pass8_LS1variables | ((FT1EventClass & 0x00003EFF)!=0) |
|
|
|
| ||||||||
LS1 | FSSC+SLAC | LS1variables | 'transient' and above |
|
| (inherited) |
| ||||||||
ELECTRONFT1 | SLAC | FT1variables | CTBParticleType==1 |
|
|
|
|
[1] /afs/slac/g/glast/ground/releases/volume04/evtClassDefs/01-00-19-04/dataNote that diffuse response is calculated for 'source' and 'clean' event classes only.01/data
Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL5-32 (RHEL6-64 are built but not yet validated), while ScienceTools builds for RHEL5-64, RHEL5RHEL6-64.
Timing and Scaling
(no data)Approx 20 CPU minutes and 30 wall clock minutes per job (run).