status: Running Complete
last update: 20 May 201427 June 2015
This page is a record of the configuration and execution of the P301 reprocessing project, regeneration of MERIT and FITS files from the P300 MERIT using TMine.
...
- P301-FITS - This task reads new P301 MERIT and produces fits files, EXTENDEDFT1 + EXTENDEDLS1 ** THIS TASK IS DEFUNCT, see P302-FITS **
Datafile names, versions and locations
...
- 3/27/2014 - Begin setting up P301 task for Pass8 re-MERIT.
- 5/5/2014 - Please see the P301 JIRA for future chronology updates
- 6/9/2014 - Begin block 2, through end of April 2014 (see JIRA)
- 6/10/2014 - Block 2 complete
- 10/21/2014 - Begin block 3, through end of September 2014 (see JIRA)
- 10/22/2014 - Block 3 complete
- 02/17/2014 - Begin block 4, through end of February 2015
- 02/18/2015 - Block 4 complete
- 04/16/2015 - Begin block 5, through 7 Apr 2015
- 04/18/2015 - Block 5 complete
- 06/11/2015 - Block 6 through 3 Jun 2015 started and completed.
- 06/26/2015 - Block 7 - the final block - through 24 June 2015 started.
- 06/27/2015 - Finished with Engines.
- 08/05/2015 - Emergency reprocessing of 40 runs that suffered bad CAL calibrations. (See P300-ROOT for details.)
Configuration
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P301-MERIT |
Task Status | |
TMineExt | 3.3.8 (17 Apr 2014) as used from $GLAST_EXT/TMineExt |
| Classification worksheet | v0r16 from $GLAST_EXT/extFiles |
Run Selection | All MERIT files produced by P300-ROOT task |
s/c data | P202 FT2SECONDS which will eventually become a "standard" Public Release https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P301-MERIT/config/runList.txt |
Code Variants used | redhat5-i686-32bit-gcc41, redhat5-x86_64-64bit-gcc41, redhat6-x86_64-64bit-gcc44 (Optimized) |
Output Data Products | MERIT |
...
This task generates all desired FITS data products.
- A slightly updated task structure has been implemented for this task: the mergeClumps step is now a separate subTask which allows overriding the batch queue parameter for extra long runs. This restructuring required quite a lot of modification related to data passed between job steps (as now certain job steps are in the new subTask).
- Task stored in my github as P301-FITS
Status chronology
This task basically mirrors the P301-MERIT task, running after the aforementioned completes
Please see the associated JIRA for operational notes.
...
Task Location | /nfs/farm/g/glast/u38/Reprocess-tasks/P301-FITS |
Task Status | http://glast-ground.slac.stanford.edu/Pipeline-II/exp/Fermi/task.jsp?refreshRate=60&task=137594093 |
Input Data | MERIT (direct from P301-MERIT task) |
spacecraft data | same as P301-MERIT |
Input Run List | ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P301-FITS/config/runList.txt |
evtClassDefs | 01-00-01 |
eventClassMap | EvtClassDefs_P8V1.xml |
ScienceTools | 09-33-01 (5/15/2014) |
Code Variants | redhat5-x86_64-64bit-gcc41 & redhat6-x86_64-64bit-gcc44 (Optimized) |
Diffuse Model | N/A |
Diffuse Response | N/A |
IRFs | N/A |
Output Data Products | EXTENDEDFT1, EXTENDEDLS1 |
| commonTools | 00-02-01 |
Generation of output data products:
...
Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL5-32 (RHEL6-64 are built but not yet validated), while ScienceTools builds for RHEL5-64, RHEL6-64.
Timing and Scaling
(no data)Approx 20 CPU minutes and 30 wall clock minutes per job (run).