Versions Compared

Old Version 1

changes.mady.by.user Tom Glanzman

Saved on

compared with

New Version Current

changes.mady.by.user Tom Glanzman

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

status: Running Complete
last update: 18 Apr 201427 June 2015

This page is a record of the configuration and execution of the P301 reprocessing project, regeneration of MERIT and FITS files from the P300 MERIT using TMine.

Notice: there is now a JIRA for this task.  Please check here for operational notes.
Pipeline tasks
  • P301-MERIT - This task reads P300 MERIT and produces reprocessed MERIT.  This is the first time TMineExt has been used to perform this function.
  • P301-FITS - This task (once it has been created) reads task reads new P301 MERIT and produces FT1 (photons) + EXTENDEDFT1 + LS1 (merit-like FITS file for photons) + EXTENDEDLS1 + ELECTRONFITS filefits files, EXTENDEDFT1 + EXTENDEDLS1 ** THIS TASK IS DEFUNCT, see P302-FITS **

Datafile names, versions and locations

...

Code Block
/glast/Data/Flight/Reprocess/P301/recon
/glast/Data/Flight/Reprocess/P301/gcr
/glast/Data/Flight/Reprocess/P301/relation
/glast/Data/Flight/Reprocess/P301/merit
/glast/Data/Flight/Reprocess/P301/filteredmerit
/glast/Data/Flight/Reprocess/P301/electronmerit
[soon]
/glast/Data/Flight/Reprocess/P301/ft1
/glast/Data/Flight/Reprocess/P301/extendedft1
/glast/Data/Flight/Reprocess/P301/electronft1
/glast/Data/Flight/Reprocess/P301/ls1
/glast/Data/Flight/Reprocess/P301/extendedls1

...

Code Block
/glast/Data/Flight/Reprocess/P301/recon/r0239557414_v301_reconmerit.root
DataCatalog location and naming

...

Code Block
Data/Flight/Reprocess/P301:RECONMERIT r0239557414

Data Sample

The currently defined data sample (as of 4/18/2014) for P301 reprocessing includes:

...

  1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
  2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P301
    1. List of all reprocessings
    2. List of all data runs reprocessed
    3. Pointers to all input data files (-> dataCatalog)
    4. Pointers to associated task processes (-> Pipeline II status)
  3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
    1. Lists of and pointers to all output data files
    2. Meta data associated with each output data product

...

P301-

...

MERIT
Anchor

...

P301-

...

MERIT

...

P301-

...

MERIT

Status chronology

  • 43/2327/2013 2014 - Begin setting up P300 P301 task for Pass8 reprocessingre-MERIT.
  • 5/105/2013 - Initial version of task running.  See this page for infomation about the task creation, and this page for performance comparison with P202
  • 7/2/2013 - after weeks of testing, final GR, jobOpts, etc. come together and production tentatively starts.
    • Block 1 is defined with this run range: 239557414 through 392842073 (2008-08-04 15:43:37 through 2013-06-13 18:47:50 UTC)
    • Update task to use the P202 generation of FT2 files (adds columns LAMBDA, RA_SUN, DEC_SUN, as well as meaningful BTIs)
    • First 10 trial jobs launched. Too many ACD INFO messages 'Missed Poca for ID 602 at...' so modify jobOpts to disable.
    • xroot disk space at the start of Pass8 = 508 TB (just after commissioning of fermi-xrd11)
  • 7/29/2013 - a new GR is prepared (20-09-01) which fixes some problems in ACD tracking issues affecting ~32 runs so far (out of 5283). trickleStream disabled, wait for pending jobs to start, rollback other failed jobs, switch to new GR, rollback of ACD-failed jobs...
    • This new GR continues to seg fault, so revert back to GR 20-09-00 and restart trickleStream.
  • 7/31/2013 - A new GR (20-09-02) with patch for ACD tracking installed after stream 5970. At the time, there were 33 failed processClumps jobs. All rolled back.
    • Last stream with GR 20-09-00 = 5970, run 274118943
    • First stream with GR 20-09-02 = 5971, run 274124672
    • Full list of rolled back substreams is here
  • 10/15/2013 - Update job options to use the new "L1current" flavor of calibrations/alignment rather than "p7repro".  First stream with new configuration: 20804.
  • 11/14/2013 - Block 1 complete except for routine cleanup

  • 11/21/2013 - Block 1 complete
    Problems encountered include:
    ** pipeline_summary file corruption causing loss of email or of it being ignored, finally jobs terminate via reaper (Tony notified)
    ** large blocks of failures due to dataCatalog not returning desired FT2 files (Brian notified, fix possible by moving query to scriptlet)
    ** jobs running out the clock due to anomalously slow xroot behavior (Wilko/Andy notified)
    ** problems with individual batch machines (Renata notified, and I also have the authority to remove batch machines from production)
    Summary of events processed in Block 1 from dataCatalog:

    NameTypeFilesEventsSizeCreated (UTC)Links
    ELECTRONMERITGroup2692901.4 GB09-May-2013 22:22:19Files
    FILTEREDMERITGroup2692901.4 GB09-May-2013 22:22:17Files
    GCRGroup2692958,653,424,7081.0 TB09-May-2013 22:22:18Files
    MERITGroup2692958,653,424,70898.3 TB09-May-2013 22:22:18Files
    RECONGroup2692958,653,424,708816.1 TB09-May-2013 22:22:20Files
    RELATIONGroup2692958,653,424,7087.7 TB09-May-2013 22:22:20Files
  • 2014 - Please see the P301 JIRA for future chronology updates
  • 6/9/2014 - Begin block 2, through end of April 2014 (see JIRA)
  • 6/10/2014 - Block 2 complete
  • 10/21/2014 - Begin block 3, through end of September 2014 (see JIRA)
  • 10/22/2014 - Block 3 complete
  • 02/17/2014 - Begin block 4, through end of February 2015
  • 02/18/2015 - Block 4 complete
  • 04/16/2015 - Begin block 5, through 7 Apr 2015
  • 04/18/2015 - Block 5 complete
  • 06/11/2015 - Block 6 through 3 Jun 2015 started and completed.
  • 06/26/2015 - Block 7 - the final block - through 24 June 2015 started.
  • 06/27/2015 - Finished with Engines.
  • 08/05/2015 - Emergency reprocessing of 40 runs that suffered bad CAL calibrations. (See P300-ROOT for details.)

Configuration

Task Location

/nfs/farm/g/glast/u38/Reprocess-tasks/P300P301-ROOTMERIT

Task Status

http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp

GlastRelease

20-09-00 streams 0 - 5970 (02 Jul 2013)
20-09-02 streams 5971 - end (31 Jul 2013)

TMineExt

3.3.8 (17 Apr 2014) as used from $GLAST_EXT/TMineExt

Classification worksheetv0r16 from $GLAST_EXT/extFiles

Run Selection

All MERIT files produced by P300-ROOT task

Run Selection

based on a modified "standard" selection, see https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets
(((sIntent=="nomSciOps" || sIntent=="nomSO_noSk_noCno_optGccc_allEna" || sIntent=="nomSciOps_diagEna" || (sIntent=="nomSciOps_Emin5MeV"&&RunMin>242070455) || nRun==242429468 ) && (RunQuality != "Bad" || is_null ( RunQuality ) ) ) || sIntent=="nadirOps" )

s/c data

P202 FT2SECONDS which will eventually become a "standard" Public Release https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets

Input Run List

ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300P301-ROOTMERIT/config/runList.txt

photonFilter

Not yet applicable, although this is defined by default:  CTBParticleType==1 && ((FT1EventClass & 0x00003EFF)!=0)
pass7.6_Extended_cuts_L1 in evtClassDefs

electronFilter

Not yet applicable, although this is defined by default:  CTBParticleType==1

Code Variants used

redhat5-i686-32bit-gcc41 (Optimized), note that rhel5-64 and rhel6-64 GR builds are not yet available

jobOpts

ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300-ROOT/config/Pass8Recon.txt

calibrations & alignments

"p7repro" and "L1current" (the latter to conform with new Level 1 flavor, and to include new ACD calibration for 9 Sep 2012, redhat5-x86_64-64bit-gcc41, redhat6-x86_64-64bit-gcc44 (Optimized)

Output Data Products

RECON, GCR, RELATION, MERIT, FILTEREDMERIT, ELECTRONMERIT (but note that FILTEREDMERIT and ELECTRONMERIT are empty or contain junk)

Timing and Scaling

...

  • job step

    average CPU time

    processClump

    183 min

    mergeClumps

    10 min

  • Plot of job throughput as of 20130813
    Image Removed
    From the plot, one can see a processing rate of ~208 runs/day. Assuming 28500 total runs, this would mean a repro time of 137 days or 4.5 months.
  • Another plot showing the number of run scratch directories cleaned up per day:
    Image Removed

...

  • TMine

    ~0.5-1M evts/hour

P301-FITS
Anchor

...

P301-FITS

...

P301-FITS

This task generates all desired FITS data products.

NOTE: a temporary task called P300x-FITS has been created. It generates only EXTENDEDFT1 files and nothing else. There is no diffuse calculation performed. An untagged version of evtClassDefs is used which contains Matthew Wood's initial event classification and selections. This will likely be a template for a future production P300-FITS task. All data produced by this task are "throw-away" and not expected to survive beyond initial validation studies.

Status chronology

...

8/7/2013 - First year of data run through the task, amounting to 5537 runs and 11790391606 events:

First run

239557417

2008-08-04 15:43:37

Last run

271850279

2009-08-13 09:57:59

...

8/28/2013 - Second year of data ready and processed -> extendedFT1. Note that 36 troublesome runs omitted from 1st year data were included in this sample. There should be no missing runs in this sample.

First run

239557417

2008-04-2008 15:43:37

Last run

302647722

2010-08-04 20:48:40

10,976 runs, 2,677,747,829 events, 234.7 GB

...

9/25/2013 - Third year of data ready.

First run

239557417

2008-04-2008 15:43:37

Last run

334184989

2011-08-04 21:09:47

This block contains 16,516 runs and 36,056,385,847 events, or an increase over year 2 of 5,540 runs.

...

NameTypeFilesEventsSizeCreated (UTC)Links
EXTENDEDFT1Group269297,149,992,671626.6 GB01-Aug-2013 17:29:46Files
  • A slightly updated task structure has been implemented for this task: the mergeClumps step is now a separate subTask which allows overriding the batch queue parameter for extra long runs.  This restructuring required quite a lot of modification related to data passed between job steps (as now certain job steps are in the new subTask). 
  • Task stored in my github as P301-FITS

 

Status chronology

This task basically mirrors the P301-MERIT task, running after the aforementioned completes

Please see the associated JIRA for operational notes.

 

Configuration

Configuration

CAUTION: The following data describes the experimental P300x-FITS task, not the production task

Task Location

/nfs/farm/g/glast/u38/Reprocess-tasks/P300xP301-FITS

Task Status

http://glast-ground.slac.stanford.edu/Pipeline-II/exp/Fermi/task.jsp?refreshRate=60&task=112641219137594093

Input Data

MERIT (direct from P300P301-ROOTMERIT task)

spacecraft data

same as P300P301-ROOTMERIT

Input Run List

ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300xP301-FITS/config/runList.txt

evtClassDefsuntagged version from 20130801

01-00-01

eventClassMap

EvtClassDefs_P8P8V1.xml

ScienceTools

09-3233-05 01 (75/3015/20132014)

Code Variants

redhat5-x86_64-64bit-gcc41 & redhat6-x86_64-64bit-gcc44   (Optimized)

Diffuse Model

based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 /v3r0
(see https://confluence.slac.stanford.edu/display/SCIGRPS/Quick+Start+with+Pass+7)

N/A

Diffuse Response

N/A

IRFs

N/A

Output Data Products

EXTENDEDFT1

...

, EXTENDEDLS1

commonTools00-02-01

Generation of output data products:

Data Product

destination

data content [1]

event selection [1]

makeFT1

gtselect

gtdiffrsp

gtmktime

EXTENDEDFT1

SLAC

pass8_FT1variables

((FT1EventClass & 0x00003EFF)!=0)
pass7.6_Extendedpass8.1_Transient_cuts_L1

(tick)

(error)

(tick)

(tick)

FT1

FSSC+SLAC

FT1variables

'source' and above
EVENT_CLASS bits 2,3,4 evclass=2 filtered from EXTENDEDFT1

(error)

(tick)

(inherited)

(tick)

EXTENDEDLS1

SLAC

pass8_LS1variables

((FT1EventClass & 0x00003EFF)!=0)
pass7.6_Extended_cuts_L1

(tick)

(error)

(tick)

(tick)

LS1

FSSC+SLAC

LS1variables

'transient' and above
EVENT_CLASS bits 0,2,3,4
evclass=0 filtered from EXTENDEDLS1

(error)

(tick)

(inherited)

(tick)

ELECTRONFT1

SLAC

FT1variables

CTBParticleType==1
pass7.6_Electronspass8.1_Transient_cuts_L1

(tick)

(error)

(error)

(tick)

[1] /afs/slac/g/glast/ground/releases/volume04/evtClassDefs/01-00-19-0401/data

Note that diffuse response is calculated for 'source' and 'clean' event classes only.Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL5-32 (RHEL6-64 are built but not yet validated), while ScienceTools builds for RHEL5-64, RHEL5RHEL6-64.

Timing and Scaling

(no data)

 

...

This task is a clone of the P300-ROOT task with the exception of the run list which contains the 264 L&EO runs.

12/10/2013 09:10 begin reprocessing

12/17/2013 Processing complete.  Each of the RECON, MERIT, GCR and RELATION files contain 613,490,351 events.

 

P300x-LEO-FITS

This is a clone of the P300x-FITS task with the exception of the run list which contains the 264 L&EO runs.

12/17/2013 begin processingApprox 20 CPU minutes and 30 wall clock minutes per job (run).