Versions Compared

Old Version 117

changes.mady.by.user Tom Glanzman

Saved on

compared with

New Version 118

changes.mady.by.user Tom Glanzman

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 4.0

...

Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdfImage Removed

Example:

Code Block
/glast/Data/Flight/Reprocess/P120/merit/r0239557414_v120_merit.root
/glast/Data/Flight/Reprocess/P120/filteredmerit/r0239557414_v120_filteredmerit.root
/glast/Data/Flight/Reprocess/P120/electronmerit/r0239557414_v120_electronmerit.root
/glast/Data/Flight/Reprocess/P120/extendedft1/gll_xp_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/ft1/gll_ph_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/electronft1/gll_el_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/extendedls1/gll_xe_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/ls1/gll_ev_p120_r0239559565_v120.fit

...

  1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
  2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P120Image Removed
    1. List of all reprocessings
    2. List of all data runs reprocessed
    3. Pointers to all input data files (-> dataCatalog)
    4. Pointers to associated task processes (-> Pipeline II status)
  3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jspImage Removed
    1. Lists of and pointers to all output data files
    2. Meta data associated with each output data product

...

  • 8/17/2011 - Begin reprocessing four orphan runs.
  • 8/1/2011 - Begin and complete final backfill through run 333880535 (2011-08-01 08:35:33 UTC). For now, the three nadirOps runs are represented by dummy place-holder entries in the runFile.txt – their pipeline streams will fail.
  • 7/29/2011 - Modified selection criteria for reprocessing run selection (findRunsRepro.py) to allow nadir-pointed data. This means adding ' || sIntent=="nadirOps"' to the dataCatalog selection string. See https://confluence.slac.stanford.edu/display/ISOC/Nadir+Obs+Test+-+26+July+2011Image Removed for a list of runs affected by the nadir-pointed test.
  • 7/26/2011 - Recovered three missing runs (below).
  • 7/21/2011 - Begin and (mostly) complete reprocessing block 13, through run 332930755 (2011-07-21 08:45:53 UTC), 528 new runs - special GRB request. Note three runs failed skim/merge and are being investigated:

    stream

    run

    UTC

    Crash location

    16136

    332054399

    2011-07-11 05:19:57

    Filtered Merit skim failure

    16151

    332140182

    2011-07-12 05:09:40

    Electron Merit skim failure

    16156

    332169056

    2011-07-12 13:10:54

    Filtered Merit skim failure

  • 6/17/2011 - Begin and complete reprocessing block 11, through run 329923889 (2011-06-16 13:31:27 UTC), 889 new runs.
  • 4/19/2011 - Begin and complete reprocessing block 10, through run 324849509 (2011-04-18 19:58:27 UTC), 52 new runs.
  • 4/15/2011 - Begin and complete reprocessing block 9, through run 324551768 (2011-04-15 10:51:27 UTC)
  • 4/13/2011 - Begin and complete reprocessing block 8, through run 324368491 (2011-04-13 06:21:29 UTC)
  • 4/3/2011 - The three missing runs have been produced by Level 1. Runlist recreated and those runs rolled back. There are no missing runs at this point.
  • 3/29/2011 - Due to some hidden I/O problems, two changes have been made to this task, neither of which should affect data content. Note that this change will take effect on runs after 321756673, or any runs rolled back after this date.

    Package

    Old Version

    New Version

    Reason for change

    GPLtools

    v1r15p1

    GPLtools-02-00-00

    Checks size of files before and after move between disk and xroot

    skimmer

    07-07-00

    08-01-00

    detects failure to open input file

  • 3/17/2011 - Last block complete (one instance of skimmer failure in mergeClumps)
  • 3/15/2011 - Expand P120 to present (last run 321756673, 2011-03-14 00:51:11 UTC). Three runs continue to be dummied out. Once those runs have proper RECON files, their streams can be rolled back.

    run

    task stream

    # subStreams

    306353950

    11624

    10

    316611240

    13431

    8

    320850543

    14171

    7

  • 2/4/2011 - Expand P120 to present (last run 318211122, 2011-01-31 23:58:40 UTC). This include three runs for which there are no Recon files which have, for the moment, been supplied with 'dummy' entries in the runFile.:

    306353950

    'bad chunk' (known previously) - awaiting GR update

    313483912

    missing 700s, being worked on

    316611240

    'bad chunk' - awaiting GR update

  • 1/12/2011 - Expand P120 coverage to include Crab ToO
    • Crab ToO 9/23/2010 15:50:50 to 9/27/2010 19:49:38, corresponding to MET r0306949696-r0307308940
    • reconfig through end of Sep 2010: 10916 -> 11841 runs, increase of 925 runs
    • No recon file for run 306353950 (being worked on, for the moment, place dummy file in runFile.txt)
    • First run in new block completed, awaiting checkered flag to continue...
  • 8/29/2010 - Discovered three merge steps that silently failed (xroot file access). TASK complete.
  • 8/28/2010 - processing formally complete (10916 runs), but some discrepancy in # of events
  • 8/26/2010 - serious xroot problems. See initial distribution of files across xroot servers. From this report (courtesy Wilko) it is easy to see where problems are likely to arise - when the number of servers involved is small, e.g. two or three.
  • 8/19/2010 - production continues at a crawl due to xroot server difficulties
  • 8/16/2010 - resume full production, but at a slow trickle (max 350 simultaneous processClump jobs)
  • 8/8/2010 - block 2 reprocessing complete. Many xroot server problems. (5 days to process 2084 runs)
  • 8/3/2010 - begin block 2 reprocessing (through 255132033 MET), bringing the total runs reprocessed to 2721, about 5-1/2 months of data.
  • 7/28/2010 - block 1 re-reprocessing complete
  • 7/27/2010 - New GlastRelease (v17r35p10) containing new evtUtils, "to make the FT1EventClass bits compatible with the ScienceTools". Cleanup, including removing all files created last week during the first attempt.
  • 7/21/2010 - block 1 reprocessing complete
  • 7/20/2010 - agree upon 'pilot block' of runs (239557417 - 243220241), 637 runs. Begin...
  • 7/19/2010 - submit first test run. success. await feedback

...

Task Location

/nfs/farm/g/glast/u38/Reprocess-tasks/P120-MERIT

Task Status

http://glast-ground.slac.stanford.edu/Pipeline-II/exp/Fermi/task.jsp?task=41146114Image Removed

GlastRelease

v17r35p8 v17r35p10

Input Data Selection

"standard" from

https://confluence.slac.stanford.edu/display/SCIGRPS/LAT+Dataset+DefinitionsImage Removed

along with "&& (RunQuality != "Bad" || is_null ( RunQuality )"

s/c data

FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production

Input Run List

ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P120-MERIT/config/runFile.txtImage Removed

photonFilter

CTBParticleType==0 && CTBClassLevel>0

electronFilter

CTBParticleType==1

jobOpts

ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P120-MERIT/config/reClassify.txtImage Removed

Output Data Products

MERIT, FILTEREDMERIT, ELECTRONMERIT

...

This task generates all desired FITS data products. An example of the code processing chain appears on a child page.

Status chronology

  • 8/1/2011 - Begin and complete final block of Pass7 reprocessing
  • 7/26/2011 - Recovered the three missing runs (see P120-MERIT chronology), and reran stream 1018 (run 245403855), which had a bogus tstart time in the datacatalog – leap second issue, and recovered 27 events in that run.
  • 7/22/2011 - Begin and (mostly) reprocessing block 13, through run 332930755 (2011-07-21 08:45:53 UTC), 525 new runs (+ 3 'dummy' runs due to skim crashes, see above) - special GRB request.
  • 7/3/2011 - Task complete through run 329923889 (2011-06-16 13:31:27 UTC), 15,763 runs
  • 6/30/2011 - New ST 09-24-00 (with gtdiffrsp fix), restart trials with task version 2.1
  • 6/14/2011 - Begin trials. Concern that gtdiffrsp is crashing often (20-25% of time)
  • 6/7/2011 - IMPORTANT UPDATE: a decision was made to rollback entire task with these changes:
    • calculate diffuse response for 'source' and 'clean' event classes
    • Produce new subset photon files for FSSC (FT1 with source and above, LS1 with transient and above)
    • Update various configurations (ScienceTools, evtClassDefs, etc.)
      This is being done by creating a whole new task, version 2.0, which from the pipeline perspective will overlay the older version 1.0.
  • 4/19/2011 - Begin and complete reprocessing block 10, through run 324849509 (2011-04-18 19:58:27 UTC), 52 new runs.
  • 4/15/2011 - Begin and complete reprocessing block 8, through run 324551768 (2011-04-15 10:51:27 UTC)
  • 4/14/2011 - Begin and complete reprocessing block 7, through run 324368491 (2011-04-13 06:21:29 UTC)
  • 4/3/2011 - The three missing runs have now been reprocessed. There are no missing runs at this point.
  • 3/17/2011 - Catch up with P120-MERIT (last run 321756673, 2011-03-14 00:51:11 UTC)
  • 3/15/2011 - Due to missing run, rollback runs 11624-11841. Bookkeeping is now correct.
  • 2/4/2011 - Catch up with merit production (through 30 Sep 2010), but with one missing run/stream
  • 1/28/2011 - Pass 7.4 reincarnation of this task complete through 31 Jul 2010
  • 1/24/2011 - Entire task, xroot files, dataCat entries deleted. Prepare to reprocess as Pass 7.4
  • 8/31/2010 - Pass 1 of this task is complete (through 31 July 2010)
  • 8/30/2010 - Problem with makeFT1 stressing /u38 (very large temporary file needed when using xml representation of event classes was being written to $PWD). Jim makes update to fitsGenApps => ST 09-18-03, put into production at stream 1400.
  • 8/29/2010 - Begin Pass 1 of task...

...

Task Location

/nfs/farm/g/glast/u38/Reprocess-tasks/P120-FT1

Task Status

http://glast-ground.slac.stanford.edu/Pipeline-II/exp/Fermi/task.jsp?task=65047878Image Removed

Input Data

MERIT (from P120-MERIT)

spacecraft data

FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production

Input Run List

ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P120-FT1/config/runFile.txtImage Removed

Reprocessing Mode

reFT1

meritFilter

FT1EventClass!=0

evtClassDefs

00-19-01

eventClassMap

EvtClassDefs_P7V6.xml

ScienceTools

09-24-00

Code Variants

redhat5-i686-32bit-gcc41 (Optimized)

Diffuse Model

based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0
(see

https://confluence.slac.stanford.edu/display/SCIGRPS/Quick+Start+with+Pass+7Image Removed

)

Diffuse Response

'source' using P7SOURCE_V6 IRF
'clean' using P7CLEAN_V6 IRF

IRFs

P6V7, contained within ScienceTools release

Output Data Products

FT1, LS1, EXTENDEDFT1, EXTENDEDLS1, ELECTRONFT1

...