Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

NOTE: One run, 242429468, of type TrigTest was declared 'good for science' but long after this task got started, so it and has been intentionally omittedincluded.

Wiki Markup
\[to be continued...\]

Bookkeeping

  1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
  2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P202Image Removed
    1. List of all reprocessings
    2. List of all data runs reprocessed
    3. Pointers to all input data files (-> dataCatalog)
    4. Pointers to associated task processes (-> Pipeline II status)
  3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jspImage Removed
    1. Lists of and pointers to all output data files
    2. Meta data associated with each output data product

...

Progress at the 1-year mark:

First run

239557414 (MET), 2008-08-04 15:43:34 (UTC)

 

Last run

271999199 (MET), 2009-08-15 03:19:57 (UTC)

 

Total runs

5600

 

Total input DIGI events

11,928,911,465

 

Total RECON events

11,928,911,465

161.4 TB

Total CAL events

11,928,911,465

36.1 TB

Total GCR events

11,928,911,465

260.5 GB

Total MERIT events

11,928,911,465

9.6 TB

all triggered events

Total FILTEREDMERIT events

1,572,783,868

1.3 TB

all photon event classes

Total EXTENDEDFT1

1,572,783,826

143.7 GB

all photon event classes

Total LS1 events

1,572,783,826

255.0 GB

all photon event classes

Total LS1 (FSSC selection) events

271,923,333

44.2 GB

event classes (bits) 0,2,3,4 (transient, source, clean, ultraclean)

Total FT1 (FSSC selection) events

24,261,962

2.4 GB

event classes (bits) 2,3,4 (source, clean, ultraclean)

Wiki Markup
\[to be continued...\]

Bookkeeping

  1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
  2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P202Image Added
    1. List of all reprocessings
    2. List of all data runs reprocessed
    3. Pointers to all input data files (-> dataCatalog)
    4. Pointers to associated task processes (-> Pipeline II status)
  3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jspImage Added
    1. Lists of and pointers to all output data files
    2. Meta data associated with each output data product

...

P202-ROOT
Anchor
P202-ROOT
P202-ROOT

Status chronology

  • 2/13/2012 - begin trials with final calibration and alignments from Leon; 5 runs reprocessed
  • 2/14/2012 - trials continue with blocks of 15, 20, 25 and 50 runs reprocessed (each run generates ~20 batch jobs)
  • 2/16/2012 - begin trickleStream production. Initial config:
    Code Block
    
    

...

Status chronology

  • 2/13/2012 - begin trials with final calibration and alignments from Leon; 5 runs reprocessed
  • 2/14/2012 - trials continue with blocks of 15, 20, 25 and 50 runs reprocessed (each run generates ~20 batch jobs)
  • 2/16/2012 - begin trickleStream production. Initial config:
    Code Block
    
    ==========================================================================================
      TRICKLE PARMS
    ===============================================================================
    task =  P202-ROOT
    maxRuns =  19172
    firstStep =  setupRun
    steps =  [['/processRun processClump', 1500, 20], ['mergeClumps', 70, 1]]
    maxStreamsPerCycle =  20
    timePerCycle =  300
    ===============================================================================
    
  • 2/21/2012 - One clump reprocessed with pointer to new mySQL DB (stream 710.0)
  • 2/22/2012 - 776 runs complete. Pausing task.
  • 3/15/2012 - resume task. New goal is 1-year of data (~5600 runs)
  • 3/31/2012 - 1-year complete (5600 runs). DataCatalog summary:There have been a few nasty problems which need to be fixed before continuing.

S/W component

bug fix

New ROOT version

5-min 'transaction timeout' triggered by xroot data server reboot

New GlastRelease

1) include new ROOT version (above); 2) exit with non-zero RC on ROOT write error

New GPL_TOOLS

check size/checksum of file written to xroot with known size/checksum

Tuned xroot on new Dell servers

silent file truncation when volume fills up JIRA

New TSkim

1) new ROOT version (above); 2) complain on ROOT write errors

Name

Files

Events

Size

FILTEREDMERIT

5600

1,572,783,868

1.3 TB

MERIT

5600

11,928,911,465

9.6 TB

CAL

5600

11,928,911,465

36.1 TB

GCR

5600

11,928,911,465

260.5 GB

RECON

5600

11,928,911,465

161.4 TB

Configuration

Task Location

/nfs/farm/g/glast/u38/Reprocess-tasks/P202-ROOT

Task Status

http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp

GlastRelease

17-35-24-gr17 (SCons RHEL4-32 build)

Run Selection

based on a modified "standard" selection, see https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets
(((sIntent=="nomSciOps" || sIntent=="nomSO_noSk_noCno_optGccc_allEna" || sIntent=="nomSciOps_diagEna" || (sIntent=="nomSciOps_Emin5MeV"&&RunMin>242070455) || nRun==242429468 ) && (RunQuality != "Bad" || is_null ( RunQuality ) ) ) || sIntent=="nadirOps" )

s/c data

"standard" Public Release 2 https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets

Input Run List

ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-ROOT/config/runList.txt

photonFilter

CTBParticleType==1 && ((FT1EventClass & 0x00003EFF)!=0)
pass7.6_Extended_cuts_L1 in evtClassDefs

electronFilter

CTBParticleType==1

Code Variants

redhat4-i686-32bit-gcc34 (Optimized)

jobOpts

ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-ROOT/config/doRecon.txt

Output Data Products

RECON, GCR, CAL, MERIT, FILTEREDMERIT, ELECTRONMERIT

...

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="6c5210c185bd597b-28adcbcb-47b94271-8127996e-4274148a05144c39bd71837d"><ac:plain-text-body><![CDATA[

Data Product

destination

data content [1]

event selection [1]

makeFT1

gtselect

gtdiffrsp

gtmktime

]]></ac:plain-text-body></ac:structured-macro>

EXTENDEDFT1

SLAC

FT1variables

((FT1EventClass & 0x00003EFF)!=0)
pass7.6_Extended_cuts_L1

(tick)

(error)

(tick)

(tick)

FT1

FSSC+SLAC

FT1variables

'source' and above
EVENT_CLASS bits 2,3,4
evclass=2 filtered from EXTENDEDFT1

(error)

(tick)

(inherited)

(tick)

EXTENDEDLS1

SLAC

LS1variables

((FT1EventClass & 0x00003EFF)!=0)
pass7.6_Extended_cuts_L1

(tick)

(error)

(tick)

(tick)

LS1

FSSC+SLAC

LS1variables

'transient' and above
EVENT_CLASS bits 0,2,3,4
evclass=0 filtered from EXTENDEDLS1

(error)

(tick)

(inherited)

(tick)

ELECTRONFT1

SLAC

FT1variables

CTBParticleType==1
pass7.6_Electrons_cuts_L1

(tick)

(error)

(error)

(tick)

...