Page History

status: Running
last update: 5 April 2012

...

Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf^{Image Removed}

Examples:

/glast/Data/Flight/Reprocess/P200/recon/r0239557414_v202_recon.root
/glast/Data/Flight/Reprocess/P200/cal/r0239557414_v202_cal.root
/glast/Data/Flight/Reprocess/P200/gcr/r0239557414_v202_gcr.root
/glast/Data/Flight/Reprocess/P200/merit/r0239557414_v202_merit.root
/glast/Data/Flight/Reprocess/P200/filteredmerit/r0239557414_v202_filteredmerit.root
/glast/Data/Flight/Reprocess/P200/electronmerit/r0239557414_v202_electronmerit.root
/glast/Data/Flight/Reprocess/P200/extendedft1/gll_xp_p202_r0239559565_v202.fit
/glast/Data/Flight/Reprocess/P200/ft1/gll_ph_p202_r0239559565_v202.fit
/glast/Data/Flight/Reprocess/P200/electronft1/gll_el_p202_r0239559565_v202.fit
/glast/Data/Flight/Reprocess/P200/extendedls1/gll_xe_p202_r0239559565_v202.fit
/glast/Data/Flight/Reprocess/P200/ls1/gll_ev_p202_r0239559565_v202.fit

...

1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P202^{Image Removed}
   1. List of all reprocessings
   2. List of all data runs reprocessed
   3. Pointers to all input data files (-> dataCatalog)
   4. Pointers to associated task processes (-> Pipeline II status)
3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp^{Image Removed}
   1. Lists of and pointers to all output data files
   2. Meta data associated with each output data product

...

• 4/5/2012 - resume task. New goal is entire science dataset.
• 4/12/2012 - due to possible overload of sulky46/u18 writing a lot of core files, have introduced one change to processClumps.py: prepend "ulimit -c 0;" to gleam command to disable all core file generation. This starts approx with run 7605 (+/-).
• 5/9/2012 - major pipeline issue...shut down pipeline and allow to drain (due to tomorrow's major outage)
• 5/10/2012 - 13:40 outage over.
  • Update GR from 17-35-24-rp04 to 17-35-24-rp07 in which the only change is replacing the 5-minute xroot time-out with 8 hours. This change effective with stream 14314 and previously failed pieces of four other runs: 14247.6, 14273.23, 14274.8, 14231.9.
  • Leon advises that as of today, calibrations are valid only thru ~15 Dec 2011 (run 345574915) - which is somewhere around stream 18,400. He asks Sasha to produce more up-to-date calibs.

Configuration

• 5/28/2012 - 15:30 Complete (through 31 March 2012)
  • Data Catalog summary:  

    Name	Type	Files	Events	Size	Created	(UTC)Links
    CAL	Group	20229	44,125,599,595	128.7 TB	25-Jan-2012 00:53:31	Files
    ELECTRONFT1	Group	5600	0	2.5 GB	02-Mar-2012 00:06:07	Files
    ELECTRONMERIT	Group	20229	90,904,582	205.7 GB	25-Jan-2012 00:53:32	Files
    EXTENDEDFT1	Group	5600	1,572,783,826	143.7 GB	02-Mar-2012 00:06:09	Files
    EXTENDEDLS1	Group	5600	1,572,783,826	255.0 GB	02-Mar-2012 00:06:09	Files
    FILTEREDMERIT	Group	20229	6,291,396,710	5.3 TB	25-Jan-2012 00:53:29	Files
    FT1	Group	5600	24,261,962	2.4 GB	02-Mar-2012 00:06:06	Files
    GCR	Group	20229	44,123,014,456	942.7 GB	25-Jan-2012 00:53:31	Files
    LS1	Group	5600	271,923,333	44.2 GB	02-Mar-2012 00:06:08	Files
    MERIT	Group	20229	44,125,679,961	35.4 TB	25-Jan-2012 00:53:30	Files
    RECON	Group	20229	44,123,612,977	590.0 TB	25-Jan-2012 00:53:33	Files

    There are discrepancies to track down!
  • Final trickleStream configuration:

    Code Block

    =============================================================================== TRICKLE PARMS =============================================================================== task = P202-ROOT maxRuns = 20229 firstStep = setupRun steps = [['/processRun processClump', 2000, 21], ['mergeClumps', 200, 1]] maxStreamsPerCycle = 20 timePerCycle = 300 ------DEBUG---------------- maxCycles = 0 chatter = False dryRun = False ===============================================================================

Configuration

Task Location	/nfs/farm/g/glast/u38/Reprocess-tasks/P202-ROOT
Task Status	http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp
GlastRelease
Task Location	/nfs/farm/g/glast/u38/Reprocess-tasks/P202-ROOT
Task Status	http://glast-ground.slac.stanford.edu/Pipeline-II/index.jspImage Removed
GlastRelease	17-35-24-gr17 (SCons RHEL4-32 build)
Run Selection	based on a modified "standard" selection, see https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets (((sIntent=="nomSciOps" || sIntent=="nomSO_noSk_noCno_optGccc_allEna" || sIntent=="nomSciOps_diagEna" || (sIntent=="nomSciOps_Emin5MeV"&&RunMin>242070455) || nRun==242429468 ) && (RunQuality != "Bad" || is_null ( RunQuality ) ) ) || sIntent=="nadirOps" )
s/c data	"standard" Public Release 2 https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets
Input Run List	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-ROOT/config/runList.txt Image Removed
photonFilter	CTBParticleType==1 && ((FT1EventClass & 0x00003EFF)!=0) pass7.6_Extended_cuts_L1 in evtClassDefs
electronFilter	CTBParticleType==1
Code Variants	redhat4-i686-32bit-gcc34 (Optimized)
jobOpts	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-ROOT/config/doRecon.txt Image Removed
Output Data Products	RECON, GCR, CAL, MERIT, FILTEREDMERIT, ELECTRONMERIT

...

Task Location	/nfs/farm/g/glast/u38/Reprocess-tasks/P202-FITS
Task Status	http://glast-ground.slac.stanford.edu/Pipeline-II/task.jsp?task=75031156 Image Removed
Input Data	MERIT (direct from P202-ROOT)
spacecraft data	same as P202-ROOT
Input Run List	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-FITS/config/runList.txt Image Removed
evtClassDefs	00-19-04 (6/23/2011)
eventClassMap	EvtClassDefs_P7V6.xml
ScienceTools	09-27-01 (2/15/2012)
Code Variants	redhat5-i686-32bit-gcc41 (Optimized)
Diffuse Model	based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 (see https://confluence.slac.stanford.edu/display/SCIGRPS/Quick+Start+with+Pass+7 Image Removed )
Diffuse Response	'source' using P7SOURCE_V6 IRF 'clean' using P7CLEAN_V6 IRF
IRFs	P6V7, contained within ScienceTools release
Output Data Products	FT1, LS1, EXTENDEDFT1, EXTENDEDLS1, ELECTRONFT1

...

...

Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL4-32, while ScienceTools builds for RHEL5-32, RHEL5-64.

Timing