P120 Reprocessing

status: Paused Under Construction
last update: 09 14 June 2011

This page is a record of the configuration for the P120 reprocessing project, event reclassification using Pass 7.3/7.4/7.6. This project involves reprocessing with Pass7 classification trees and (ultimately) new IRFs. The name "P120" derives from the word "processing" and the initial file version to be used for the output data products, e.g., r0123456789_v120_merit.root.

...

/glast/Data/Flight/Reprocess/P120/merit
/glast/Data/Flight/Reprocess/P120/filteredmerit
/glast/Data/Flight/Reprocess/P120/electronmerit
/glast/Data/Flight/Reprocess/P120/ft1
/glast/Data/Flight/Reprocess/P120/extendedft1
/glast/Data/Flight/Reprocess/P120/electronft1
/glast/Data/Flight/Reprocess/P120/ls1
/glast/Data/Flight/Reprocess/P120/extendedls1

File naming:

Note: Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf

...

/glast/Data/Flight/Reprocess/P120/merit/r0239557414_v120_merit.root
/glast/Data/Flight/Reprocess/P120/filteredmerit/r0239557414_v120_filteredmerit.root
/glast/Data/Flight/Reprocess/P120/electronmerit/r0239557414_v120_electronmerit.root
/glast/Data/Flight/Reprocess/P120/extendedft1/gll_xp_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/ft1/gll_ph_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/electronft1/gll_el_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/extendedls1/gll_xe_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/ls1/gll_ev_p120_r0239559565_v120.fit

...

Data/Flight/Reprocess/P120:MERIT r0239557414
Data/Flight/Reprocess/P120:FILTEREDMERIT r0239557414
Data/Flight/Reprocess/P120:EXTENDEDFT1 r0239557414
Data/Flight/Reprocess/P120:FT1 r0239557414
Data/Flight/Reprocess/P120:ELECTRONFT1 r0239557414
Data/Flight/Reprocess/P120:EXTENDEDLS1 r0239557414
Data/Flight/Reprocess/P120:LS1 r0239557414

...

The currently defined data sample for P120 reprocessing includes:P120 reprocessing includes:

\[OBSOLETE/INCOMPLETE DATA\]

...

P120-FT1

Anchor
P120-FT1 P120-FT1

This task will be run twice: Pass 1 will perform event classification for source and transient events and allow analysis to produce diffuse class IRFs; Pass 2 will be identical to Pass 1 but will include diffuse classification. The latest word from C&A is that diffuse response will only be calculated for 'source' class eventsThis task generates all desired FITS data products.

Status chronology

• 6/14/2011 - Begin trials. Concern that gtdiffrsp is crashing often (20-25% of time)
• 6/7/2011 - IMPORTANT UPDATE: a decision was made to rollback entire task with these changes:
  • calculate diffuse response for 'source' and 'clean' event classes
  • Produce new subset photon files for FSSC (FT1 with source and above, LS1 with transient and above)
  • Update various configurations (ScienceTools, evtClassDefs, etc.)
• 4/19/2011 - Begin and complete reprocessing block 10, through run 324849509 (2011-04-18 19:58:27 UTC), 52 new runs.
• 4/15/2011 - Begin and complete reprocessing block 8, through run 324551768 (2011-04-15 10:51:27 UTC)
• 4/14/2011 - Begin and complete reprocessing block 7, through run 324368491 (2011-04-13 06:21:29 UTC)
• 4/3/2011 - The three missing runs have now been reprocessed. There are no missing runs at this point.
• 3/17/2011 - Catch up with P120-MERIT (last run 321756673, 2011-03-14 00:51:11 UTC)
• 3/15/2011 - Due to missing run, rollback runs 11624-11841. Bookkeeping is now correct.
• 2/4/2011 - Catch up with merit production (through 30 Sep 2010), but with one missing run/stream
• 1/28/2011 - Pass 7.4 reincarnation of this task complete through 31 Jul 2010
• 1/24/2011 - Entire task, xroot files, dataCat entries deleted. Prepare to reprocess as Pass 7.4
• 8/31/2010 - Pass 1 of this task is complete (through 31 July 2010)
• 8/30/2010 - Problem with makeFT1 stressing /u38 (very large temporary file needed when using xml representation of event classes was being written to $PWD). Jim makes update to fitsGenApps => ST 09-18-03, put into production at stream 1400.
• 8/29/2010 - Begin Pass 1 of task...

...

Task Location	/nfs/farm/g/glast/u38/Reprocess-tasks/P120-FT1
Task Status	http://glast-ground.slac.stanford.edu/Pipeline-II/indextask.jsp?task=63390236Image Modified
Input Data Selection	MERIT (from P120-MERIT) from P120-MERIT)
spacecraft data	FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production
Input Run List	ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P120-FT1/config/runFile.txt
Reprocessing Mode	reFT1
meritFilter	FT1EventClass!=0
evtClassDefs	00-19-01
eventClassMap	EvtClassDefs_P7V6.xmlspacecraft data
FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production	ScienceTools	09-23-01
Code Variants	redhat4-i686-32bit-gcc34 and redhat5-i686-32bit-gcc41 (Optimized)
Diffuse Model	based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 (see https://confluence.slac.stanford.edu/display/SCIGRPS/Quick+Start+with+Pass+7 )
Diffuse Response	'source' using P7SOURCE_V6 IRF 'clean' using P7CLEAN_V6 IRF
IRFs	P6V7, contained within ScienceTools release
Output Data Products	FT1, LS1, FT1EXTENDEDEXTENDEDFT1, LS1EXTENDEDEXTENDEDLS1, ELECTRONFT1

Processing chain for FITS data products

...

Note that diffuse response is calculated for 'source' and 'clean' event classes only.

Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RedHat Enterprise Linux 4, and 5, gcc 3.4, 4RHEL5-64, gcc v4.1, etc.).

Timing

• 1/28/2011 - Without diffuse response, the mergeClumps jobs are taking about 10 hequ-minutes of CPU to complete.
• 8/31/2010 - The primary batch job, mergeClumps, took a (mean) time of 42 cpu minutes (primarily a mixture of hequ and fell class machines). With P120-MERIT files nicely distributed across xroot servers, there were no xroot limitations to the processing. After the update to makeFT1, there was no longer an issue with overloading /u38 ($PWD). The next bottleneck was the pipeline processing itself. This task consists of three batch jobs and four scriptlets; it was observed that the pipeline allowed hundreds of jobs to dwell in the READY state for extended periods of time, thus making it impossible to keep LSF saturated. Nevertheless, the maximum number of simultaneous jobs approached 2000. The task essentially completed in 8 hours, although some lingerers kept 'running' for another nine hours (mostly in SSUSP). A profile of job processing rate appears in this plot:

...