status: Done

 last update: 25 September 2017

This page is a record of the configuration and execution of the P304 reprocessing project, generation FITS files from the P301 MERIT files.

Notice: there is now a JIRA for this task.  Please check here for operational notes.

Pipeline tasks

• P304-FITS - This task reads new P301 MERIT and produces fits files, FT1 + LS1 + EXTENDEDFT1 + EXTENDEDLS1

Datafile names, versions and locations

Data file version numbers for this reprocessing will begin with v304.

XROOT location and file naming

Location template:

/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>

Locations for P304:

/glast/Data/Flight/Reprocess/P304/ft1
/glast/Data/Flight/Reprocess/P304/extendedft1
/glast/Data/Flight/Reprocess/P304/ls1
/glast/Data/Flight/Reprocess/P304/extendedls1

File naming:

Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf

Example:

root://glast-rdr.slac.stanford.edu//glast/Data/Flight/Reprocess/P304/ft1/gll_ph_p304_r0246823875_v304.fit

DataCatalog location and naming

Logical directory and group template:

Data/Flight/Reprocess/<reprocessName>:<dataType>

Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.

Naming examples:

/Data/Flight/Reprocess/P304 Group FT1

Data Sample

The currently defined data sample (as of ) for P304 reprocessing includes:

Bookkeeping

1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P304
   1. List of all reprocessings
   2. List of all data runs reprocessed
   3. Pointers to all input data files (-> dataCatalog)
   4. Pointers to associated task processes (-> Pipeline II status)
3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
   1. Lists of and pointers to all output data files
   2. Meta data associated with each output data product

P304-FITS

This task generates all desired FITS data products.

• A slightly updated task structure has been implemented for this task: the mergeClumps step is now a separate subTask which allows overriding the batch queue parameter for extra long runs.  This restructuring required quite a lot of modification related to data passed between job steps (as now certain job steps are in the new subTask). 
• Task stored in my github as P304-FITS

Status chronology

This task basically mirrors the P301-FITS task but updating certain packages and expanding the generated data products to include FT1 and LS1.

Please see the associated JIRA for operational notes.

• 2017-09-18: start year 1 production
  
  

  Name	Files	Events	Size	Created (UTC)
  EXTENDEDFT1
  EXTENDEDLS1
  FT1
  LS1

Configuration

Generation of output data products:

Data Product	destination	data content [1]	event selection [1]	makeFT1	gtselect	gtdiffrsp	gtmktime	Notes
EXTENDEDFT1	SLAC	pass8.2_Extended_cuts_L1	pass8.2_Transient_cuts_L1
FT1	FSSC	pass8_FT1variables	evclass=128					produced from extendedFT1
EXTENDEDLS1	SLAC	pass8.2_Extended_cuts_L1	pass8.2_Transient_cuts_L1
LS1	FSSC	pass8_LS1variables	evclass=65544					produced from extendedLS1

[1] /afs/slac/g/glast/ground/releases/volume04/evtClassDefs/01-01-04/data

Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL5-32 (RHEL6-64 are built but not yet validated), while ScienceTools builds for RHEL5-64, RHEL6-64.

Timing and Scaling

Approx 20 CPU minutes and 30 wall clock minutes per job (run).