You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

P116 Reprocessing

status: Under Construction
last update: 14 October 2010

This page is a record of the configuration for the P116 reprocessing project, rebuilding FT1 files (and friends) for all of science data with new, extended event classifications.

  • P116-FT1 - this task will read existing (P105+level1) MERIT files and produce FT1 (photons) + LS1 ("fat" FT1) files

Datafile names, versions and locations

Data file version numbers for this reprocessing will begin with v116.

XROOT location and file naming

Location template:

/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>

Locations for P120:

/glast/Data/Flight/Reprocess/P116/ft1
/glast/Data/Flight/Reprocess/P116/electronft1
/glast/Data/Flight/Reprocess/P116/ls1

File naming:

Data Type

aka

Send to FSSC

Naming template

FT1

LS-002

Yes

gll_php<procVer>r<run#>_<version>.fit

LS1

LS-001

Yes

gll_evp<procVer>r<run#>_<version>.fit

Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_ICD_RevA.pdf

Example:

/glast/Data/Flight/Reprocess/P120/ft1/gll_ph_p116_r0239559565_v116.fit
/glast/Data/Flight/Reprocess/P120/ls1/gll_ev_p116_r0239559565_v116.fit
DataCatalog location and naming

Logical directory and group template:

Data/Flight/Reprocess/<reprocessName>:<dataType>

Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.

Naming examples:

Data/Flight/Reprocess/P116:FT1 r0239557414
Data/Flight/Reprocess/P116:LS1 r0239557414

Data Sample

The currently defined data sample for P116 reprocessing includes:

First run

239557414 (MET), 2008-08-04 15:43:34 (UTC)

Last run

307584002 (MET), 2010-10-01 00:00:00 (UTC)

Total runs

11844

Total MERIT events

25,815,656,695

Note that the merit files used for input to this reprocessing derive from two sources:

MET start

MET end

DataCatalog Source

Note

239557414

242047683

/Data/Test/Flight/Repro/ReproTest7

431 runs: reprocessed for alignment fix

242053458

307584002

/Data/Flight/Level1/LPA/

11413 runs: Standard Level 1 output

It is expected that a backfill operation will happen to mesh with Level 1 once it begins to produce these new event classifications. Ultimately, these data are destined for the FSSC.

Bookkeeping

  1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
  2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P116
    1. List of all reprocessings
    2. List of all data runs reprocessed
    3. Pointers to all input data files (-> dataCatalog)
    4. Pointers to associated task processes (-> Pipeline II status)
  3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
    1. Lists of and pointers to all output data files
    2. Meta data associated with each output data product

P116-FT1

This task is roughly equivalent to the P105-FT1 task but with modernization.

Status chronology

  • 10/1/2010 - Begin task construction
  • 10/14/2010 - Validation complete, begin production

Configuration

Task Location

/nfs/farm/g/glast/u38/Reprocess-tasks/P116-FT1

Task Status

http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp

Input Data Selection

MERIT

Input Run List

 ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P116-FT1/config/runFile.txt

Reprocessing Mode

reFT1

evtClassDefs

00-18-00

photon cut

pass6_FSW_cuts

eventClassifier

dataclean_classifier.py

eventClassMap

n/a

s/c data

FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production

ScienceTools

09-18-05 (SCons build)

Allowed Code Variants

redhat4-i686-32bit-gcc34, redhat4-x86_64-64bit-gcc34,
redhat5-i686-32bit-gcc41, redhat5-x86_64-64bit-gcc41 (Optimized)

Diffuse Model

/afs/slac.stanford.edu/g/glast/ground/releases/analysisFiles/diffuse/v2/source_model_v02.xml
(

https://confluence.slac.stanford.edu/display/SCIGRPS/Diffuse+Model+for+Analysis+of+LAT+Data

)

Diffuse Response IRFs

P6_V3_DIFFUSE, P6_V3_DATACLEAN

Output Data Products

FT1, LS1

Processing chain for FITS data products

Data Product

makeFT1

gtdiffrsp

gtmktime

gtltcube

FT1

true

true for
evclass==3,4

true

false

LS1

true

false

true

false

Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RedHat Enterprise Linux 4, and 5, gcc 3.4, 4.1, etc.). Currently, all batch machines run with the 64-bit operating systems.

Note on diffuse response calculation: gtdiffrsp is called two times in succession. The first time with IRF P7_V2_DIFFUSE and evclass==3, followed by IRF P7_32_dataclean and evclass==4. The resulting FT1 file has four columns of diffuse response, two columns (galactic and extragalactic response) for each of the two IRFs.

Timing

  • No labels