You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

P130 Reprocessing

status: In Development
last update: 25 January 2012

This page is a record of the configuration for the P130 reprocessing project, recalculating the diffuse response (see here for a description of the change). This task is identical with P120-FT1, run in July 2011, except for updating the version of ScienceTools to reflect the change in gtdiffrsp.

=============== WHAT FOLLOWS IS TO BE UPDATED ====================

The name "P120" derives from the word "processing" and the initial file version to be used for the output data products, e.g., r0123456789_v120_merit.root.

  • P120-MERIT - this task reads DIGI+RECON+MERIT and produces reprocessed MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT
  • P120-FT1
    • V1.0 of this task reads MERIT and produces FT1 (photons) + LS1 (merit-like FITS file for photons) + electron FITS file
    • V2.1 of this task is modified such that FT1 and LS1 files are filtered for FSSC, while new EXTENDEDFT1 and EXTENDEDLS1 files are produced containing all photon event classes.
  • P120-LEO-MERIT - this task reads DIGI+RECON+MERIT and produces reprocessed MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT for 200 runs of earth limb (L&EO) data

Datafile names, versions and locations

Data file version numbers for this reprocessing will begin with v120.

XROOT location and file naming

Location template:

/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>

Locations for P120:

/glast/Data/Flight/Reprocess/P120/merit
/glast/Data/Flight/Reprocess/P120/filteredmerit
/glast/Data/Flight/Reprocess/P120/electronmerit
/glast/Data/Flight/Reprocess/P120/ft1
/glast/Data/Flight/Reprocess/P120/extendedft1
/glast/Data/Flight/Reprocess/P120/electronft1
/glast/Data/Flight/Reprocess/P120/ls1
/glast/Data/Flight/Reprocess/P120/extendedls1

File naming:

Data Type

aka

Send to FSSC

Naming template

MERIT

 

No

r<run#>_<version>_<dataType>.root

FILTEREDMERIT

 

No

r<run#>_<version>_<dataType>.root

ELECTRONMERIT

 

No

r<run#>_<version>_<dataType>.root

ELECTRONFT1

 

No

gll_el_p<procVer>_r<run#>_<version>.fit

EXTENDEDFT1

 

No

gll_xp_p<procVer>_r<run#>_<version>.fit

FT1

LS-002

Yes

gll_ph_p<procVer>_r<run#>_<version>.fit

EXTENDEDLS1

 

No

gll_xe_p<procVer>_r<run#>_<version>.fit

LS1

LS-001

Yes

gll_ev_p<procVer>_r<run#>_<version>.fit

Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf

Example:

/glast/Data/Flight/Reprocess/P120/merit/r0239557414_v120_merit.root
/glast/Data/Flight/Reprocess/P120/filteredmerit/r0239557414_v120_filteredmerit.root
/glast/Data/Flight/Reprocess/P120/electronmerit/r0239557414_v120_electronmerit.root
/glast/Data/Flight/Reprocess/P120/extendedft1/gll_xp_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/ft1/gll_ph_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/electronft1/gll_el_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/extendedls1/gll_xe_p120_r0239559565_v120.fit
/glast/Data/Flight/Reprocess/P120/ls1/gll_ev_p120_r0239559565_v120.fit
DataCatalog location and naming

Logical directory and group template:

Data/Flight/Reprocess/<reprocessName>:<dataType>

Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.

Naming examples:

Data/Flight/Reprocess/P120:MERIT r0239557414
Data/Flight/Reprocess/P120:FILTEREDMERIT r0239557414
Data/Flight/Reprocess/P120:EXTENDEDFT1 r0239557414
Data/Flight/Reprocess/P120:FT1 r0239557414
Data/Flight/Reprocess/P120:ELECTRONFT1 r0239557414
Data/Flight/Reprocess/P120:EXTENDEDLS1 r0239557414
Data/Flight/Reprocess/P120:LS1 r0239557414

Data Sample

The currently defined data sample for P120 reprocessing includes:

First run

239557414 (MET), 2008-08-04 15:43:34 (UTC)

beginning of Science

Last run

333880535 (MET), 2011-08-01 08:35:33 (UTC)

Official Pass7 release

Total runs

16,459

 

Total MERIT events

35,921,666,747

all "events"

Total FILTEREDMERIT/EXTENDEDFT1/LS1 events

5,035,929,409

all photon event classes

Total ELECTRONMERIT/ELECTRONFT1 events

68,055,849

 

Total LS1 (FSSC selection) events

1,025,359,231

event classes (bits) 0,2,3,4 (transient, source, clean, ultraclean)

Total FT1 (FSSC selection) events

142,042,060

event classes (bits) 2,3,4 (source, clean, ultraclean)

Total disk space used

33.9 TB

 

Summary from DataCatalog as of 8/2/2011.

Name

Files

Events

Size

ELECTRONFT1

16459

68,055,849

6.4 GB

ELECTRONMERIT

16459

68,055,849

147.4 GB

EXTENDEDFT1

16459

5,035,929,409

441.3 GB

EXTENDEDLS1

16459

5,035,929,409

816.5 GB

FILTEREDMERIT

16459

5,035,929,409

4.0 TB

FT1

16459

142,042,060

12.9 GB

LS1

16459

1,025,359,231

166.6 GB

MERIT

16459

35,921,666,747

28.3 TB

NOTE: One run, 242429468, of type TrigTest was declared 'good for science' but long after this task got started, so it has been intentionally omitted.

8/17/2011 update: Four orphaned runs are being reprocessed, including one TrigTest run and four nadir-pointed runs.

Stream

run

type

16459

242429468

(TrigTest)

16460

333355876

(nadirOps)

16461

333358500

(nadirOps)

16462

333365716

(nadirOps)

Bookkeeping

  1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
  2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P120
    1. List of all reprocessings
    2. List of all data runs reprocessed
    3. Pointers to all input data files (-> dataCatalog)
    4. Pointers to associated task processes (-> Pipeline II status)
  3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
    1. Lists of and pointers to all output data files
    2. Meta data associated with each output data product

P120-FT1

This task generates all desired FITS data products. An example of the code processing chain appears on a child page.

Status chronology

Configuration (version 2)

Task Location

/nfs/farm/g/glast/u38/Reprocess-tasks/P120-FT1

Task Status

http://glast-ground.slac.stanford.edu/Pipeline-II/task.jsp?task=63390236

Input Data

MERIT (from P120-MERIT)

spacecraft data

FT2 from P105 (runs 239557414 - 271844560), then from current Level 1 production

Input Run List

 ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P120-FT1/config/runFile.txt

Reprocessing Mode

reFT1

meritFilter

FT1EventClass!=0

evtClassDefs

00-19-01

eventClassMap

EvtClassDefs_P7V6.xml

ScienceTools

09-24-00

Code Variants

redhat5-i686-32bit-gcc41 (Optimized)

Diffuse Model

based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0
(see

https://confluence.slac.stanford.edu/display/SCIGRPS/Quick+Start+with+Pass+7

)

Diffuse Response

'source' using P7SOURCE_V6 IRF
'clean' using P7CLEAN_V6 IRF

IRFs

P6V7, contained within ScienceTools release

Output Data Products

FT1, LS1, EXTENDEDFT1, EXTENDEDLS1, ELECTRONFT1

Processing chain for FITS data products

Data Product

selection

makeFT1

gtdiffrsp

gtmktime

gtltcube

FT1

'source' and above
EVENT_CLASS bits 2,3,4

true

true

true

false

LS1

'transient' and above
EVENT_CLASS bits 0,2,3,4

true

true

true

false

FT1EXTENDED

FT1EventClass!=0

true

true

true

false

LS1EXTENDED

FT1EventClass!=0

true

true

true

false

ELECTRONFT1

CTBParticleType==1

true

false

true

false

Note that diffuse response is calculated for 'source' and 'clean' event classes only.

Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.).

Timing

  • No labels