You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

status: Running
last update: 2 July 2013

This page is a record of the configuration and execution of the P300 reprocessing project, full reprocessing from DIGIs using Pass8 analysis code.

Pipeline tasks
  • P300-ROOT - This task reads DIGI and produces reprocessed RECON + GCR + MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT
  • P300-FITS - This task (once it has been created) reads MERIT and produces FT1 (photons) + EXTENDEDFT1 + LS1 (merit-like FITS file for photons) + EXTENDEDLS1 + ELECTRONFITS file

Datafile names, versions and locations

Data file version numbers for this reprocessing will begin with v300.

XROOT location and file naming

Location template:

/glast/Data/Flight/Reprocess/<reprocessName>/<dataType>

Locations for P300:

/glast/Data/Flight/Reprocess/P300/recon
/glast/Data/Flight/Reprocess/P300/gcr
/glast/Data/Flight/Reprocess/P300/relation/glast/Data/Flight/Reprocess/P300/merit
/glast/Data/Flight/Reprocess/P300/filteredmerit
/glast/Data/Flight/Reprocess/P300/electronmerit
/glast/Data/Flight/Reprocess/P300/ft1
/glast/Data/Flight/Reprocess/P300/extendedft1
/glast/Data/Flight/Reprocess/P300/electronft1
/glast/Data/Flight/Reprocess/P300/ls1
/glast/Data/Flight/Reprocess/P300/extendedls1

File naming:

Data Type

aka

Send to FSSC

Naming template

RECON

 

No

r<run#>_<version>_<dataType>.root

RELATION

 

No

r<run#>_<version>_<dataType>.root

GCR

 

No

r<run#>_<version>_<dataType>.root

MERIT

 

No

r<run#>_<version>_<dataType>.root

FILTEREDMERIT

 

No

r<run#>_<version>_<dataType>.root

ELECTRONMERIT

 

No

r<run#>_<version>_<dataType>.root

ELECTRONFT1

 

No

gll_el_p<procVer>_r<run#>_<version>.fit

EXTENDEDFT1

 

No

gll_xp_p<procVer>_r<run#>_<version>.fit

FT1

LS-002

Yes

gll_ph_p<procVer>_r<run#>_<version>.fit

EXTENDEDLS1

 

No

gll_xe_p<procVer>_r<run#>_<version>.fit

LS1

LS-001

Yes

gll_ev_p<procVer>_r<run#>_<version>.fit

Note: 'procVer' is a field added to the file name (and the keyword "PROC_VER" in the primary header) added to the FFD 5/12/2010. Ref: http://fermi.gsfc.nasa.gov/ssc/dev/current_documents/Science_DP_FFD_RevA.pdf

Example:

/glast/Data/Flight/Reprocess/P300/recon/r0239557414_v202_recon.root
DataCatalog location and naming

Logical directory and group template:

Data/Flight/Reprocess/<reprocessName>:<dataType>

Note that the <dataType> field (following the colon) is a DataCatalog 'group' name, and file names are of the form r<run#>.

Naming examples:

Data/Flight/Reprocess/P300:RECON r0239557414

Data Sample

The currently defined data sample (as of 7/2/2013) for P300 reprocessing includes:

First run

239557414 (MET), 2008-08-04 15:43:34 (UTC)

Last run

present

Bookkeeping

  1. (This page): Define ingredients of reprocessing (processing code/configuration changes)
  2. Processing History database: http://glast-ground.slac.stanford.edu/HistoryProcessing/HProcessingRuns.jsp?processingname=P300
    1. List of all reprocessings
    2. List of all data runs reprocessed
    3. Pointers to all input data files (-> dataCatalog)
    4. Pointers to associated task processes (-> Pipeline II status)
  3. Data Catalog database: http://glast-ground.slac.stanford.edu/DataCatalog/folder.jsp
    1. Lists of and pointers to all output data files
    2. Meta data associated with each output data product

P300-ROOT

Status chronology

  • 4/23/2013 - Begin setting up P300 task for Pass8 reprocessing.
  • 5/10/2013 - Initial version of task running.  See this page for infomation about the task creation, and this page for performance comparison with P202
  • 7/2/2013 - after weeks of testing, final GR, jobOpts, etc. come together and production tentatively starts.
    • block 1 is defined with this run range: ?239557414 through 392842073 (2008-08-04 15:43:37 through 2013-06-13 18:47:50 UTC)

Configuration

NOT YET UPDATED

Task Location

/nfs/farm/g/glast/u38/Reprocess-tasks/P300-ROOT

Task Status

http://glast-ground.slac.stanford.edu/Pipeline-II/index.jsp

GlastRelease

20-09-00

Run Selection

based on a modified "standard" selection, see https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets
(((sIntent=="nomSciOps" || sIntent=="nomSO_noSk_noCno_optGccc_allEna" || sIntent=="nomSciOps_diagEna" || (sIntent=="nomSciOps_Emin5MeV"&&RunMin>242070455) || nRun==242429468 ) && (RunQuality != "Bad" || is_null ( RunQuality ) ) ) || sIntent=="nadirOps" )

s/c data

P202 FT2SECONDS which will eventually become a "standard" Public Release https://confluence.slac.stanford.edu/display/SCIGRPS/Official+LAT+Datasets

Input Run List

ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300-ROOT/config/runList.txt

photonFilter

Not yet applicable, although this is defined by default:  CTBParticleType==1 && ((FT1EventClass & 0x00003EFF)!=0)
pass7.6_Extended_cuts_L1 in evtClassDefs

electronFilter

Not yet applicable, although this is defined by default:  CTBParticleType==1

Code Variants

redhat5-i686-32bit-gcc41 (Optimized), note that rhel5-64 and rhel6-64 GR builds are not yet available

jobOpts

ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P300-ROOT/config/Pass8Recon.txt

Output Data Products

RECON, GCR, RELATION, MERIT, FILTEREDMERIT, ELECTRONMERIT (but note that FILTEREDMERIT and ELECTRONMERIT are empty or contain junk)

Timing and Scaling

  • Current performance results for the Pass8 code are on this page .
  • ||job step||average CPU time||
    |processClump|183 min|
    |mergeClumps|10 min|
  •  

P300-FITS

This task generates all desired FITS data products. This task does not yet exist.

Status chronology

Configuration

NOT YET UPDATED

Task Location

/nfs/farm/g/glast/u38/Reprocess-tasks/P300-FITS

Task Status

http://glast-ground.slac.stanford.edu/Pipeline-II/task.jsp?task=75031156

Input Data

MERIT (direct from P300-ROOT)

spacecraft data

same as P300-ROOT

Input Run List

ftp://ftp-glast.slac.stanford.edu/glast.u38/Reprocess-tasks/P202-FITS/config/runList.txt

evtClassDefs

00-19-04 (6/23/2011)

eventClassMap

EvtClassDefs_P7V6.xml

ScienceTools

09-27-01 (2/15/2012) 09-31-01 (3/5/2013)

Code Variants

redhat5-i686-32bit-gcc41 (Optimized)

Diffuse Model

based on contents of /afs/slac.stanford.edu/g/glast/ground/GLAST_EXT/diffuseModels/v2r0 /v3r0
(see https://confluence.slac.stanford.edu/display/SCIGRPS/Quick+Start+with+Pass+7)

Diffuse Response

'source' using P7SOURCE_V6 P7REP_SOURCE_V10 IRF
'clean' using P7CLEAN_V6 P7REP_CLEAN_V10 IRF

IRFs

P7REP_*_V10, contained within ScienceTools release

Output Data Products

FT1, LS1, EXTENDEDFT1, EXTENDEDLS1, ELECTRONFT1

Generation of output data products:

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="3dbc9fcd-17b9-434a-b522-5cecad12d526"><ac:plain-text-body><![CDATA[

Data Product

destination

data content [1]

event selection [1]

makeFT1

gtselect

gtdiffrsp

gtmktime

]]></ac:plain-text-body></ac:structured-macro>

EXTENDEDFT1

SLAC

FT1variables

((FT1EventClass & 0x00003EFF)!=0)
pass7.6_Extended_cuts_L1

(tick)

(error)

(tick)

(tick)

FT1

FSSC+SLAC

FT1variables

'source' and above
EVENT_CLASS bits 2,3,4
evclass=2 filtered from EXTENDEDFT1

(error)

(tick)

(inherited)

(tick)

EXTENDEDLS1

SLAC

LS1variables

((FT1EventClass & 0x00003EFF)!=0)
pass7.6_Extended_cuts_L1

(tick)

(error)

(tick)

(tick)

LS1

FSSC+SLAC

LS1variables

'transient' and above
EVENT_CLASS bits 0,2,3,4
evclass=0 filtered from EXTENDEDLS1

(error)

(tick)

(inherited)

(tick)

ELECTRONFT1

SLAC

FT1variables

CTBParticleType==1
pass7.6_Electrons_cuts_L1

(tick)

(error)

(error)

(tick)

[1] /afs/slac/g/glast/ground/releases/volume04/evtClassDefs/00-19-04/data

Note that diffuse response is calculated for 'source' and 'clean' event classes only.

Note on 'Code Variant': The SLAC batch farm contains a mixture of architectures , both hardware (Intel/AMD 64-bit) and software (RHEL5-64, gcc v4.1, etc.). At this time, GlastRelease builds only on RHEL5-32 (RHEL6-64 are built but not yet validated), while ScienceTools builds for RHEL5-32, RHEL5-64.

Timing and Scaling

  • No labels