Page History

...

• P202-ROOT - This task reads DIGI and produces reprocessed RECON + CAL + GCR + MERIT + FILTEREDMERIT (photons) + ELECTRONMERIT

• P202-FITS - This task reads MERIT and produces FT1 (photons) + EXTENDEDFT1 + LS1 (merit-like FITS file for photons) + EXTENDEDLS1 + ELECTRONFITS file

...

• 3/2/2012 - Define block 1 as the 776 runs in P202-ROOT block 1. Configure trickleStream and begin (14:08)
• 3/31/2012 - Define block 2 as 5600 runs. Reconfig trickleStream and begin (18:05)
• 4/01/2012 - Block 2 complete (most of the 4824 jobs completed in about six hours w/1000 job limit).
• 5/31/2012
  • Discover stream 5599 (run 271999199) requires rollback - new MERIT file version (v203->v205).
  • Regenerate runlist with 20,229 input MERIT files.
  • Minor config change: twoClumpMin=False (formerly True, but caused unnecessary extra processClump dummy jobs)
• 6/5/2012 - Final cleanup
  • Five runs are responsible for discrepant event tallies:

    Run	Stream	Reason	Action
    239557414	0	bad TCut	rollback ROOT + FITS
    241599746	352	bad processing order	rollback FITS
    245403855	1019	bad start time	change start time in runList and rollback
    332661583	16244	silent root errors	rollback ROOT
    339081502	17416	silent root errors	rollback ROOT

  • Code changes:

    directory	script	modification
    commonTools/00-01-00	repTools.py	added new getKey() function to extract #events from FITS event files
    <ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="73772f1604284ffa-bbcb714a-4531430b-87c29e5d-e29311013f6f4bd3d7872ab5"><ac:plain-text-body><![CDATA[	P202-FITS/config	config.py	added os.environ['HEADASNOQUERY']='true' to enable 'ftlist' to run in batch	]]></ac:plain-text-body></ac:structured-macro>
    P202-FITS/config	makeFITS.py	add diagnostic print of #evts in FITS files for each processing sub-step

  • After these actions, the dataCatalog tallies are now consistent:

    Name	Type	Files	Events	Size	Created (UTC)	Links
    ELECTRONFT1	Group	20229	0	8.5 GB	02-Mar-2012 00:06:07	Files
    ELECTRONMERIT	Group	20229	90,904,582	205.7 GB	25-Jan-2012 00:53:32	Files
    FT1	Group	20229	189,323,074	17.8 GB	02-Mar-2012 00:06:06	Files
    LS1	Group	20229	1,325,204,821	215.3 GB	02-Mar-2012 00:06:08	Files
    EXTENDEDFT1	Group	20229	6,291,424,926	574.7 GB	02-Mar-2012 00:06:09	Files
    EXTENDEDLS1	Group	20229	6,291,424,926	1,020.1 GB	02-Mar-2012 00:06:09	Files
    FILTEREDMERIT	Group	20229	6,291,424,926	5.3 TB	25-Jan-2012 00:53:29	Files
    MERIT	Group	20229	44,125,679,961	35.4 TB	25-Jan-2012 00:53:30	Files
    RECON	Group	20229	44,125,679,961	590.0 TB	25-Jan-2012 00:53:33	Files
    GCR	Group	20229	44,125,679,961	942.7 GB	25-Jan-2012 00:53:31	Files
    CAL	Group	20229	44,125,679,961	128.7 TB	25-Jan-2012 00:53:31	Files

    Note that the number of events in ELECTRONFT1 files are not currently tallied by the dataCatalog.
• 8/24/2012 - Configure and run backfill through 31 July 2012
• 10/7/2012 - Rollback the following seven streams to fix corrupt FITS files. The makeFT1 app received a large number of ROOT error, but terminated normally. The resulting FT1/LS1 files had multiple symptoms, including one discovered by the FSSC during transfer to them: one EVENT_ID was repeated ~1000 times. Scan of log files uncovered an additional five mergeClumps with these errors.

  Code Block

  %INFO: 20120331:20:34:32 - makeFITS(run)/line-127 - Running: makeFT1 ---------------- start commentary ---------------- About to run [time makeFT1 rootFile=/scratch/glastmp/P202-FITS/3345/r0259101994_v202_merit.root fitsFile=/scratch/glastmp/P202-FITS/3345/gll_xp_p202_r0259101994_v202.fit TCuts=/afs/slac.stanford.edu/g/glast/ground/releases//volume04/evtClassDefs/00-19-04/data/pass7.6_Extended_cuts_L1 dict_file=/afs/slac.stanford.edu/g/glast/ground/releases//volume04/evtClassDefs/00-19-04/data/FT1variables tstart=259101996.933000 tstop=259106272.085000 file_version=202 tempRootFile=/scratch/glastmp/P202-FITS/3345/dummy.root xml_classifier=/afs/slac.stanford.edu/g/glast/ground/releases//volume04/evtClassDefs/00-19-04/xml/EvtClassDefs_P7V6.xml evtclsmap=FT1EventClass chatter=4 debug=yes] at Sat Mar 31 20:34:32 2012 ---------------- start log ---------------- This is makeFT1 version ScienceTools-09-27-01 applying TCut: ((FT1EventClass & 0x00003EFF)!=0) && (EvtElapsedTime >= 259101996) && (EvtElapsedTime <= 259106273) Warning in <TClass::TClass>: no dictionary for class FileHeader is available Warning in <TClass::TClass>: no dictionary for class RootObj<int> is available R__unzip: error in header Error in <TBasket::ReadBasketBuffers>: fNbytes = 15550, fKeylen = 86, fObjlen = 31912, noutot = 0, nout=0, nin=11010118, nbuf=7864444 Error in <TBranch::GetBasket>: File: /scratch/glastmp/P202-FITS/3345/r0259101994_v202_merit.root at byte:459100837, branch:Vtx2LongDoca, entry:510338, badread=0, nerrors=1, basketnumber=1 [...]

  Stream	Run
  3345	259101994	<- found by FSSC
  4122	263571912
  4707	266893978
  13927	319436826
  16181	332306548	<- found by FSSC
  17430	339161346
  17479	339408141

• 10/15/2012 - Reconfigure for block 4 backfill and begin trickleStream.
  • Discover run 22240 makeFT1 contains error messages as the mentioned last week (above). Rolling back the entire chain of processing, starting with P202-ROOT seemed to do the trick.
• 10/17/2012 - block4 backfill complete.
• 12/18/2012 - Prepare for block 5. Before:

  Name	Type	Files	Events	Size
  EXTENDEDFT1	Group	23141	7,252,996,601	662.6 GB
  EXTENDEDLS1	Group	23141	7,252,996,601	1.1 TB
  FT1	Group	23141	229,126,932	21.5 GB
  LS1	Group	23141	1,549,133,207	251.7 GB

• 12/19/2012 - block5 backfill complete. Final DataCatalog numbers:

  Name	Type	Files	Events	Size
  EXTENDEDFT1	Group	24142	7,578,468,156	692.3 GB
  EXTENDEDLS1	Group	24142	7,578,468,156	1.2 TB
  FT1	Group	24142	240,949,332	22.6 GB
  LS1	Group	24142	1,621,773,636	263.5 GB

...

...