Blog

Code Versions

GlastRelease (sim/recon):

The I&T pipeline is now using GlastRelease instead of EngineeringModel. GR v13r6p1 is nearly identical to GR v13r5p6 which is used for the Big Run and has been extensively validated for MC (more about real data later).  Here I only list the differences with respect to GR v13r5p6:

configData v1r4p1

- 17-Dec-07 MK Added -o option (for output file name) to dump full GEM information.
- 03-Dec-07 EAC Added broadcast settings.
- 28-Nov-07 EAC Fix up comparison tools
- 27-Nov-07 EAC Fix xml tag names for new top level files

ntupleWriterSvc v3r19:
- 14-dec-2007 HMK: allow basket size to be set and adjust default size to 1000000 bytes

Gleam v6r33p7:
- v6r33p7 AWB 13-Dec-07 Updated some more.
- v6r33p6 AWB 13-Dec-07 Updated JO for pipeline.

Trigger v6r0p4:
-16-dec-07 LSR limit info message about trigger bit mis-matches

calibData v0r19p3:
- 17-Dec-2007 jrb Hide statics needed for Acd calibrations to make Windows happy (thanks, Tracy)

Differences with respect to GlastRelease v13r5p6

System Tests for this version

System tests results: 

No changes with respect to GR v13r5p6 except for ACD DIgi IDs. This is due to a change in the system tests themselves which now use the dense ACD tile ID notation and not the sparse one. Note that the system tests for GlastRelease do not contain any tests using real data. I have done additional validation on real cosmic ray data like remaking a digi+recon SVAC report and compared it to the original report. No problems have been observed.

Fred version:

v0r99

Pipeline version:

v1.4.7

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

svac pipeline code and tasks:

code/tasks v4r0p0:

pipelineDatasets v0r4

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p3

TestReport:
v4r1 awb
- Fixed Acd Mips maps so it works correctly with the new ACD code.
- Updated to run with Root V5.16 (GR v13).
- Changed the word 'EngineeringModel' to 'GlastRelease' in the report.

EngineeringModelRoot (SVAC ntuple):
v3r9p2 awb
- Removed obsolete variables, added new ones and changed some existing ones. Additional details can be found in this Science Operations email and this Science Operations presentation.
- Increased the ROOT basket size to deal with runs with many events like DITL. It also decreases the file size. 

EMI Monitoring for the ACD:

svac/Monitor: version mk20070719
dataMonitoring/Common: version acdemi2

New! Configuration report
We will start making a configuration report for each run. While the main effort will be put into the one for pipeline 2 we will put in something for pipeline 1 and TVAC.
Note that this replaces the old (and broken) config report. Currently we extract the LatC Moot key and a) dump a text file with detailed trigger information and b) an ntuple containing register values into the 'configReport' directory which will be linked from the runs database (replacing the old link to the old configReport). This is done using applications in the configData package which is part of GR. We will work on making a more presentable report in the very near future.    

Code Versions

Engineering Model (sim/recon): v8r1109p12

Small patch release:

Trigger:
v4r11p1em1 HMK - patch along GlastReleasev11r9 branch for EM to turn TRIROWBITS back on see JIRA GRINF-34.

AnalysisNtuple:
v2r23p7em1 HMK-  branch tag to apply Eric's patch to calc. on top tiles to convert MIPs to MeV.

LatIntegration:
v2r76 awb Reduced some system tests to 50k events to speed up things.

Differences with respect to EngineeringModel v7r0913p10

System Tests for this version

System tests results: 

No differences seen wrt to EM v7r0913p11. There are a few differences between v7r0913p11 and  v7r0913p10 (current EM in the pipeline). They are due to the decreased statistics we now use in the system tests.

Fred version:

v0r99

Pipeline version:

v1.4.7

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r9p4:

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p2
ConfigTables: v4r0p3

TestReport:
v3r7p5 awb - Replaced m_nbrMissingGps with m_isSourceGPS to print correct message about whether we have GPS or not.

EngineeringModelRoot (SVAC ntuple):
v3r1p3 awb - Replaced 'MissingGps' with the correct name 'IsSourceGps'. Note that the accessor functions have not been updated yet.

EMI Monitoring for the ACD:

svac/Monitor: version mk20070719
dataMonitoring/Common: version acdemi2

Code Versions

Engineering Model (sim/recon):  (v8r1109p8)   v8r1109p10

Based on GR V11 and using Root 5.14.00g. Among many other things it has the new LATC keys that we need to query Moot and new ACD variables in Merit that Alex needs. 

(Differences with respect to EngineeringModel v7r0913p15)

Differences with respect to EngineeringModel v7r0913p15

Since we are resynching to a much more recent GR most of the tags have changed. I will not give details here - the curious can find them in the link above.

System Tests for this version

System tests results: 

A lot of differences. Mostly for the MC and due to a new version of Geant (including pruning of secondaries). For data there are a few small differences in aCD related quantites. Not surprising since the ACD code was upgraded. 

The LAT_LICOS-FSW test did not run, but this is now understood. This is an old run which "doesn't contain a "real "secondary-boot-script file key" so the upstream translation service (CHS/eventFile) failed. This has now been fixed.

Fred version:

v0r99

Pipeline version:

v1.4.7

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r9p4:

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p2
ConfigTables: v4r0p3

TestReport v3r7p4:
v3r7 mutsumi - Was already in the for EMI testing!
                       - move TkrNoiseOcc module from svac/EngineeringModelRoot
                       - add TKR noise analysis in TestReport digi analysis
                       - The output root hist file includes the directory "TkrNoiseOcc"
                       - It is used to generate TKR noise report
v3r7p1 awb - Added early event counter.
v3r7p2 awb - Retagging TRdp20070706 (changes to MergeHistFiles.cxx used in Pipeline 2).
v3r7p3  awb - Added warning about sweep events not properly accounted for in extra event warning.
v3r3p4 awb - Fixed typo in sweep events warning. 

EngineeringModelRoot v3r1p2 (SVAC tuple):
v3r1 mutsumi:  Move TkrNoiseOcc module to TestReport (this was already in for the EMI testing).
v3r1p1 awb: Added earlyEvent flag for time tones.
v3r1p2 awb -  Tagging Eric's updated ntuple description.

EMI Monitoring for the ACD:

svac/Monitor: version acdemi6
dataMonitoring/Common: version acdemi2

Code Versions

Engineering Model (sim/recon): v7r0913p15

New version of calibGenCAL containing the CAL EMI report.

This change only affects the package calibGenCAL, is only used in the standalone CAL EMI report running in the pipeline and has no effect on the reconstruction code (confirmed by the system tests).

There are also two new standalone packages that will run in the pipeline to do the ACD EMI monitoring. 

Differences with respect to EngineeringModel v7r0913p14.

calibGenCAL v4r4_emi_p2

System Tests for this version

System Tests results:

No changes.

Fred version:

v0r99

Pipeline version:

v1.4.7

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r9p4:

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p2
ConfigTables: v4r0p3

TestReport v3r6p41 Same as before

EngineeringModelRoot v3r0p3 (SVAC tuple): Same as before

New EMI Monitoring for the ACD:

svac/Monitor: version acdemi6
dataMonitoring/Common: version acdemi2

Code Versions

Engineering Model (sim/recon): v7r0913p13

New Tkr report pass/fail limits after discussions between Eric Grove and Hiro.

This change only affects the package calibTkrUtil, is only used in the standalone TKR report running in the pipeline and has no effect on the reconstruction code (confirmed by the system tests).

Differences with respect to EngineeringModel v7r0913p12.

calibTkrUtil v1r7

System Tests for this version

System Tests results:

No changes.

Fred version:

v0r99

Pipeline version:

v1.4.7

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r9p4:

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p2
ConfigTables: v4r0p3

TestReport v3r6p41 Same as before

EngineeringModelRoot v3r0p3 (SVAC tuple): Same as before

Code Versions

Engineering Model (sim/recon): v7r0913p12

Per Eric Grove's request: "I see that the TKR noise occupancy test that's in the pipeline still uses the 1e-4 _per_Tower_ threshold to evaluate pass/fail.  We need to change this ASAP for Observatory level EMI tests that are scheduled to start in less than two weeks. This value is far too stringent and far below the TKR Level III spec of 1e-4 for the total TKR."

This change only affects the package calibTkrUtil, is only used in the standalone TKR report running in the pipeline and has no effect on the reconstruction code (confirmed by the system tests).

Differences with respect to EngineeringModel v7r0913p11.

calibTkrUtil v1r6

System Tests for this version

System Tests results:

No changes.

Fred version:

v0r99

Pipeline version:

v1.4.7

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r9p4:

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p2
ConfigTables: v4r0p3

TestReport v3r6p41 Same as before

EngineeringModelRoot v3r0p3 (SVAC tuple): Same as before

Code Versions

Engineering Model (sim/recon): v7r0913p11

Fixes JIRA GRINF-35. When we moved to a GR v9 based EngineeringModel, MeritTuple was moved from the output sequence to the reconstruction sequence. In case of events with errors, we skip the reconstruction sequence and inserts an empty event in the root files and ntuples, thus preserving the 1-1 correspondence between all the data files. Unfortunately, because of this change the empty events were not inserted into the Merit ntuple, thus breaking the 1-1 correspondence. This new release fixes this bug.

Differences with respect to EngineeringModel v7r0913p10.

LatIntegration v2r65:
30-MAR-07 v2r65 hmk add Leon's brillant NtupleMaker Sequence to insure ntuple is written when recon is skipped.

System Tests for this version

System Tests results:

No changes.

Fred version:

v0r99

Pipeline version:

v1.4.7

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r9p3:

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p2
ConfigTables: v4r0p3

TestReport: v3r6p41 (digi & recon reports): awb
- Added line break between digi and recon files at the top of the report.

EngineeringModelRoot: v3r0p3 (SVAC tuple): Same as before
    

Code Versions

Engineering Model (sim/recon): v7r0913p10

This release contains a one-line bug fix to the Event package to avoid that all calibrated ACD hits get associated with tile number 0. In addition, Joanne had a new version of xmlUtil.

Event v11r13em0:
Erich Charles - Set ID of calibrated ACD hit. 

xmlUtil v3r2:
v3r2 23-Feb-2007 jrb Add support for tag.

Differences wrt the previous pipeline EM release (EM v7r0913p9)

System Tests for this version

System Tests results:

No differences as none of the system tests have any histograms with calibrated ACD quantities. I reprocessed a muon run and reran the recon report and verified that the fix indeed fixed the problem. 

Fred version:

v0r99

Pipeline version:

v1.4.6

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r6p0:

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p1
ConfigTables: v3r2p1

TestReport: v3r6p38 (digi & recon reports): awb
- v3r6p38 Explicit check on time tones to see if time goes backwards or stands still (JIRA COM-66/FSW-878).
- v3r6p37 Fixed a format problem in the table with TKR numbers.

EngineeringModelRoot: v3r0p3 (SVAC tuple): Same as before

Code Versions

Engineering Model (sim/recon): v7r0913p9

This is a GlastRelease v9 based EngineeringModel i.e. we have finally caught up with the rest of the world. This means new versions of a lot of infrastructure code like Root, Geant4, CLHEP and Gaudi. Because of the major release number change a lot of tags are different. Non-infrastructure code has not changed much with respect to the current EM release in the pipeline, however, as we already had many EM specific patches in the old release. Upgrading to this new release makes maintenance much easier as we now are much more in synch with GlastRelease.

Interesting bug fixes include a change to the size of the TKR cluster arrays so they can accommodate large events (JIRA ROOT-25), more protection from the Geant4 propagator getting stuck (JIRA GEANT-8) and a divide by 0 problem in the time stamp calculation in case of funny time tones (JIRA COM-65). A complete list of all the differences with respect to the current EngineeringModel release in the pipeline can be found in:  

Differences wrt the previous pipeline EM release (EM v6r070329p29em1)

Also note that the current EM release v6r070329p29em1 seems to have an intermittent problem with the time tones in the digi.root file (JIRA ROOT-26). This is not seen in any other release. I looked at the previous release, EM v6r070329p28, and also EM v7r0913p8 and the new one, EM v7r0913p9. None of them has this problem. Heather was supposed to look at it to better understand what is happening - in case the problem is still there, but just shows up in different place - but this is obviously on hold now ....

System Tests for this version

System Tests results:

There are changes in MC tests, none of the major. Since Geant4 has changed this is probably not unexpected. In the data system tests there were changes (KS value different from 1), but none were significant.

I processed part of a data run with the new release and made digi and recon  reports. They look normal.

Fred version:

v0r99

Pipeline version:

v1.4.6

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r6p0:

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p1
ConfigTables: v3r2p1

TestReport: v3r6p36 (digi & recon reports): awb
- v3r6p36 Additional protection in time stamp code.
- v3r6p35 Fixed bug in <6 hits in the TKR and still a TKR trigger table.
- v3r6p34 Added MergeHistFiles.cxx and related lines to cmt/requirements

EngineeringModelRoot: v3r0p3 (SVAC tuple): Same as before
    

Code Versions

Engineering Model (sim/recon): v6r070329p29em1

Because of FSW bugs in the new (de)compression we sometimes get corrupted events. Currently this stops digitization. This new release is more lenient with corrupted events so we can digitize the rest of the run. We currently have 53 runs we have not been able to digitize - limiting the monitoring we can do of LAT data. These changes are all in LDF related packages.

This new release also introduces a proper event time stamp intead of using the CCSDS package header time which is common for all events in a datagram.  

Differences wrt the previous pipeline EM release (EM v6r070329p28)

New tags:

ldfReader v3r31p1 (see link above for a complete list of changes):
- hmk patch to AcdParser to store tileName correctly when no remapping occurs.
- hmk Anoter patch from Anders for the timestamp code
- hmk Add new LDF Warning flag if read bytes beyond end of TEM
- hmk Patch timestamp code to convert datagram creation time from starting 1/1/1970 to mission start 1/1/2001
- hmk check for missing readouts in CAL ALLRANGE and flag

enums v0r10:
- HMK Add a new error flag: LDFWARNING

digiRootData v9r5p2em1:
- HMK Add methods to query new bits in eventflag
- HMK Patch for MetaEvent copy ctor to init properly
- HMK Patch for windows in unit test 

RootIo v17r9gr11:
- HMK Insert debugging statement

LdfConverter v2r7gr1:
- HMK Continue processing event even if missing CAL readouts in ALLRANGE

iExternal/ldfExt v0r620p1:
- HMK New directory struct for windows, drop the VC8 and VC8debug portion.
- HMK LDF v06-02-00 fix up new errors found during beamtest

System Tests for this version

System Tests results:

Only changes are in some CAL quantities and are consistent with using new CAL calibration files. The Tower A test did not run in the two previous releases (timed out) which prevents histograms for this test to be displayed (independently of which release on chooses as reference).

Fred version:

v0r99

Pipeline version:

v1.4.6

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r6p0:

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p1
ConfigTables: v3r2p1

TestReport: v3r6p35 (digi & recon reports): awb    [This one did not make it into the pipeline!]
- v3r6p31 Trying to deal with cycles and datagram gaps for LCI runs.
- v3r6p32 Fixed event time stamps.
- v3r6p33 Typo in previous tag fixes.
- v3r6p34 Added MergeHistFiles.cxx and related lines to cmt/requirements
- v3r6p35 - Fixed bug in <6 hits in the TKR and still a TKR trigger.
               - Commented out runStrip from requirements file because of missing files.

EngineeringModelRoot: v3r0p3 (SVAC tuple): Same as before

Code Versions

Pipeline version:

v1.4.6

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r8p3:
No longer necessary to let pipeline drain before upgrades. Multiple version can run concurrently.
Added TkrReport task
If some recon chunks fail, don't redo the ones that succeeded
Split recon into many TPs to give finer rollback granularity
Longrunning TPs run on glastdataq
Simplified data directory structure
Better automatic cleanup after recon failures
More sharing of wrappers between TPs
Make .htaccess files to prevent HTTP downloads of ROOT files
Delete .evt, .pkt, and some RECON.root files after they are used
Raise recon chunk size to take advantage of new LSF queue limits

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p2

ConfigTables: v4r0p3
Partial support for LATC/FSW

TestReport: v3r6p30 (digi & recon reports):
 - v3r6p15 awb - Fixed bug in TKR layers per tower (first half was repeated twice). Hmmm .... probably benign ....  only a (not used) label problem.
                     - Added rate of some quantities. Uncommented the livetime calculation.
- v3r6p16 awb - Not plot CAL occupancy for periodic triggers.
                        - Added livetime corrected average trigger rate.
                        - Added time plot of livetime.
                        - Fixed time of first and last event.
                        - Fixed stat opt for some plots.
- v3r6p17 awb Added plot of datagram creation time minus event time.
- v3r6p18 awb Added correct stat mode for the new histograms.
- v3r6p19 awb Fixed some things.
- v3r6p20 awb - Added histogram for delta event ID per CPU.
                        - Fixed bug in datagram creation time minus event time plots.
                        - Fixed bug in calculation of average rates.
- v3r6p21 awb Corrected typo in axis label in one histogram.
- v3r6p22 awb Fixed formating in report.
- v3r6p23 awb Fixed bug in 'Not plot CAL occupancy for periodic triggers.'.
- v3r6p24 awb - Fixed normalization in CAL plots without periodic triggers.
                        - Plot negative delta GEM IDs, not only print a warning in case it happens.
                        - Added expected livetime.
- v3r6p25 awb Fixed time calculations when more than one second between current and previous timetone 1-PPS.
- v3r6p26 awb Now use gmtime to avoid Daylight Savings Time bug.
- v3r6p27 awb Another tag, another rollover ....
- v3r6p28 awb Added explicit check to see if we read out hits from the left in TKR Bay 10, layer 0.
- v3r6p29 awb Added check for gaps in datagram sequence number.
- v3r6p30 awb Made rate plots linear scale.

EngineeringModelRoot: v3r3 (SVAC tuple) 
- v3r0p1 awb Minor update in merge Root script so it now points to an existing release.
- v3r0p2 awb Added a few more variables.
- v3r0p3 awb Fixed typo in ACD non-attached tile message.
   

Code Versions

Engineering Model (sim/recon): v6r070329p28

Small update to calibTkrUtil to get overall LAT occupancy for the trackers.

 
Differences wrt the previous pipeline EM release (EM v6r070329p21)

New tags:

calibTkrUtil v1r5:
- Overall LAT TKR occupancy.

System Tests for this version

System Tests results:

No changes wrt to EM p28. 

Fred version:

v0r99

Pipeline version:

v1.4.6

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r6p0:

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p1
ConfigTables: v3r2p1

TestReport: v3r6p14 (digi & recon reports): awb
- New events per datagram histogram.
- Corrected text in report ('missing time tones' to 'events with missing time tones').

EngineeringModelRoot: v3r0 (SVAC tuple)   Same as before
     

Code Versions

Engineering Model (sim/recon): v6r070329p27

This release can handle a missing range in 4-range readout without crashing, some updates to the CAL ntuple
and we now use the new DFI that correctly interprets the TimeTone error flags.

With this release we can now run the TKR noise monitoring automatically in the pipeline. This will be a new pipeline task
that will run after the generation of the SVAC ntuple, use the TKR histograms that we store in that Root file and make
a report. This report will now be linked for each run that we run recon on, just like the usual SVAC reports (digi and recon reports).
This new report was made manually during EMI testing and turned out to be very useful, especially for flare monitoring. This new report
was added to the pipeline at Eric Grove's request for TVAC operations.
 

Differences wrt the previous pipeline EM release (EM v6r070329p21)

New tags:

lsfData v0r6:
- RBL Add LsfTimeTone.earlyEvent and LciConfiguration.strobe flags to match DFI V3-4-1

enums  v0r9:
- RBL Add EMPTY_EVENT and STROBE flags in Lsf.h to match DFI V3-4-1
- HMK Add flag for missing CAL readouts in ALLRANGE
- HMK Add constants for flight OBF 

RootIo v17r9gr10:
- HMK fix up check of kWriteError bit
- HMK Check kWriteError bit and report error if necessary
- HMK write out CAL ALLRANGE even if missing readouts

RootConvert v1r5em3:
-  HMK remove old debug statement from LsfDigiConvert.cxx

CalXtalResponse v0r9p2em4:
- fewtrell Branch tag enables 4-range for all CalTuple's. also fixes bug where CalTuple forces all tuples to write after every event. 
- fewtrell enable 4-range faceSignal for CalTuple (not just adc values). also fixes indexing bug in 4 range adc from previous tag.
- fewtrell also ensure that below pedestal adc values do not result in negative energy  

CHS/eventFile v2:
- RBL new file format. Added strobe and earlyEvent flags from DFI V3-4-1.

LatIntegration v2r59:
- awb Changed input file for system tests to new evt file version. 

System Tests for this version

System Tests results:

No changes wrt to EM p21. 

Fred version:

v0r99

Pipeline version:

v1.4.6

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r6p0:

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p1
ConfigTables: v3r2p1

TestReport: v3r6p12 (digi & recon reports): awb
-  Improve code to catch missing datagrams for LCI runs, JIRA SVAC-107.

EngineeringModelRoot: v3r0 (SVAC tuple)   Same as before
     

July 31, 2006: Approved unanimously (Richard unavailable)

Code Versions

Engineering Model (sim/recon): v6r070329p16

This release uses calibrated ACD quantities. In addition, the ACD VETO threshold is changed back to 0.3 MIP (Real data from April and May use both 0.1 and 0.3), and the CNO threshold is set to 20 MIPs (data threshold). Run and event ID in AnalysisNtuple are doubles to avoid truncation for real data. We also now have better error decoding (can distinguish TKR FIFO full errors from the generic TEM error for example). The first step on a long road ..... 

Differences wrt the previous pipeline EM release (EM v6r070329p5)

New tags:

Event v11r8em4: I
- EAC Fixes AcdHit tile ID not set bug. 

xmlGeoDbs v1r29gr2: R
 - hmk Merge VDG updates from Anders into this branch for EM
 - jrb Update requirements and this file correctly
 - jrb Update requirements and this file correctly (I hate having to do this by hand)
 - jrb Merge in 2 little fixes from head
 - jrb Get rid of ACDTop composition; put contents in new fragment ACDTop.xml

merit v6r28p5: E
- THB fix FT1Energy; new define required by OnboardFilter on Win32
- THB FT1 mod to set pointing stuff for background
- THB fix FT1Livetime

ldfReader v3r19p2: A
- hmk Forgot to add ErrParser.cxx
- hmk print eventId and apid when reading LSF with EbfDebug set
- hmk Store TKR fifo errors in all their glory and update
- hmk Eventflags to include common error bits in error contribution
- hmk Modify ACD parsing to create AcdDigi objects during the header method rather than just during PHA. This takes care of the unlikely occurance of veto latching occurring after GEM.

enums v0r6: L
- HMK Check in new include DetectorConstants.h
- HMK New eventflags for error contribution

idents v2r17: L
- jrb Add a constructor and some get fctns to AcdId
- eac add AcdGapId

commonRootData  v2r8: Y
- EAC Added AcdMap to convert back to electronics space 

calibUtil  v1r6: H
- jrb Add new dtd for Acd calibs & a couple sample files 

calibGenCAL v3r9p6: A
- fewtrel: fixed stupid bug in MuonCalib::readAsymTXT(), probably introduced when I added tower info to output txt files in v3r9
- dwood/fewtrell XML parser - switched base XML parser from python standard (minidom) to PyXML pacakage (4DOM)
- dwood/fewtrell allows for DTD validation:
       - DTD validation is now enabled for XML files required to have DTD information prefixed
       - DTD validation still disabled for XML files which have no DTD yet
       - better for quality control, but quite a bit slower 
- dwood/fewtrell adcsmooth - numerous changes to work correctly with sparsely sampled characterization data
- dwood/fewtrell extrapolate data near the pedestal noise region; promotes sharp cutoff at low end of curves
- dwood/fewtrell LAC pedestal noise is zeroed out:
       - smoothing filter does not run at endpoints of curves; promotes sharp cutoff
       - better extrapolation of FLE coarse range values past measurement range
      - special processing for ULD data to reconstruct saturation plateau with sparse data  
- dwood/fewtrell charplot - new utility to plot raw and smoothed characterization data. Mainly a tool to diagnose the output of adcsmooth  
- dwood/fewtrell charVal - new validation application for characterization data XML files. Will catch gross errors, but probably still needs work to improve validation criteria  
- dwood/fewtrell dacVal - new validation application for DAC settings XML files:
     - works backwards from DAC setttings to calculate threshold in energy units
     - requires a complete set of files to run
     - nothing yet for ULD 
- dwood/fewtrell adc2nrgVal - fixed bug with ROOT output option 
- dwood/fewtrell xxxXML2TXT.py - new scripts generate collumnar text files from offline Cal calibration files 
- dwood/fewtrell dacDiff.py - generates per channel differences between any 2 cal dac settings snapshot files. 
- dwood/fewtrell roothist2csv.py - generates collumnar text files from all histograms in ROOT file. 
- dwood/fewtrell dumpROOTPlots.py - generates images files for all histograms & Canvases in ROOT file. 
- dwood/fewtrell unit_test - now 2 tower instead of 8 to save processing time  
- dwood/fewtrell runSuiteParallel.py - now supports commandline override of twr_id as well as each individual processing phase. Allows work to be easily broken up into several threads w/ same cfg file.
- dwood/fewtrell C++ - TXT format output now supports new format used by python scripts. improved format includes tower information & contains all 4 asym types & both MeVPerDAC types into single file each.
- fewtrell C++ apps now use TH1S instead of TH1F across board. Results are the same & it saves memory
- fewtrell future updates w/ 16 tower support will use a lot of memory so this is important step. C++ apps now use float instead of double in most places. the precision is not needed & it saves some memory, particularly in larger arrays. 
- fewtrell All python scripts now have .bat & .sh shell launcher scripts. All .sh scripts have executable bit set. 
- fewtrell runSuiteParallel.py, gensettings.py, build_adcsmooth.py all generate appropriate validation outputs along w/ their primary output.
- dwood/fewtrell For DAC settings XML file produced from genXXXsettings tools, modified the values of the element 'hierarchy' and 'shape' attributes. Hierarchy is set to the value '['GCCC','GCRC','GCFE','']' and shape is set to the value '(8,2,12)'. 
- dwood/fewtrell Added support to tools for handling LEX1 energy range FLE DAC characterization files. Currently, a simple scale factor is used to convert to LEX8 ADC units and process as before.
- dwood/fewtrell Better file type checking for files which use the ADC/DAC XML tables as input.  
- dwood/fewtrell Muon asymmetry now smarter for cross diode asymmetry. Uses average ratio of Large2Small diode signal instead of the slope. This will make energy & position response @ 1 GeV +- 100 MeV better. 
- dwood/fewtrell runCIFit can now process broadcast mode calibGen as well as the older, 1 column at a time method. 
- dwood/fewtrell uldVal: new application to validate ULD DAC settings XML files (from genULDsettings)  
- dwood/fewtrell dacVal:
    - fixed report of FAILED tests
    - relaxed FHE error and warning limits 
- dwood/fewtrell charVal:
    - added support for ULD characterization files
    - added check for incorrect pedestal subtraction
    - relaxed FHE and LAC error and warning limits  
- dwood/fewtrell build_adcsmooth: added call to charVal to do validation on ULD characterization files
- dwood/fewtrell gensettings:
    - added call to uldVal to do validation on ULD DAC settings files
    - fixed bug which put 'cp' instead of 'copy' in Windows BAT output files
    - made processing of muon gain FHE files optional; command line -m or --muon 
- dwood/fewtrell build_tholdci_cfg: make sure FHE threshold is properly converted from GeV to MeV 
- dwood/fewtrell runSuiteParallel: fixed undefined environment variable error
- dwood Added titles and axes labels to ROOT output graphs generated by plot and validation scripts 
- dwood genULDsettings - removed one DAC step safety margin; output margins are closer now to user requested value  
- dwood uldVal:
    - look at all energy ranges when comparing threshold to saturation level
    - add check to make sure (saturation - threshold) is not less than margin paramter 
    - tholdCIVal - removed check of FLE threshold verses LEX8 ULD value; not needed now that FLE data is LEX1
- fewtrell fixed bug introduced w/ broadcast mode in v3r9p

calibGenACD v1r4p1: T
- EAC Fix a spurious include in cmt/requirements
- EAC Added runMuonCalib_Svac.cxx to run mip peak calibrations on SVAC ntuple files.
- EAC Make changes so that xml format agrees with the dtd file that joanne has
- EAC Remove a dead src/runMuonCalib_Roi.cxx file which was moved to apps subdir
- EAC Initilize pedestal file to zero to prevent crash if it don't exist
- EAC Fairly big update, added AcdMeritCalib, changed the structure a bit.
- EAC Fairly big update, added AcdVetoCalib, changed the structure a bit.
- EAC Minor update. Add information about the datasets used to generate calibrations to the txt and xml output files. Also, added apps/ subdir for application main() routines.
- EAC Major re-write. Brings in some code from the ACD. Uses polynomials to smooth pha distributions and find the peak values. Uses a simple algorithm to decide what range to fit over. Supports both ped-subtracted and raw values.

digiRootData v9r5: E
-  HMK Add Tkr FIFO Full bits to Error class 

RootIo v17r9gr7: C
-  HMK TKR FIFO errors from LDF

LdfEvent v3r7: O
- HMK store TKR fifo errors from err contribution 

LdfConverter v2r7: N
- HMK store error contribution in eventFlags and really store TKR fifo errors in their full glory

TkrRecon v10r8p1em0: F
- LSR Fix the Vtx status bit thing for latest EM

LatIntegration v2r58: L
- hmk Patch calibration.txt JO file for AcdCalibSvc
- awb Added JO so we get CTB variables in data and MC.
- awb Added LICOS/FSW LPA run for system test.
- awb More ACD calib JO updates. Removed 'LivetimeSvc.Deadtime'.
- awb Added 'ideal' as valid calibration flavor for MC (for the ACD simulations - for now).
- awb Turn off LivetimeSvc in ldf2digi.
- awb Updated single tower VDG source location. 

CalibSvc v0r21p8em0: U
- jrb Merge in ACD calibration support 

CalibData  v0r12p3em0: E
- jrb Merge in support for Acd calibrations

CHS/eventFile  v1r0p3: N
- RBL added dumpEvent single-dump utility

AnalysisNtuple  v2r9p3gr7: C
- LSR EvtRun and EvtEventId to double
(awb) This is just an emergency fix for GRINF-29. The correct solution is to make the ULong64_t. 

AcdUtil  v1r1p3em4: E
- EAC "ideal" now uses built-in code instead of file in calibDB
- EAC Added apply_pattern use_CalibData_symbols for windows
- EAC Work some obscure magic so that windows understands what to do
- EAC Pull in updates from branch EM-v6r070329p0, tag v0r2p10em0
- EAC Added AcdCalibSvc: So far this handles pedestals and gains (aka mip peaks). Adds a dependency of CalibData
- HMK updates for CLHEP 1.9.2.2
- Along Branch EM-v6r070329p0:
       - EAC No really, actually tag the stuff this time
       - EAC And tag correct versions of cmt requirements
       - EAC And tag correct versions of the dll stuff
       - EAC pull in requirements file from main branch for AcdCalibSvc
       - EAC pull in calibration stuff from main branch
       - EAC Cache transform to local coords in AcdTileDim and AcdRibbonDim 

AcdRecon :v3r0em8 !
- EAC Arggh fix some missing bits from v3r0em7
- EAC Pull back in calibration stuff, which seems to have gotten lost in v3r0em6
- EAC Pull in a bug fix to the AcdGap stuff
- EAC Pull in use of AcdCalibSvc to calibration hits from main branch 

AcdDigi v1r14p2em3: !
- HMK Set veto threshold hold back to 0.3 MIP and update CNO threshold to 20 MIPs as it probably always should have been

System Tests for this version

System Tests results:

Differences in vdg_oneTower test due to changes in VDG source location. Changes in VerticalGamma100MeV due to ACD veto change from 0.1 MIP to 0.3 MIP: All trigger bits with the ROI are lower thus changing the total number of events in the other histograms. The current system tests do not include data with the full LAT (including the calibrated ACD) yet. I've done additional checks running recon myself and looked at the ACD variables in the SVAC ntuple.

The test all_gamma_sixTower has a frontend problem - only the first nine histograms are visibible and the statistics page is blank. I've opened a JIRA.  The plots for SixTower_Data and TowerA_Data are missing for EM p16. I looked at the ones for EM p14 (wrt EM p5). No differences.

Fred version:

v0r99

Pipeline version:

v1.4.3

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r6p0:

Make LicosLaunch die if run directory isn't in rawData.

More & better timestamps in recon chunk script.

Put output from each chunk in a separate subdir of staging dir.

Merge chunks into staging dir, then move merged files to destination.

Merge recon chunks before merit & cal.

Use new AFS staging dir.

Switch rootData to u34.

Split recon/recon into 6 processes to get better rollback granularity.

When recon of some chunks fail, save the output from the successful ones and don't re-run them.

# Don't do merge on single-chunk recon runs. # Disabled! - do do merge.

All file copies & moves performed by recon are attempted up to 5 times, then fail (used to try until the job timed out on copies to local disk, and only once for moves away from local disk or staging).

pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p1
ConfigTables: v3r2p1

TestReport: v3r6p4 (digi & recon reports): EAC and awb
- Added ACD electronics space and ACD trigger quantities.  Also improved descriptions.
- Protect code against time hacks being the same in two events.
- Get GEM discarded counter from context information instead of directly from the GEM.
- Added number of prescaled and dead zone events. Added histogram with delta deadzone events.
- Added check on the consistency between the GEM sequence counter and the number of events in the digi file.
- Added trigger/deadzone/discarded rates for 30 time intervals. In case of problems it may tell us where in a run it happened and speed up troubleshooting.
- Added consistency check on all extended counters so we'll know when the go backwards.
- Added much better error decoding. Will now know what the TEM errors are (TKR FIFO full or not).
- Added trigger arrival time histograms.
- Added ACD MIP peak histograms.

EngineeringModelRoot: v2r2 (SVAC tuple)
- Added a few useful AcdMips variables (sum and max and error flag).
- Added more error flags.
- Updated documention.

Approved: unanimous (Bill not available) - 5/19/06

Added 06.01.2006 for book keeping purposes (for the moment): Bryson's CHS task
- Bryson IExternal/fsw v0r069  -- incorporates DFI, etc. from FSW B0-6-9 with the 24/25-bit fix for the livetime counter.
- Bryson CHS/eventFile v1r0p9 -- bunch of tweaks mostly to rqts file for building on Windows.
- Bryson CHS v2r2                  -- updated checkout package to pull in these two new tags.

 TestReport v3r4p18
- Added histogram for crate number
- Fixed roll-over in GEM discarded events counter.

EngineeringModelRoot v1r17
- Added one more context variable (that I had forgotten).
- Corrected the types of some context variables.

Code Versions

Engineering Model (sim/recon): v6r070329p5

System Tests for this version

System Tests results:

Fred version:

v0r99

Pipeline version:

v1.4.2

GRITS tag (web browsing and task configuration)

glast-ground v0r3p7
grits-gino-web version 0.55 (v0r5p5)
grits-gino version 0.95 (v0r9p5)
grits-gino-xml version 1.42 (v1r4p2)
grits-common version 0.32 (v0r3p2)

online/svac (task defs, scripts):

pipeline tasks:

online: v2r4p1

svac pipeline code and tasks:

code/tasks v3r5p7:
pipelineDatasets v0r3

ISOC code and tasks:

v0r5p0

Apps that run in pipeline:

eLog: v2r3p1
ConfigTables: v3r2p1

TestReport: v3r4p18 (digi & recon reports): *changed*
- Put in a 'isLATTE' flag. Only show SBC histograms for LATTE runs.
- Add in a zoom of the GEM delta event time (1 ms).
- Added (datagram) time stamps for LICOS runs

EngineeringModelRoot: v1r17 (SVAC tuple) *changed*
- Added Context variables.
- Added Acd TkrPoint and Acd Gap variables.
- Updated Acd Poca code to work with new AcdRecon.