Blog from May, 2008

Reason for change

After the file versioning the site name that used to be appended to the file name is now a separate parameter. Unfortunately it was not removed from the file name. Because of this all files registered with the data catalog with L1Proc 1.52 have a '@SLAC_XROOT' incorrectly appended to the file name. While it does not break the data processing or file registration, it does break everything that tries to retrieve the files. Also, the command to register new datasets had a bug (the fileFormat was mistakenly harcoded to "root", instead to the appropriate fileFormat).

We are now also posting the alarms from the TKR monitoring to the Logwatcher. This was already planned and it was deemed unpractical to try to remove this from the code because of the emergency nature of this request. 

Test Procedure

We have processed both MC and real data on the DEV server with this version of L1Proc and verified that files can be correctly retrieved.

Rollback procedure

L1Proc can be rolled back to the previous version (not that it would be very useful).

CCB Jira

SSC-56@JIRA

Details

L1Pipeline v1r53
- removed the site name (@SLAC, @SLAC_XROOT) that was incorrectly appended to the file name
- fixed a bug in the fileFormat field (now it's the actual file Format instead of "root")

AlarmsCfg v1r1p1
- Added config files (alarms) for trackermon quantities. This addresses Jira GDQMQ-119.

Complete set of tags for L1Proc 1.53

Code Versions

GlastRelease (sim/recon) v15r2

ScienceTools (Level 2) : v9r5p4

Science Ops (task defs, scripts):

Level 1 pipeline code and applications running in L1:

svac/L1Pipeline: v1r53*

dataMonitoring/AlarmsCfg: v1r1p1*
dataMonitoring/FastMonCfg: v1r1p11

dataMonitoring/DigiReconCalMeritCfg: v1r1p16
dataMonitoring/Common: v3r1p5
dataMonitoring/FastMon: v3r1p4
datMonitoring/IGRF: v1r0p1

svac/Monitor: v1r1p13
svac/EngineeringModelRoot: v3r14p4
svac/TestReport: v6r8

calibTkrUtil v2r2p2

users/richard/pipelineDatasets: v0r4

ft2Util: v1r2p15

evtClassDefs v0r4

GPLtools: v1r10

Reason for change

The main reason is to include changes requested by the ASP users after OpsSim2. These changes take also into account of the changes that
RSP users have requested in the ASP DB tables definitions. They include also afterglow visualization pages.

Test Procedure

This version has been tested on the DEV server.

Rollback procedure

To rollback to a previous version, the ASP tables would have to be rolled back.

Details

1) Implemented the visualization of the GRB Catalog and GRB Afterglow data including the possibility by the Burst advocate to insert/modify interctiverly the relatives ASP DB tables fields

2) Implemented the display on the web page of only the light curves produced by ASP, taking into account of the changes to the DB tables requested by the RSP users.

Reason for change

Add memory usage improvements.

Test Procedure

This version has been tested on the DEV server.

Rollback procedure

Version 1.0.7 could be reinstalled

CCB Jira

SSC-55@JIRA

Reason for change

Add memory usage improvements.

Test Procedure

This version has been tested on the DEV server.

Rollback procedure

Version 0.1.13 could be reinstalled

CCB Jira

SSC-54@JIRA

Reason for Change

RSP users have requested a number of features so the table definitions have been modified accordingly. Even though these features are not needed by ASP, its sql queries need to be changed thereby requiring code changes.
A few unrelated changes are included: code to generated the light curve data products delivered to the GSSC; code to fill the test statistic column in the lightcurves table; suppression of the info message in PipelineCommand.py; cleaning up of locally staged files for GRB launcher tasks.

Test Procedure

Tested in dev using data in /ASP/TestSims.

Rollback Procedure

Revert to old tables using rollback_tables.sql

Revert to ASP v2r4 + grbASP v4r2

CCB Jira

ssc-53@jira

Details

• New tables:
  • TASKGROUPLIST
• Modified tables:
  • ENERYBANDS
  • FREQUENCIES
  • ROIS
    For each of the three tables above, a column "GROUP_ID" (referencing TASKGROUPLIST(GROUP_ID)) has been added to distinguish ASP-relevant quantities from RSP usages.
  • LIGHTCURVES
    • add column spectral_index_error
• These changes will be implemented using the script modify_tables.sql
• Code changes:
  • AspLauncher v1r3p1
    • use group_id in query of FREQUENCIES table
  • drpMonitoring v1r3p2
    • use group_id in query of ENERGYBANDS table
    • fill test statistic data in LIGHTCURVES table
    • update script and template file for GSSC light curve data product
  • pyASP v3r4p1
    • suppress pipeline server info message
  • grbASP v4r1p2
    • clean up staged files in launchRefinementStream.py and launchAfterglowStream.py

Reason for change

File versioning in the L1 pipeline: The reason for this is that a run will usually be processed piecemeal  (run cut by a downlink/data transfer, missing data showing up, etc). While we only process any contiguous chunk of data once, we always merge up everything every time we process a piece of a run. It's the latest version of the merged file that will be the most complete and which will be of (most) interest. The infrastructure (Data catalog) needs to take this into account. LONE-58@JIRA

Run quality status: L1 should update the run quality table to show the L1 processing status and whether new data have come in and the shifters should look at the monitoring again. LONE-60@JIRA

Test Procedure

We have processed both MC and real data on the DEV/TEST server with this version of L1Proc.

Rollback procedure

Both L1Proc and infrastructure code can be rolled back to the previous versions.

CCB Jira

SSC-50@JIRA

Details

evtClassDefs v0r4
- Update to event classifications for P6_v1 irfs. 

Complete set of tags for L1Proc 1.52

Code Versions

GlastRelease (sim/recon) v15r2

ScienceTools (Level 2) : v9r5p4

Science Ops (task defs, scripts):

Level 1 pipeline code and applications running in L1:

svac/L1Pipeline: v1r52*

dataMonitoring/AlarmsCfg: v1r1p0
dataMonitoring/FastMonCfg: v1r1p11

dataMonitoring/DigiReconCalMeritCfg: v1r1p16
dataMonitoring/Common: v3r1p5
dataMonitoring/FastMon: v3r1p4
datMonitoring/IGRF: v1r0p1

svac/Monitor: v1r1p13
svac/EngineeringModelRoot: v3r14p4
svac/TestReport: v6r8

calibTkrUtil v2r2p2

users/richard/pipelineDatasets: v0r4

ft2Util: v1r2p15

evtClassDefs v0r4*

GPLtools: v1r10

Reason for Change

Updates to code for processing GCN Notices receive via email.

Test Procedure

Tested for dev db tables using GCN Notices sent to the glastgcn account and using test data in /ASP/TestSims on dev.

Rollback Procedure

Revert to grbASP v4r1

CCB JIRA

ssc-52@jira

Details

• grbASP v4r1p1
  • write location of file containing archived notice(s) to db tables
  • add code to skip GLAST test notices and other notices not related to GRB detections (Swift slews etc.)
  • set ingest and archive locations
  • reactivate email notifications from ASP blind search and add UTC version of trigger time

Reason for change

We have a new GR, v15r2, which includes the Pass7 changes and also updates to the GCR algorithm.

The previous L1Proc introduced a new type of monitoring quantity that needed to be propagated to the Tree merging. It was not (oversight). While the testing was done with multi-chunk runs they were not not adjacent and did not trigger this bug. The script is intelligent enough to only do a straight copy in that case. Testing procedures will be changed.

We also have a fix for a memory leak in merging Hiro's TKR monitoring histograms. 

Test Procedure

We have processed both MC and real data on the DEV server with this version of L1Proc.

Rollback procedure

We can roll back to the previous version of L1Proc.

CCB Jira

SSC-49@JIRA

Details

GlastRelease v15r2:
- The GR v15 series is identical to GR v14 except for the Pass 7 stuff.
- Systemtestsfor GR v15r2wrt GR v14r11 show no differences.
- RM diffwith respect to the previous production release, GR v14r11.

svac/Monitor: v1r1p13
- Modfied treemerge.cxx so that the merged quantity is not the largest value, but the last value. This is done in case the DataTransferId has decreasing numbers.
- Added merging of quantity outputnumber (prefix Number_) which is an UInt_t. The merging consists in using the largest value. This is a temporary solution to deal with quantity Number_LastDataTransferId. The proper way to go is to define this quantiy as ValChange and update the data base to deal with ValChange quantities. And that ain't happening anytime soon

calibTkrUtil v2r2p2:
- Fixing a memory leak when merging histograms. 

svac/EngineeringModelRoot v3r14p4:
- Added ObfFilterStatusBits (Gleam filters).

Complete set of tags for L1Proc 1.51

Code Versions

GlastRelease (sim/recon) v15r2*

ScienceTools (Level 2) : v9r5p4

Science Ops (task defs, scripts):

Level 1 pipeline code and applications running in L1:

svac/L1Pipeline: v1r51*

dataMonitoring/AlarmsCfg: v1r1p0
dataMonitoring/FastMonCfg: v1r1p11

dataMonitoring/DigiReconCalMeritCfg: v1r1p16
dataMonitoring/Common: v3r1p5
dataMonitoring/FastMon: v3r1p4
datMonitoring/IGRF: v1r0p1

svac/Monitor: v1r1p13*
svac/EngineeringModelRoot: v3r14p4*
svac/TestReport: v6r8

calibTkrUtil v2r2p2*

users/richard/pipelineDatasets: v0r4

ft2Util: v1r2p15

evtClassDefs v0r3

GPLtools: v1r10

Reason for change

The main reason is to adapt the queries to the table changes described in SSC-47@JIRA for ASP that would otherwise break the prod version.
Other improvements are described below.

Test Procedure

This version has been tested on the DEV server.

Rollback procedure

To rollback to a previous version, the ASP tables would have to be rolled back. Only then we could reinstall version 0.1.2.
These changes affect only the ASP part, not the main processing table.

CCB Jira

SSC-48@JIRA

Details

Reason for Change

New db tables were requested by the GRB group to provide information for the new GRB catalog definition. GCN Notice handling has been improved to provide access to the email texts from GCN. A dev version of the upper limit code has been installed to avoid having to propagate these changes to an ST release.

Also will move to ST v9r5p3 in order to have fixes to irfs/latResponse v2r1p3 for Pass 6 IRFs.

Test Procedure

Built against ST v9r5p3. Tested on /ASP/TestSims data on dev.

Rollback Procedure

Revert to ASP v2r3 and ST v9r5p2. Revert db tables using revision 1.24 of dbCreation.sql script.

CCB Jira

ssc-47@jira

Details

For reference, see the 13 May 2008 Status report to SC Steering committee.

• ASP v2r4
  • AspPolicy v0r5
    • refactor wrapperGenerator.py to obviate need for ASP_PATH and ST_INST env vars to be set
    • remove incorrect location of isoc-installed logging module from PYTHONPATH in wrapperGenerator.py
    • add "$@" to exec command so wrapper scripts will pass command line arguments to python script
  • pyASP v3r4
    • add dev version of UpperLimits.py for use by ASP (superceding ST/pyLikelihood/UpperLimits.py module). This version has a more efficient algorithm for computing the upper limit corresponding to a given confidence limit.
    • add function to resolve nfs paths
  • grbASP v4r1
    • Rewrite SQL queries in dbAccess.py to handle new db tables.
    • Provide more context for prompt burst light curve plots: include data before and after prompt burst phase to plot background levels and overlay Bayesian Blocks light curve solution.
    • add scripts to ingest, parse, and archive GCN Notice emails (ingestEmailNotice.py, ingestNoticeQueue.py)
    • Use GRB_ID to set the stream id for GRB_afterglow tasks (this is already done for GRB_refinement)
  • AspLauncher v1r3
    • Remove dependence on ASP_PATH env var, rely instead on wrapper scripts for environment
    • add code to generate timeintervals db table entries based on current data (intervalAccess.py, insertIntervals.py, AspInsertIntervals.xml)
  • asp_pgwave v1r4p1
    • remove unneeded argument from launchDrpMonitoring command
  • drpMonitoring v1r3p1
    • refactor ROI_ID assignments

• database table changes
  • GRB
    • Changed to conform to new GRB catalog definition. See ASP db table summary.
  • GCNNOTICES
    • Add MISSION, NOTICETYPE, TRIGGER_NUM columns
  • FREQUENCIES
    • Add column TYPE to distinguish between ASP and RSP usage
    • Add column DURATION for duration of intervals in seconds

Reason for change

This is the first version of L1Proc using the GR v14 series. Even though GR v14r11 has a few missing things that are critical for launch (like the filter status bits), it has still been chosen to be the branching point and the first launch candidate release.

GR v14r11 and ST v9r5p4 are also in synch concerning the versions of Root and cfitsio. 

There are also a few bug fixes and updates to the monitoring, in particular we now have the normalized rates for each trigger engine.

In addition, we now have a dedicated TKR monitoring from Hiro. Because of this we use calibTkrUtil v2r2p1 which is a slightly more recent version that what is in GR v14r11. Note that nothing in GR depends or uses calibTkrUtil i.e. it is only included for bookkeeping reasons. This new version of calibTkrUtil will be included in a subsequent GR.

Test Procedure

We have processed both MC and real data on the DEV server with this version of L1Proc.

Rollback procedure

We can roll back to the previous version of L1Proc.

CCB Jira

SSC-44@JIRA

Details

GlastRelease v14r11:
- The GR v14 series has gone through extensive MC based testing by the C&A group. it has also been verified that this GR can process real data. It should also be noted that this GR uses Root 5.18.00b.
- Systemtests for GR v14r11show many differences with respect to the previous production version of GR, but this is as expected.
- RM diff with respect to the previous production release, GR v13r11p9. I will not detail the changes here as there are understandably too many.

Science Tools v9r5p4:
Science Tools has been built against Root 5.18.00b to be consistent with GR v14r11. 

ft2Util: v1r2p15
- Identical to the previous version except that it contains a lot more documentation (added because of the upcoming ft2Util code review).  

calibTkrUtil v2r2p1: 
- New version for TKR monitoring running in the L1 pipeline.  

dataMonitoring/FastMon: v3r1p4
- Added quantity evt_data_transfer_id that contains de data transfer id and fixed meta_context_run_id that now contains the real run id, aka run.startedat. Relevant jira(s): GDQMQ-115

dataMonitoring/FastMonCfg: v1r1p11
- Fix variable type for meta_LATC_master and evt_data_transfer_id to uint32 in baseConfig. Relevant jira(s): GDQMQ-115
- New quantity : evt_data_transfer_id that contains the data transfer id, retrieved from the evt header, via LSEReader.runId(). Relevant jira(s): GDQMQ-115

dataMonitoring/DigiReconCalMeritCfg: v1r1p16
- Added new FastMon trending quantity FastMon_Evt_DataTransferId and FastMon_MetaContext_LATC_Master that have type outputnumber, the last value in the time bin is considered. That addresses Jiras GDQMQ-115 and 107.
- Change variable type object for quantity DataTransferId. Prefix will be Number_ instead of OutF_.  The variable type will be UInt_t instead of Float_t.  That addresses Jira GDQMQ-115.
- Add DataTransferId. That addresses Jira GDQMQ-115.
- Additional monconfig file (monconfig_trackermon_trending.xml) to handle high level tracker monitoring information produced by Hiro Tajima etal. This addresses Jira GDQMQ-119.
- Added new histograms. This addresses Jira GDQMQ-113 and GDQMQ-114.

dataMonitoring/Common: v3r1p5
- Debug message added in the ACD/CAL peds/gains fitting tool. Relevant jira(s): GDQMQ-65.
- Bug fix in the ACD/CAL peds/gains fitting tool (check on a zero division error). Relevant jira(s): GDQMQ-65.

svac/Monitor: v1r1p10
- Added new FastMon trending quantities FastMon_Evt_DataTransferId and FastMon_MetaContext_LATC_Master that are of type outputnumber. JIRAs GDQMQ-115 and GDQMQ-107
- Change variable type object for quantity DataTransferId. Prefix will be Number_ instead of OutF_. New object was defined for that MonSecondListNumber. It is identical to MonSecondListUInt, the the prefix used is Number_ instead of OutUI_.
- Changes in code to handle high level tracker monitoring information produced by Hiro Tajima etal. This addresses Jira GDQMQ-119. Several of the base/classes needed to be changed:
- Added possibility of normalize rates stored in multidimensional variables. This is useful for Trigger Engines, for instance. This addresses the Jira GDQMQ-83.

svac/TestReport: v6r8
- Tracked through JIRA GDQMQ-9and this Confluence page.
- Checking CCSDS time vs event time stamp: JIRA GDQMQ-120.
- Checking GPS second counter: JIRA GDQMQ-120
- Checking extended GEM scalers: JIRA GDQMQ-120.
- Checking number of event consistency with extended GEM sequence counter: JIRA GDQMQ-120.
- Checking livetime consistency: JIRA GDQMQ-120.
- Handling error details in xml: JIRA GDQMQ-85.
- Checking Ground Id: JIRA GDQMQ-120.
- Checking Latc Key: JIRA GDQMQ-120.

org-glast-dataqualitymonitoring : v0r1p14
- Updated ingestion script to support integers

Complete set of tags for L1Proc 1.50

Code Versions

GlastRelease (sim/recon) v14r11*

ScienceTools (Level 2) : v9r5p4*

Science Ops (task defs, scripts):

Level 1 pipeline code and applications running in L1:

svac/L1Pipeline: v1r50*

dataMonitoring/AlarmsCfg: v1r1p0
dataMonitoring/FastMonCfg: v1r1p11*

dataMonitoring/DigiReconCalMeritCfg: v1r1p16*
dataMonitoring/Common: v3r1p5*
dataMonitoring/FastMon: v3r1p4*
datMonitoring/IGRF: v1r0p1

svac/Monitor: v1r1p10*
svac/EngineeringModelRoot: v3r14p3
svac/TestReport: v6r8*

calibTkrUtil v2r2p1* 

users/richard/pipelineDatasets: v0r4

ft2Util: v1r2p15*

evtClassDefs v0r3

GPLtools: v1r10

Reason for change

The telemetry trending application is now being used by the Reports application to fetch data. Each time a plot is fetched a session is created on the server. This seem to cause memory problems over time.

A fix has been added to the infrastructure to invalidate the session once the data is shipped.

A jar file needs to be upgraded to propagate the fix.

Test Procedure

This version has been tested on the DEV server.

Rollback procedure

Version 1.0.6 can be easily put back in place.

CCB Jira

SSC-42@JIRA

Details

Reason for change

Some small user-requested improvements. The most significant is to allow overlay of 1D histograms when 2 runs are selected.

Test Procedure

This version has been tested on the DEV server.

Rollback procedure

Version 0.1.12 can be easily put back in place.

CCB Jira

SSC-41@JIRA

Details

Reason for Change

Updates to asp_pgwave and drpMonitoring for handling sources found by source detection step, position refinement step added for new sources, new implementation for generating epoch-specific ROIs.

Test Procedure

Tested in DEV on data in /ASP/TestSims.

Rollback Procedure

Revert to ASP v2r2, ASP/AspLauncher v1r2, ASP/asp_pgwave v1r3, ASP/drpMonitoring v1r2p1, ASP/grbASP v4r0p1, ASP/pyASP v3r2

CCB Jira

SSC-40@JIRA

Details

• ASP/pyASP v3r3
  • add script (crearoi.py) to generate ROIs from a list of source positions
• ASP/asp_pgwave v1r4
  • Perform position refinement of candidate sources from pgwave using pointfit
  • Use GtApp to run pgwave2D and read parameters from db table
• ASP/drpMonitoring v1r3
  • add script (assignRois.py) to identify sources in fixed ROIs and to run crearoi.py on remaining sources.
  • modify pipeline task to run assignRois as a separate process and to read ROI ids from assignRois for launching subtasks; remove num_RoIs parameter from xml def.

Reason for change

We have added pipeline scripts to use features added in monitoring applications in L1Proc 1.48 (trend alarm output, fit pedestals). In addition, we now have normalized event rates (using PtMcIlwainL).

We have a new tag of ft2Util that enables us to process both data and MC in the pipeline.

We have a new GlastRelease where we have gone back to the original buffer size. This was increased earlier this year to deal with ground high rate test runs, but causes other problems.

Test Procedure

We have processed data on the DEV server with this version of L1Proc.

Rollback procedure

We can roll back to the previous version of L1Proc.

CCB Jira

SSC-39@JIRA

Details

GlastRelease v13r11p9:

Systemtestsshow no differences with respect to the previous production version of GR.

RM diffwith respect to the previous production release, GR v13r11p7. Changes are: 

- ntupleWriterSvc v3r22:
- Reset default buffer size to 32000 bytes
- Get the branch address correct for updated items that were already in the original input TChain
- Handle branch names such as PtPos[3] correctly
- Create branch pointers for input TChain in getTree routine to prevent missing any
- make certain current ROOT directory is saved and restored
- Fix careless typo in requirements file
- SetDirectory when TTree is created in a separate routine
- Patch TChain init in getTree
- Use TChains for input tuples
- Fix to handle vectors like PtPos[3]
- A couple of gcc warnings
- Kludge mechanism to read and ntuple and clone for output as well 

facilities v2r18:
- Add method to expand env variables in a vector of strings facilties::Util::expandEnvVarList 

configData v1r6p6:
- Patch to include cassert for windows 

astro v3r1p6:
- Disable SwigModule since it does not work on Linux
- Mod to Quaternion interface to protect against unnormalized intialization
- Needed to make python runtime library available for clients.
- Use new SwigModule instead!
- Undo python module interface in build, since it is not allowed to use SwigPolicy
- Implement python module interface in build

Gleam v6r33p17gr1:
- Remove FlavorRibbon for the current L1 P2 branch 

ft2Util v1r2p14:
- MC mode does not interpolate ATT entries missing at the edges of the run, but just replace first non 0 from M7.
- Changed requirements for fitsgen.
- Force to add padded entries also in Gleam mode.
- Check on the Quaternion normalization, added option in the command line to set some paramters.

Science Tools v9r5p2:
- To synch up with ASP: SSC-36

svac/EngineeringModelRoot v3r14p3:
- Added two ACD energy variables useful for future filter studies. 

dataMonitoring/FastMonCfg v1r1p9:
- Bug fix : histograms added in the previous tag are also now in the MergeHisto text file. Relevant jira(s): GDQMQ-113 and GDQMQ-114
- Added histograms of the condition arrival time for other trigger types where a specific trigger opens the window.
- Added histograms for condition summary word with a specific trigger type. Relevant jira(s): GDQMQ-113 and GDQMQ-114
- Added quantities meta_LATC_master, aka meta.hardwareKey, and meta.softwareKey for good measure. That's the hardware configuration for the current run. Relevant jira(s): GDQMQ-107

dataMonitoring/DigiReconCalMeritCfg  v1r1p11:
- Added possibility of trending and histograming normalized rates (from merit). This addresses the Jira GDQMQ-83.
- Added trending and histograms for other normalized rates. This addresses the Jira GDQMQ-83.

dataMonitoring/Common v3r1p3:
- Minor bug fix in the ACD/CAL peds/gains fitting tool (now if an histogram is missing in the input file, the program issues an error instead of crashing). Relevant jira(s): GDQMQ-65 (the above mentioned ACD/CAL peds/gains fitting functionalities are now implemented in the pipeline).
- Bug fix in the ACD/CAL peds/gains fitting tool (the fit is not performed when there are no entries in a histogram). Relevant jira(s): GDQMQ-65

dataMonitoring/FastMon v3r1p3:
- Added quantities meta_LATC_master, aka meta.hardwareKey, and meta.softwareKey for good measure. This is the hardware configuration for the current run. Relevant jira(s): GDQMQ-107

svac/Monitor v1r1p7:
- Added possibility of normalize rates stored in multidimensional variables. This is useful for Trigger Engines, for instance. This addresses the Jira GDQMQ-83.
- Added possibility of trending and histograming normalized rates (from merit). This addresses the Jira GDQMQ-83
- Decrease resolution of ACD nice plots (gif); from 1100x900 to 600x600.

Complete set of tags for L1Proc 1.49

Code Versions

GlastRelease (sim/recon) v13r11p9*

ScienceTools (Level 2) : v9r5p2*

Science Ops (task defs, scripts):

Level 1 pipeline code and applications running in L1:

svac/L1Pipeline: v1r49*

dataMonitoring/AlarmsCfg: v1r1p0
dataMonitoring/FastMonCfg: v1r1p9*

dataMonitoring/DigiReconCalMeritCfg: v1r1p11*
dataMonitoring/Common: v3r1p3*
dataMonitoring/FastMon: v3r1p3*
datMonitoring/IGRF: v1r0p1

svac/Monitor: v1r1p6*
svac/EngineeringModelRoot: v3r14p3*
svac/TestReport: v5r8

users/richard/pipelineDatasets: v0r4

ft2Util: v1r2p14*

evtClassDefs v0r2

GPLtools: v1r10