Reason for change
This is really a request for L1Proc 1.48. It turns out you can't edit the title of a News page in Confluence. Silly constraint .....
We have a new GlastRelease that enables us to run the Gleam version of the HIP-filter on real data. This is done by setting the CNO bit for all ACD digis attached to a FREE board that had the GEM CNO bit set. This is only necessary for real data as we know this on a per PMT basis for MC. This also means the CNO bit is now set in the GltWord for real data.
There are also some bug fixes and general updates to the monitoring. One new feature is the capability to trend quantities once per run (previous trending only used fixed time intervals like 15 seconds or 5 minutes). This version of L1Proc will trend the number of alarms and errors per run.
Note that in this version of L1Proc we continue using the version of ft2Util that works with MC as L1 stress testing continues.
Test Procedure
We have processed data on the DEV server with this version of L1Proc.
Rollback procedure
We can roll back to the previous version of L1Proc.
CCB Jira
Details
GlastRelease v13r11p7:
Systemtestsshow no differences with respect to the previous production version of GR. This proves the lack of real data runs in the system tests as the CNO bit is now set in the GltWord and the HIP-filter is running.
ldfReader v5r0:
- Add AcdDigi::m_cable to stash the cable number associated with this ACD PMT, so we can refer to it in LdfConverter and decode the CNO vector bits: ACD-19.
LdfConverter v4r1p1:
- Set the CNO bits in the ACD digis using the GEM CNO word: ACD-19.
Both of these issues are in fact related to a solution of GRINF-43.
svac/EngineeringModelRoot v3r14p1:
- Added data transfer ID: SVAC-113.
dataMonitoring/AlarmsCfg v1r1p0:
- Added a configuration file for the alarm post-processing tool. Relevant jira(s): GDM-27. The corresponding pipeline script to make use of this capability will come in the next version of L1Proc.
dataMonitoring/FastMonCfg v1r1p6:
- Bug fix : histograms added in the previous tag are also now in the MergeHisto text file. Relevant jira(s): GDQMQ-113and GDQMQ-114.
- Added histograms of the condition arrival time for other trigger types when a specific trigger opens the window.
- Added histograms for condition summary word with a specific trigger type. Relevant jira(s): GDQMQ-113 and GDQMQ-114.
- Added quantities meta_LATC_master, aka meta.hardwareKey, and meta.softwareKey for good measure. That's the hardware configuration for the current run. Relevant jira(s): GDQMQ-107
- Added new histograms in MergeHisto file: DeltaEventTime_TH1, DeltaEventTime_Zoom_TH1 DeltaWindowOpen_TH1 DeltaWindowOpen_Zoom_TH1, PrescaledDelta_TH1. Relevant jira(s): GDQMQ-102.
- Added quantities gem_delta_event_time, gem_delta_window_open_time, PrescaledDelta, and corresponding histograms to match the DigiEOR. Relevant jira(s): GDQMQ-102.
- Added cut on sollicited triggers for all the plots. Relevant jira(s): GDQMQ-49.
- Added cut for the corresponding trigger primitive when plotting the condition arrival time. Relevant jira(s): GDQMQ-108.
- Added cuts for events with 0 hits for CalX_Total_NHit_TH1 and CalX_NHit_TH1. Relevant jira(s): GDQMQ-32.
- Added quantity new_second that is True for the first event for which the second has changed, to be used in fastmon trending. Relevant jira(s): GDQMQ-103.
- Updated cut on plots for clock ticks deviation to 20MHz. Relevant jira(s): GDQMQ-104 and 105.
- Added quantities spacecraft_zra, spacecraft_zdec. They're calculated from M7 in pSCPosition from the quaternion. Relevant jira(s): GDQMQ-62.
- Added quantities spacecraft_roll, spacecraft_pitch, spacecraft_yaw. They're calculated from M7 in pSCPosition from the quaternion. Relevant jira(s): GDQMQ-62.
dataMonitoring/Common v3r1p1:
- Minor bug fix in the alarm postprocessing tool (wrong import).
- "_TH1" added to the postprocessing tool output histograms name, as requested by Max. Relevant jira(s): GDM-27.
- Added a post-processing tool to make distributions of the alarm outputs (requires dataMonitoring/AlarmsCfg v1r1p0 or later). Relevant jira(s): GDM-27.
- Added the plot of the chisquare probability into the fitting tool for CAL and ACD pedestals/gains. Relevant jira(s): GDQMQ-65. The corresponding pipeline script will be added in the next version of L1Proc.
- +-inf removed from the alarm limits. Relevant jira(s): GDM-80.
dataMonitoring/FastMon v3r1p2:
- Added quantities meta_LATC_master, aka meta.hardwareKey, and meta.softwareKey for good measure. This is the hardware configuration for the current run. Relevant jira(s): GDQMQ-107
- Added a function to process the PrescaledDelta in pGEMcontributionBase. Relevant jira(s): GDQMQ-102.
- Added the processing of the quantity new_second that is True for the first event for which the second has changed, in python/pEvtMetaContextProcessor.py. new_second is to be used in fastmon trending. Relevant jira(s): GDQMQ-103.
- Improved the calculation of clocktics_dev_20MHz : It is now done only if the context incomplete flag is not set. Initialization is done using the context.previous time hack. Better integration with Time tone error handler. Relevant jira(s): GDQMQ-104and 105.
- pSCPostion now retireve the satellite z-axis Ra and Dec from the attitude quaternion into, information are saved via pGeomagProcessor. Relevant jira(s): GDQMQ-62.
- pSCPostion now convert the attitude quaternion into Euleur angles (pitch,roll and yaw), information are saved via pGeomagProcessor. Bug fix when reading the quaternion in pM7Parser. Relevant jira(s): GDQMQ-62.
- pEvtMetaContextProcessor now processes the TimeTone errors, via the ErrorHandler. Error handler documentation was updated. Relevant jira(s): GDQMQ-70.
- pDataProcessor modified as to use the internal event counter (instead of the GEM event sequence) as event number for the error handler ONLY when processing an .ldf file. This change has no impact on processing .evt file. Relevant jira(s): GDQMQ-96.
svac/Monitor v1r1p5:
- Added quantities source_gps and new_second in FastMon trending to calculate the rate of GPS out of lock. This addresses Jira GDQMQ-103.
- Change MCOktTest by MC in MonValueImpls_t.cxx. This addresses Jira GDQMQ-111.
- Added merit objects that were already present in the MonObjFactory.cxx committed in v1r1p2. The code in v1r1p2 did not compile because those objects were missing... Those objects are needed for the normalization of the merit rates. The overall normalization is still not ready. It will have to wait to a later version.
- Added sourceGPS. This addresses jira GDQMQ-103.
- Added quantity to check completeness of time tone. This addresses Jira GDQMQ-104.
- Added the following quantities from FastMon trending : FastMon_SpaceCraft_Pitch, FastMon_SpaceCraft_Roll, FastMon_SpaceCraft_Yaw, FastMon_SpaceCraft_ZaxisRA, FastMon_SpaceCraft_ZaxisDec. GDQMQ-62.
- Bug fix addressed by Jira GDQMQ-98.
dataMonitoring/DigiReconCalMeritCfg v1r1p9:
- Implemented quantity FastMon_MetaContext_GPS_OUTofLock that counts the rate of seconds in which the GPS is out of lock, GDQMQ-103. Added a cut to remove solicited triggers in FastMon trending and FastMon trending histograms, GDQMQ-49. This is related to Jiras GDQMQ-49and GDQMQ-103.
- Bunch of merit new rates and histograms of rates. This is related to Jira GDQMQ-83.
- Change MCOktTest for MC in config files. This addresses Jira GDQMQ-111.
- Changes to address jiras GDQMQ-32 and GDQMQ-110.
- Added histograms to be used for calculation of LAC thresholds. This addresses jira GDQMQ-109.
- Change FastMon_NTkrHits_TowerPlane[16][36] from counter to rate. This addresses Jira GDQMQ-77.
- Limited range of histograms with distributions of ACD, CAL pedestals and optical gains. This addresses Jira GDQMQ-106.
- Added sourceGps info. This addresses Jira GDQMQ-103.
- Added a check of completeness of time tone. This addresses Jira GDQMQ-104.
- Added the following quantities from FastMon trending: FastMon_SpaceCraft_Pitch, FastMon_SpaceCraft_Roll, FastMon_SpaceCraft_Yaw, FastMon_SpaceCraft_ZaxisRA, FastMon_SpaceCraft_ZaxisDec. GDQMQ-62.
- Upgrade in merit quantities proposed in Jiras GDQMQ-81and GDQMQ-101.
- Bug fix in Trigger Rates corrected with the LiveTimeFraction. That is in a way related also to Jira GDQMQ-98.
- Bug fix addressed by Jira GDQMQ-98.
Complete set of tags for L1Proc 1.48
Code Versions
GlastRelease (sim/recon) v13r11p7*
ScienceTools (Level 2) : v9r5
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r48*
dataMonitoring/AlarmsCfg: v1r1p0*
dataMonitoring/FastMonCfg: v1r1p9*
dataMonitoring/DigiReconCalMeritCfg: v1r1p9*
dataMonitoring/Common: v3r1p1*
dataMonitoring/FastMon: v3r1p3*
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r1p5*
svac/EngineeringModelRoot: v3r14p1*
svac/TestReport: v5r8
users/richard/pipelineDatasets: v0r4
ft2Util: v1r1p44
GPLtools: v1r10
Reason for Change
The non-ASP specific parts of the pgwave package was added to ST v9r5p2 so that the Catalog group can have prebuilt binaries. The ASP package has been renamed asp_pgwave to avoid CMT-related naming conflicts, and the compiled executables (now in ST/pgwave) have been removed. UpperLimit code from pyLikelihood v1r5p4 will be used instead of replicated code in grbASP v4 and drpMonitoring v1r2. Some minor, unrelated changes to the ASP code will also be included.
Testing Procedure
ST v9r5p2 passed the standard RM tests. Tested on dev pipeline using /ASP/TestSims data.
Rollback Procedure
Revert to ST v9r5, ASP/pgwave v1r2, ASP/grbASP v4, ASP/drpMonitoring v1r2, ASP/AspLauncher v1r1
CCB JIRA
Details
- ST v9r5p2
- ASP/AspLauncher v1r2
- use asp_pgwave v* in req file
- pass streamId to PGWave task (unrelated to ST v9r5p2 change)
- use afs installation of catalog files (unrelated to ST v9r5p2 change)
- use ST v9r5p2 in xml def
- ASP/asp_pgwave v1r3
- Names for flaring sources have been changed to use Catalog group convention based on IAU standards. (unrelated to ST v9r5p2 change)
- xml task def and os.environ references in python scripts were updated to point to ASP_PGWAVEROOT. (unrelated to ST v9r5p2 change)
- ASP/grbASP v4r0p1
- use UpperLimits from pyLikelihood v1r5p4 instead of local version
- save time history log-probability evaluations for each data delivery epoch (unrelated to ST v9r5p2 change)
- ASP/drpMonitoring v1r2p1
- use UpperLimits from pyLikelihood v1r5p4 instead of local version
- make folder containing pgwave output configurable via pipeline command line argument (unrelated to ST v9r5p2 change)
Reason for change
The major change is the change of the query to fetch the runs in the welcome page. It is now possible to select runs on different times: data taking, delivery and processing.
Test Procedure
This version has been tested on the DEV server.
Rollback procedure
Version 0.1.11 can be easily put back in place.
CCB Jira
Details
Reason for change
For next week's stress testing of the pipeline we need to process MC. As explained in the previously accepted L1Proc 1.45 request, the latest version of ft2Util can no longer do this. So for the test phase we would like to temporarily revert back to the old version of ft2Util. This new version of L1Proc will not be used to process any real data. After the stress testing we will make a new L1Proc, v1.48, which will be an identical copy of the current PROD version of L1Proc. This request covers both of these changes.
The reason we can not do the stress testing on the DEV pipeline server is that only PROD is on the new Oracle server.
Test Procedure
The old tag of ft2Util has already been in Prod. This version of L1Proc is only for next week's stress testing.
Rollback procedure
We can roll back to the previous version of L1Proc.
CCB Jira
Details
ft2Util v1r1p44:
- Version of ft2Util that was in L1Proc 1.44.
Complete set of tags for L1Proc 1.47
Code Versions
GlastRelease (sim/recon) v13r11p6
ScienceTools (Level 2) : v9r5
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r47
dataMonitoring/AlarmsCfg: v1r0p4
dataMonitoring/FastMonCfg: v1r0p10
dataMonitoring/DigiReconCalMeritCfg: v1r0p15
dataMonitoring/Common: v3r0p13
dataMonitoring/FastMon: v3r0p13
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r0p13
svac/EngineeringModelRoot: v3r14
svac/TestReport: v5r8
users/richard/pipelineDatasets: v0r4
ft2Util: v1r1p44*
GPLtools: v1r10
Reason for change
These changes address some issues that arose during OpsSim2: processing and archiving of GCN Notices via email, identifying non-detections and computing upper limits for GRB candidates, and keeping track of log-probability distributions for calibrating the blind search threshold.
Test Procedure
These changes have been tested on data in /ASP/TestSims (datacatalog) and /Data/OpsSim2/Level1 (datacatalogPROD) on the dev pipeline.
Roll-back Procedure
Revert to grbASP v3r2, pyASP v3r1
CCB Jira
Details
- grbASP v4
- added email GCN notice processor
- update GCN_Notice.tpl template file to exclude uneccesary fields and those filled automatically by GCN
- move GCN Notice generation to GRB_refinement task
- use new datatypes for registering afterglow plots with datacatalog
- flag non-detections from Bayesian Blocks (BB) analysis of event arrival times
- Perform TS and upper limit calculations and skip TS map generation BB non-detections
- fill TS_VALUE, UPPER_LIMIT columns in GRB db table
- Save histograms of log-probabilities for each run of Blind Search task for threshold calibration
- modify xml task defs so that target folder for datacatalog.registerDataset(...) is configurable at stream creation
- database table modifications
- GRB
- add TS_VALUE, UPPER_LIMIT, GCN_NOTICE_FILE columns
- GRB_ASP_CONFIG
- add NOMINAL_WINDOW column
- GCNNOTICES
- add GCNNOTICE_ID column and make it the primary key
- create sequence GCNNOTICES_SEQ
- GRB
- pyASP v3r2
- add code to compute vector dot and cross-products for SkyDir objects in order to facilitate GRB_PHI and GRB_THETA calculations
Reason for change
ft2Util had the LAT mode from the Magic 7 file as an 'unsigned int' instead of an 'signed int'. This caused a crash in fitsGen for mode values of 0 (it apparently causes some casting problem). And this weekend's data contained mode zero data.
Please note that the complete range of LAT modes and their meaning is GD proprietary and ITAR controlled and I have no idea what they all are 
Test Procedure
We have processed real data with mode zero on DEV with this version of L1Proc.
Rollback procedure
We can roll back to the previous version of L1Proc.
CCB Jira
Details
ft2Util v1r2p9:
- Observing mode is declared as a signed integer variable. In the ft2Util it was an unsigned one. This generated and error in FITSIO due to the typecasting only when the value of the observing mode is 0. LONE-45
Complete set of tags for L1Proc 1.46
Code Versions
GlastRelease (sim/recon) v13r11p6
ScienceTools (Level 2) : v9r5
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r45
dataMonitoring/AlarmsCfg: v1r0p4
dataMonitoring/FastMonCfg: v1r0p10
dataMonitoring/DigiReconCalMeritCfg: v1r0p15
dataMonitoring/Common: v3r0p13
dataMonitoring/FastMon: v3r0p13
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r0p13
svac/EngineeringModelRoot: v3r14
svac/TestReport: v5r8
users/richard/pipelineDatasets: v0r4
ft2Util: v1r2p9*
GPLtools: v1r10
Reason for change
This version collects various changes that were requested during OpsSim2. It also includes some minor bug-fixes that were found in testing against ST v9r5.
Test Procedure
These changes have been tested in dev pipeline on test data in /ASP/TestSims in the dev datacatalog and on OpsSim2 data.
Roll-back Procedure
Revert to ASP v2r1 + grbASP v3r1p1
CCB Jira
Details
Here is a list of the proposed subpackage tags along with summaries of the changes so far:
- AspLauncher v1r1
- set pgwave process stream ids using nMetStart + offset
- add option to set datacatalog implementation in xml for testing using PROD data in DEV pipeline
- AspPolicy v0r4p1
- use GPLtools/prod in PYTHONPATH instead of GPLtools/L1prod
- drpMonitoring v1r2
- use pgwave_streamId instead of process ids
- modify source selection to exclude sources not found by pgwave that are imported from POINTSOURCES table from previous epochs
- add option to set datacatalog implementation in xml
- grbASP v3r2
- overlay plotting of wcs coordinate grids on counts map plots
- bug-fix for pixel offset in interpreting wcs coordinates
- use new gttsmap parfile interface
- add option to set datacatalog implementation in xml
- pgwave v1r2
- propagate streamIds to drpMonitoring streams
- register data products by stream id
- use new gtselect par file interface
- add option to set datacatalog implementation in xml
- pyASP v3r1
- make FileStager.finish() a separate function that can be called outside of destructor
- move to numpy from numarray in SkyCone.py
Reason for change
On Monday we will process 5 days worth of real data from the LAT from Ops Sims and ETEs from Thursday-Monday. These changes are needed to properly process this data.
We have a new GlastRelease that correctly deals with the run number for data in pipeline 2. We have a new version of ft2Util with improved interpolations and which also avoids NaN seen in Ops Sim 2 in addition to dealing with gaps in the event data. And finally there is a new version of the SVAC ntuple that no longer runs code that was producing things we don't use anymore in pipeline 2.
Note that because of the way the Magic 7 file is generated when we make L0 files from MC, the validity time of the Magic 7 file is sometimes a little shorter than the event data, we have had to make fake M7 entries at the start and end of MC runs. This is because Gleam will stop if an event is outside the corresponding FT2_Gleam validity time. It doesn't matter if it's a tiny fraction of a second. Because of technical reasons, FSW uses the start of the Magic 7 file to seed the absolute event time, padded Magic 7 files can not be directly produced. These fake Magic 7 entries caused NaN as seen in Ra and Dec in Ops Sim 2. This version of ft2Util does not make these fake Magic 7 entries. It will, thus, only work properly with real data where this issue of short Magic 7 files do not appear. It is also important to gain experience with real Magic 7 files and real data losses.
Test Procedure
We have processed real data on DEV with this version of L1Proc.
Rollback procedure
We can roll back to the previous version of L1Proc.
CCB Jira
Details
GlastRelease v13r11p6:
Systemtests show no differences wrt to current GR v13r11p4 in L1Proc.
lsfData v2r0:
- v2r0 17-Apr-2008 HMK Add RunInfo::dataTransferId to store the runid in response to JIRA ROOT-34.
ldfReader v4r0:
- v4r0 17-Apr-2008 hmk JIRA ROOT-34, change default method for setting the runid, to use ctx.run.startedAt, rather than LSEReader::runid()
- v3r34 15-Apr-2008 awb Updated flags for time stamp calculation: JIRA GRINF-46.
digiRootData v10r3:
- v10r3 17-Apr-2008 HMK add in RunInfo::dataTransferId to address JIRA ROOT-34
- v10r2 31-Mar-2008 TU HMK tagging Tracy's update to TkrDigi to use the new DigiObjectManager
RootConvert v1r30:
- v1r30 17-Apr-2008 HMK address JIRA ROOT-34 to add in RunInfo::dataTransferId
LdfConverter v4r0:
- v4r0 17-Apr-2007 HMK JIRA ROOT-34 adding new JO parameter OldStyleRunId which defaults to false, but when set to true, will obtain the runId the same way I&T did, rather than the method that is used for flight via the ctx.run.startedAt item
AnalysisNtuple v2r32gr2:
- v2r32gr2 16-Apr-08 HMK Include patch to EventAuditor on this branch
Gleam v6r33p17: Note that all but one of these new tags are old (pre-freeze). Most do not affect pipeline job options.
- v6r33p17 awb 16-Apr-08 Added 'TriRowBitsAlg' to pipeline JO. JIRA GRINF-34.
- v6r33p16 HMK 12-Mar-08 Comment out LATCalibRoot in requirements file like it should be
- v6r33p15 HMK 12-Mar-08 Update calibrations.txt to set FlavorRibbon="ideal"
- v6r33p14 HMK 10-Mar-08 Add use of CalibSvc so that when building Gleam CalibSvc will be as well includes Toby's updates to calibration.txt
- v6r33p13 AWB 15-JAN-08 Yet another filter JO update for pipeline.
- v6r33p12 AWB 15-JAN-08 Filter JO update for pipeline.
- v6r33p11 LSR 15-Jan-08 Another little fix to jobOptions.txt
- v6r33p10 LSR 15-Jan-08 Add EventAuditor to basicOptions. Add placeholder for UserAlg: Sequencer/User
- v6r33p9 AWB 11-JAN-08 Updated I&T mc.txt with correct ACD calibrations.
- v6r33p8 AWB 07-JAN-08 Updated I&T mc.txt with trigger config.
svac/EngineeringModelRoot v3r14:
- Commented out TKR histogram code which produce histograms we don't use in pipeline 2.
- Not building extra applications not used by the pipeline by default anymore.
- Added AcdCnoMap. JIRA SVAC-110.
ft2Util v1r2p8:
- ORB interpolation via Parabola or the method used by gtbary: LONE-34
- ATT interpolation via SLERP: LONE-35
- Flag for quality of the interpolations: LONE-36
- Digi Time calibration: LONE-33
- Handling of Digi file Gaps: LONE-37
- Magic7 read in order to have ATT and ORB entries right at Tstart of a FT2 bin: LONE-38
- Corrected interpolation ATT flag on output file: LONE-36
- Correction of the Attitude interpolation flag in the textual version of the ft2 file. LONE-40
- Interpolation in Attitude entries are often necessary even if the Magic7 file has no gaps. When it is not possible to use SLERP (specially at the end or beginning of a run),
quaternion etrapolations via angular velocities may make crash Gleam. This can be fixed avoiding to pad entries at the edges of the ft2 file, and simply widening the time span
give by run start and stop: LONE-41
- Correction apported in v1r2p6 generated a bug in the code reading Magic7 file, when the time span between first Digi event and firs line in the Magic7 file is larger than 1 second: LONE-42
- ft2Utils checks that RunId in the Gasp file match RunId of the DigiFile: LONE-43
Complete set of tags for L1Proc 1.45
Code Versions
GlastRelease (sim/recon) v13r11p6*
ScienceTools (Level 2) : v9r5
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r45*
dataMonitoring/AlarmsCfg: v1r0p4
dataMonitoring/FastMonCfg: v1r0p10
dataMonitoring/DigiReconCalMeritCfg: v1r0p15
dataMonitoring/Common: v3r0p13
dataMonitoring/FastMon: v3r0p13
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r0p13
svac/EngineeringModelRoot: v3r14*
svac/TestReport: v5r8
users/richard/pipelineDatasets: v0r4
ft2Util: v1r2p8*
GPLtools: v1r10
Reason for Change
Burst advocates requested that the counts spectra for GRBs also show the residuals of the model fit. In addition, the y-axis range needed to be set to the data range, rather than be allowed to autoscale to the (possibly very low-level) model components.
Test Procedure
The new scripts have been run on the OpsSim2 data.
Roll-back Procedure
Revert to ASP/grbASP v3r1
CCB Jira
Details
Here is an example of the new plot:
Reason for Change
There is a bug in ScienceTools v9r4p2 (in Likelihood v13r12p1 and earlier) for which very short time intervals of data (less than 1 sec) are not handled correctly in the livetime calculation. This bug was uncovered in the analysis of a sub 1 second burst during OpsSim2.
ST v9r5 also has a new version of gttsmap (Likelihood v13r14) that allows for more general projections on the sky. This will be useful for plotting error contours for bursts at high Celestial or Galactic latitudes.
Test Procedure
The fix version, Likelihood v13r13, and current release version, Likelihood v13r14 now in ST v9r5, have passed the ST unit and system tests. ASP has been run on a standard test dataset in /ASP/TestSims in the dev datacatalog; it has also been run on the OpsSim 2 data to verify that the short-time scale GRB is handled correctly.
Roll-back Procedure
Reset the ST_INST pipeline stream arguments to point to the v9r4p2 build. Reset the ST_INST environmnent variable to point to v9r4p2 and rebuild the ASP package in the prod area.
CCB Jira
Details
Various code changes are required in ASP:
- grbASP/python/refinePosition.py: use new gttsmap pfile parameters
- pgwave/python/getPgwDataInputData.py: use new gtselect pfile parameters
- drpMonitoring/python/fitEnergyBand.py: use pyLikelihood/python/UpperLimits.py instead of local implementation
Reason for change
The major change is the change of the database schema (see ssc-15@Jira).
The infrastructure of the web front-end is now much more dynamic when new data types are added. It is driven by database tables so that when a new type of alarm is added the change can be easily propagated to the web front-end.
Test Procedure
This version has been tested on DEV and TEST (the new database server).
Rollback procedure
Version 0.1.10 can be easily put back in place but along with it we have to rollback the db schema.
CCB Jira
Details
Reason for change
The main change is adding a set of indexes to the tables to improve the ingestion time.
These indexes also improve the cleanup time AND avoid the conflicts generated when multiple ingests were ran simultaneously.
The schema is also changed to allow different processes (Digi, Recon, FastMon, etc) to ingest data at different frequencies (like 15 secs, run level, 5 mins, etc).
Test Procedure
These changes have been added to the DEV and TEST databases. We have an updated ingestion script that is happily running on DEV and TEST (this one for stress testing the new database). A version of the web front end is deployed on the Development server and is working fine when looking at the DEV and TEST dbs.
Rollback procedure
To rollback we would have to undo the schema changes.
There is one change to the schema that breaks the existing version of the web front-end and the ingestion script are two. It's a column being removed to one of the tables.
It should be pretty easy to add back this column and populate it.
CCB Jira
Reason for change
We have added new quantities, renamed a few for consistency reasons, fixed a few minor bugs.
Test Procedure
We have processed data on DEV with this version of L1Proc.
Rollback procedure
We can roll back to the previous version of L1Proc.
CCB Jira
Details
dataMonitoring/AlarmsCfg v1r0p4
- Added doc/release.notes: GDQMQ-64
dataMonitoring/FastMonCfg: v1r0p10
- Updated the MergeHistos_FastMon.txt file with the 3 Histograms for Clock tick deviation to 20MHz with the digiEOR. Relevant jira(s): GDQMQ-27
- Synchronized Histograms for Clock ticks deviation to 20MH with the digiEOR. Relevant jira(s): GDQMQ-27
- Bug fix on the previous fix. Numpy uint8 is interpreted by ROOT as Char_t. Original type of the quantities was enum, that is compatible with any uint type. So DFI enum types are now uint16 and the monitoring application is happy. Relevant jira(s): GDQMQ-80
- Fix the type of the meta_context quantities according to DFI documentation. Relevant jira(s): GDQMQ-80
- Add the quantity clocktics_dev_20MHz that counts the number of tics between 1pps and gives the deviation with respect to expected 20MHz, and 2 associated histograms in the config configuration file. Relevant jira(s): GDQMQ-27
- Added meta_context_current_earlyevent and meta_context_previous_earlyevent in the baseConfig configuration file. Relevant jira(s): GDQMQ-70 (Ghttps://jira.slac.stanford.edu/browse/DQMQ-70)
- Bug fix in the text configuration file for the merging of FastMon histograms. Relevant jira(s): GDQMQ-71
- Release notes updated with references to the relevant jira. No change in any of the functionalities at all. Relevant JIRA: GDQMQ-64
dataMonitoring/Common: v3r0p13
- The set of tools for fitting ACD pedestals and CAL peds/gains is now in a reasonable shape, with many improvements and bug fixes. Not yet in the pipeline, so no change in the basic functionality at the moment. Relevant jira(s): GDQMQ-65
- More work done on the fitting tool for the CAL pedestals/gains. Some work done on getting the normalization and rms values right for the modified gaussian function. Not yet in the pipeline, so no change in the basic functionality at the moment. Relevant jira(s): GDQMQ-65
- Bug fix in the alarm handler (a variable in pAlarmBaseAlgorithm was referenced before assignment). Relevant jira(s): GDQMQ-78
- Handling of error bars into the alg__values algorithm. Relevant jira(s): GDQMQ-57
dataMonitoring/FastMon: v3r0p13
- Add the quantity clocktics_dev_20MHz that counts the number of tics between 1pps and gives the deviation with respect to expected 20MHz in pEvtMetaEventProcessor.py. Relevant jira(s): GDQMQ-70
- Spacecraft latitude and longitude are now in degrees instead of radian. Conversion is done in pSCPosition. Relevant jira(s): GDQMQ-63
- Added meta_context_current_earlyevent and meta_context_previous_earlyevent in pEvtMetaEventProcessor.py. Relevant jira(s): GDQMQ-70
- Bug fix in the script generating the text configuration file for the merging from the FastMon xml configuration file. Relevant jira(s): GDQMQ-71
- Release notes updated with references to the relevant jira. No change in any of the functionalities at all. Relevant JIRA: GDQMQ-64 (Ghttps://jira.slac.stanford.edu/browse/DQMQ-64)
svac/Monitor: v1r0p13
- Add FastMon trending quantity meta_context_open_modechanges to trend the LAT mode changes as requested by GDQMQ-13 (Ghttps://jira.slac.stanford.edu/browse/DQMQ-13).
- Fix meta_context_open_modechanges and meta_context_open_mode variable types according to GDQMQ-80.
- Changes to address issue reported in Jira GDQMQ-74
- Add FastMon trending quantity meta_context_open_mode to trend the LAT mode as requested by GDQMQ-13.
- Corrected bug in MergeHistFiles.cxx: GDQMQ-88
- Corrected bugs in MonInput_DatagramInfo.cxx, and MonInput_Delta_CCSDSTime_EvtTime.cxx: GDQMQ-89
- Corrected issue addressed by GDQMQ-84.
dataMonitoring/DigiReconCalMeritCfg: v1r0p15
- Add FastMon trending quantity "meta_context_open_modechanges" to trend the LAT mode changes as requested by GDQMQ-13.
- Change of names in config files to keep consistency: GDQMQ-76.
- Changes in the config files to address the Jiras: GDQMQ-74, GDQMQ-75, GDQMQ-77.
- Add FastMon trending quantity "meta_context_open_mode" to trend the LAT mode as requested by GDQMQ-13 (Ghttps://jira.slac.stanford.edu/browse/DQMQ-13).
- Change of names in OBF variables in digi trending job to be consistent with the new filter names: GDQMQ-67
- Added new quantities in digi trending (monconfig_digi_trending.xml): GDQMQ-90
- Added ACD pedestals in digi CalLong trending/histos. This addresses JIRA GDQMQ-59.
- Added actual distributions of pedestals for ACD and CAL and the optical gains. Those distributions will be fitted later on, extracting the meaningful quantities Mean/RMS; that should be comparable to the truncated ones: GDQMQ-91
- Level of truncation is decreased from 10%, per side of distribution, down to 2%. In this way the RMS from the truncation will be sensitive to outliers whose fraction is more than 2% of total number of events: GDQMQ-92.
- Change in monconfig_recon_trending.xml to address problem described in GDQMQ-97.
- Live time fraction related quatities removed from monconfig_digi_trending.xml
svac/TestReport: v5r8
- New file release.notes added: GDQMQ-64
- xml utilities included. Changed range for histos.
ScienceTools v9r5
- Synching up with ASP, SSC-17.
- Affects makeFT1 (used to make FT1 and LS1) makeFT2, and makeLS3.
Complete set of tags for L1Proc 1.44
Code Versions
GlastRelease (sim/recon) v13r11p4
ScienceTools (Level 2) : v9r5*
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r44*
dataMonitoring/AlarmsCfg: v1r0p4*
dataMonitoring/FastMonCfg: v1r0p10*
dataMonitoring/DigiReconCalMeritCfg: v1r0p12*
dataMonitoring/Common: v3r0p13*
dataMonitoring/FastMon: v3r0p13*
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r0p11*
svac/EngineeringModelRoot: v3r12
svac/TestReport: v5r8*
users/richard/pipelineDatasets: v0r4
ft2Util: v1r1p44
GPLtools: v1r10
Reason for change
These are small bug fixes and improvements from Ops Sim 2 feedback. We have added monitoring of Merit quantities. And we have added the new AFS buffers.
Test Procedure
We have processed data with this version of L1Proc.
Rollback procedure
We can roll back to the previous version of L1Proc.
CCB Jira
Details
dataMonitoring/AlarmsCfg: v1r0p2
- Added doc/release.notes: GDQMQ-64
dataMonitoring/FastMonCfg: v1r0p2
- Added doc/release.notes: GDQMQ-64
dataMonitoring/DigiReconCalMeritCfg: v1r0p6
- Added file to merge histograms from merit: GDQMQ-3
- Added monitoring of multiple scattering: GDQMQ-38
- Fixed various Digi End-of-run typos: GDQMQ-21
- Updates to ACD Pha Mips histogram: GDQMQ-22
- Fixed name problems of Acd extrapolated track histograms: GDQMQ-35
- Add histogram of the TOT distribution: GDQMQ-36
- Additional TOT histograms: GDQMQ-39
- Typo fixes: GDQMQ-40
dataMonitoring/Common: v3r0p9
- New script for fitting cal pedestals and gain ratios at the end of the run if needed. Currently not used in production. GDQMQ-65
- Improved script for comparing the FastMon plots to the corresponding digi ones. It writes an output xml summary file. GDQMQ-46.
- Some modification to the ACD tiles dimensions/rate correcting factors (to be used in the alarms): GDQMQ-53.
- Piece of code for calculating the Cal gains added. Not yet used, though. GDQMQ-65.
- New module for normalizing ACD quantities to tile position/area added. No change in functionality at the moment: GDQMQ-48.
GPLtools: v1r10
- Raise an exception when staging in a file failsLONE-30.
svac/L1Pipeline: v1r43
- New AFS buffers: LONE-26.
- To send messages to the log watcher, compatible with ISOC 4.4.0: LONE-27.
- New ways to communicate alarm information to data processing application: LONE-28.
- Add merit monitoring LONE-29
- Compare digi and merit end-of-run histograms LONE-31.
Complete set of tags for L1Proc 1.43
Code Versions
GlastRelease (sim/recon) v13r11p4
ScienceTools (Level 2) : v9r4p2
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r43
dataMonitoring/AlarmsCfg: v1r0p2*
dataMonitoring/FastMonCfg: v1r0p2*
dataMonitoring/DigiReconCalMeritCfg: v1r0p6*
dataMonitoring/Common: v3r0p9*
dataMonitoring/FastMon: v3r0p8
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r0p6
svac/EngineeringModelRoot: v3r12
svac/TestReport: v5r6
users/richard/pipelineDatasets: v0r4
ft2Util: v1r1p44
GPLtools: v1r10
Reason for change
Change query to take into account database schema changes introduced by ISOC release 4.4.
These changes caused the production application to either hang or crash.
Additional small fixes have also been added.
Test Procedure
The new query has been tested on the development server.
Rollback procedure
Version 0.1.1 can be easily put back in place.