Reason for change
We have a new release of GlastRelease that solves the problem with NaN in the Astro package.
Test Procedure
We have processed real on-orbit data on the DEV server with this version of L1Proc.
Rollback procedure
L1Proc can be rolled back to the previous version.
CCB Jira
Details
L1Pipeline v1r58
-Fix bug that caused LCI runs to not get finalized.
-Use the same tree merger as recon, digi, & gcr for cal tuples.
GlastRelease v15r21:
- Includes (filter) configuration related changes.
- Systemtestsfor GR v15r21with respect to the previous production release, GR v15r19.
- RM diffwith respect to GR v15r19.
Complete set of tags for L1Proc 1.58
Code Versions
GlastRelease (sim/recon) v15r21*
ScienceTools (Level 2) : v9r5p5
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r58*
calibTkrUtil v2r2p3
dataMonitoring/AlarmsCfg: v1r1p1
dataMonitoring/FastMonCfg: v1r2p0
dataMonitoring/DigiReconCalMeritCfg: v1r1p35* (Has been in since L1Proc 1.56 - was never reported. My mistake.)
dataMonitoring/Common: v3r1p6
dataMonitoring/FastMon: v3r2p3
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r2p1
svac/EngineeringModelRoot: v4r1p5
svac/TestReport: v6r12
users/richard/pipelineDatasets: v0r4
ft2Util: v1r19
evtClassDefs v0r5
GPLtools: v1r10
Reason for Change
Several relatively minor issues have arisen in GRB processing so far in LEO. Jiras are posted below. ASP-8 resulted from unexpectedly small (zero to one event) FT1 files delivered by L1Proc. ASP-3 and ASP-10 were uncovered by trying to process the test GCN Notices from GBM.
Test Procedure
Tested by hand on flight data associated with the identified failures. Tested in dev on data in /ASP/TestSims and /ASP/TestSims2.
Rollback Procedure
Revert to ASP v2r5p1 + ASP/pyASP v3r5p1 + ASP/asp_pgwave v1r5
CCB JIRA
Details
- ASP/grbASP v4r3
- ASP-3@JIRA GRB_refinement_launcher fails when no FT1 data are available. Modified launchRefinementStreams.py to exit in this case.
- ASP-8@JIRA GRB_blind_search crashes if there is only one event in the FT1 file for a given run. As a short term solution, will skip files with a single event in BlindSearch.py.
- ASP-10@JIRA GRB refinement process crashes when there are no photons returned from the data selection for the candidate GRB. For the short term there is no convenient way to avoid a failed process. The best short term fix is to have the extractLatData process report that no events for this burst were found, and if appropriate, that the burst was too far off-axis to be seen. The long term fix will be to find a way to exit the train of processes in the xml task definition without setting a failure condition. Also, the info regarding the off-axis angle of the burst or lack of photons will need to be propagated to the data processing page somehow. AFAIK, there are no mechanisms for implementing either of these last two items.
Complete set of tags for ASP v2r6 (* indicates a revision for this CCB)
- ASP/AspHealPix v0r0p1
- ASP/AspLauncher v1r3p1
- ASP/AspPolicy v0r6
- ASP/BayesianBlocks v0r2
- ASP/asp_pgwave v1r5
- ASP/drpMonitoring v1r3p2
- ASP/grbASP v4r3 *
- ASP/pyASP v3r5p1
- ASP/sourceDet v0r0p1
Reason for change
We have a new release of GlastRelease that solves problems with the GCR. According to Fred Piron these changes are necessary to recover protons and He in the calibOps runs we will start taking at 10 pm PDT tonight (June 27). Richard has a detailed email from Fred with additional information. There are also updates to the Gleam filters (running the correct gamma-filter, correct calibration constants, fixing the HIP-filter so it doesn't pass all events).
This GR also has a fix from Toby to remove the NaN we see in 15% of the events. There is also an update to the FT2 making to deal with special cases in the Magic 7 file and avoid crashing. In addition, we have a few small bug fixes for the Data Monitoring and a bug fix for the SVAC ntuple.
Test Procedure
We have processed real on-orbit data on the DEV server with this version of L1Proc.
Rollback procedure
L1Proc can be rolled back to the previous version.
CCB Jira
Details
L1Pipeline v1r57
-Attach MOOT key and alias as metadata when registering files. LONE-72
-Avoid the merging issue described in LONE-67.
GlastRelease v15r19:
- Includes (filter) configuration related changes.
- Systemtestsfor GR v15r19wrt GR v15r13 show no differences.
- RM diffwith respect to the previous production release, GR v15r13.
svac/EngineeringModelRoot: v4r1p5
- Type error for the event size. Caused data corruption in the SVAC ntuple.
svac/TestReport: v6r12
- Correct types for FSW scalers.
dataMonitoring/FastMon: v3r2p3
- Bug fix in Julian Date calculation, that was the cause of the wrong latitude and longitude. GDQMQ-133.
- Added X-axis Ra and Dec pointing copying on Zaxis methods that are however probably wrong. GDQMQ-160 and GDQMQ-167. Relevant jira(s): GDQMQ-133, GDQMQ-160 and GDQMQ-167
- Bug fix FastMon Bug in pM7Parser when there is an invalid index in the pSCPosition list. FastMin still not able to handle an empty/corrupted M7 file. Relevant jira(s): GDQMQ-156
ft2Util: v1r19
- No more ATT extrapolation.
Complete set of tags for L1Proc 1.57
Code Versions
GlastRelease (sim/recon) v15r19*
ScienceTools (Level 2) : v9r5p5
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r57*
calibTkrUtil v2r2p3
dataMonitoring/AlarmsCfg: v1r1p1
dataMonitoring/FastMonCfg: v1r2p0
dataMonitoring/DigiReconCalMeritCfg: v1r1p32
dataMonitoring/Common: v3r1p6
dataMonitoring/FastMon: v3r2p3*
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r2p1
svac/EngineeringModelRoot: v4r1p5*
svac/TestReport: v6r12*
users/richard/pipelineDatasets: v0r4
ft2Util: v1r19*
evtClassDefs v0r5
GPLtools: v1r10
Reasons for Change
One bug-fix:
- PGWave crashes if it finds no sources. See asp-6@jira
Several improvements from Gino:
- GroupManager is queried to get FlareAdvocate email addresses for sending flare notifications
- Counts map resolution is adjusted for time interval size to account for higher statistics in longer time intervals
- gtexposure is used in computing light curves for variability analysis
Test Procedure
Run by hand on LEO data for the offending time interval (236066400, 236088000). Tested in dev using /ASP/TestSims2 data.
Rollback Procedure
Revert to ASP/asp_pgwave v1r4p1
CCB Jira
Details
- ASP/asp_pgwave v1r5
- See the package Changelog revision 1.1.1.50
Reason for change
Fix incorrect link from L1 processing status bar when a run is present in multiple downlinks.
GDP-22@jira Don't see very many blockers.....
Test Procedure
This version has been tested on the DEV server.
Rollback procedure
Version 0.2.0 could be reinstalled
CCB Jira
Details
Reason for change
Two reasons:
- improved memory usage when tabulating selected data. This will prevent crashes but not avoid them. Large volumes of data will inevitably take the server down.
- added an option (in the form of a link) to view the whole tree, containing ALL the current mnemonics. This has been requested by a few shifters.
Test Procedure
This version has been tested on the DEV server.
Rollback procedure
Version 2.0.1 could be reinstalled
CCB Jira
Details
Reason for change
The only change is a fix to the calendar code to fix bug: GBW-4@JIRA.
Test Procedure
This version has been tested on the DEV server.
Rollback procedure
Version 0.1.4 could be reinstalled
CCB Jira
Reason for change
We have a new release of GlastRelease that includes all the new (filter) configuration information i.e. getting the Moot key, extracting the correct filter information (DGN prescales for example) and automatically configuring and running the correct filters in Gleam. This GR also has the real FSW filter bits in the Merit ntuple.
This GR has a known problem: The Gleam HIP fillter passes (actually, leaks) all events. It's possible that the Gleam MIP filter suffers from a smaller degree of the same problem.
There are updates to the SVAC ntuple and the monitoring to adapt to this new GR.
Test Procedure
We have processed real data and MC on the DEV server with this version of L1Proc.
Note that there are no changes in the HalfPipe because of this GR. The relevant tables (Run Id - Moot key association) are being filled independently of L1 (and has been for some while).
Rollback procedure
L1Proc can be rolled back to the previous version. We also have the capability to use Job Options to disable Moot access.
CCB Jira
Details
L1Pipeline v1r56
-
GlastRelease v15r13:
- Includes (filter) configuration related changes.
- Systemtestsfor GR v15r13wrt GR v15r6p1 show no differences.
- RM diffwith respect to the previous production release, GR v15r6p1.
svac/Monitor: v1r2p1
- Change HFC for Hip and DFC for Dgn in MonInput_OBFStatus.cxx because of corresponding name changes in GR v15r8.
- Change getStatus32() by getStatusWord() to match with the new GLAST release (v15r8).
svac/EngineeringModelRoot: v4r1p4
- Changes induced by GR v15r8: Added the Moot key. Added filter prescale factors and indices for all the filters. Added some Gleam gamma-filter quantities and updated to new names of Gleam filters.
evtClassDefs: v0r5
- Updated for name changes in Gleam filters.
svac/TestReport: v6r9
- Changed the range for Datagram Histos so the complete distribution is included.
dataMonitoring/DigiReconCalMeritCfg: v1r1p32
- Remove quantity AcdVtxActiveDist. This addresses Jira GDQMQ-161.
- Change truncation from 1.1 mip to 2.5 mip in plots Recon_ReconAcdPhaMipsCorrectedAngle_PMTX_Zoom_TH1_AcdTile. This addresses Jira GDQMQ-162.
- New ACD rate asymmetry trending quantities. This addresses Jira GDQMQ-163.
dataMonitoring/Common: v3r1p6
- Now a pAlarm object is instantiated for each pAlarmSet object, even when the corresponding ROOT object cannot be found in the file. The corresponding output status is set to UNDEFINED and it appears in the summary. Usefull for debug to make sure we don't screw up thing when messing around with the variable names. Relevant jira(s): GDQMQ-159
ft2Util: v1r2p18
- Various bug fixes in interpolation.
Complete set of tags for L1Proc 1.56
Code Versions
GlastRelease (sim/recon) v15r13*
ScienceTools (Level 2) : v9r5p5
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r56*
calibTkrUtil v2r2p3
dataMonitoring/AlarmsCfg: v1r1p1
dataMonitoring/FastMonCfg: v1r2p0
dataMonitoring/DigiReconCalMeritCfg: v1r1p32*
dataMonitoring/Common: v3r1p6*
dataMonitoring/FastMon: v3r2p1
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r2p1*
svac/EngineeringModelRoot: v4r1p4*
svac/TestReport: v6r9*
users/richard/pipelineDatasets: v0r4
ft2Util: v1r2p18
evtClassDefs v0r5*
GPLtools: v1r10
Reason for change
Many users have complained about how slow the navigation tree has become.
This version adds three new features to improve the performance and help usability of the tree:
- Mode Filtering: there are a Shifter and Expert mode. Shifter mode has a reduced tree while Expert will have the full tree
- Alphanumerical sorting: to sort either ways
- Mnemonic Filtering: to quickly find a mnemonic in the tree.
Test Procedure
This version has been tested on the DEV server.
Rollback procedure
Version 2.0.0 could be reinstalled
CCB Jira
Details
Reason for Change
For top-level tasks that launch other top-level tasks, e.g., any ASP task from L1Proc, the Pipeline server mode [PROD,DEV,TEST] needs to be inherited. The revised code checks the PIPELINE_FROMADDRESS env var to determine the server mode and propagates that information to the pipeline createStream command. This is needed since L1Proc instances are running in dev, but they are running the prod installation of ASP (which has the pipeline server mode set to PROD by default).
Test Procedure
Tested in dev with code instrumented to verify that the PIPELINE_FROMADDRESS variable is being examined and override performed.
Rollback Procedure
Revert to ASP/pyASP v3r5
CCB Jira
Details
- ASP/pyASP v3r5p1
- Modifications to PipelineCommand.py
Reason for change
Bug fixes for the monitoring is the driver for this request. They affect critical quantities like GPS out of lock, LAT mode (changes) and longitude. This is the last planned change to the monitoring.
We also have a new GlastRelease which contains a bug fix to the event time stamp in case of lost GPS Lock (we should let the UDL do its job!) in addition to updated versions of the CAL calibration code. Also, as of this version of L1Proc we now use a new CAL calibration quantity: "NeighborXtalkTool". In addition, CAL has provided a new set of calibration files to be used for the first on-orbit data and the calibration meta-database has been updated.
We also have a new version of Science Tools that uses the same release of Root as GlastRelease i.e. the two are now in synch.
Test Procedure
We have processed real data and MC on the DEV server with this version of L1Proc.
Rollback procedure
L1Proc can be rolled back to the previous version.
CCB Jira
Details
L1Pipeline v1r55p1
- Don't clean up temporary files if merging didn't find all input files.
- Fix bugs in handling of incomplete runs.
GlastRelease v15r6p1:
- New versions of CAL calibration code.
- Bug fix for the event time stamp: Should really let the UDL do its job and not use 20MHz if we lose GPS lock! JIRA GRINF-48
- Systemtestsfor GR v15r6p1wrt GR v15r5 show no differences.
- RM diffwith respect to the previous production release, GR v15r5.
Science Tools: v9r5p5
- Uses Root v5.18.00c-gl1 (same as GR v15r6p1).
dataMonitoring/FastMonCfg: v1r2p0
- Update MergedHisto File with CondArrCNO_CalLoOpensWindow_TH1 - GDQMQ-139
- Updated xml/configLCI.xml with newest quantities for LCI runs - GDQMQ-141
- Updated xml/config.xml with a cut to remove sollicited triggers from FastMon CondSummaryWord_TH1 histogram - GDQMQ-142
- Updated xml/config.xml with a new range for CondSummaryWord_WithSolicited_TH1 histogram - GDQMQ-143.
- Relevant jira(s): GDQMQ-139, 141, 142, 143
- Renamed CondArrCalCNO_CalLoOpensWindow_TH1 to CondArrCNO_CalLoOpensWindow_TH1 in xml/config.xml. Relevant jira(s): GDQMQ-139
dataMonitoring/DigiReconCalMeritCfg: v1r1p29
- Add some unprescaled qunatities: Jira GDQMQ-155.
- Added quantity AcdTkr1ActDistTileEnergy to monconfig_merit_trending.xml. This addresses Jira GDQMQ-154.
- Add the cut Fsw DGN filter to the calculation of the UnPrescaled Trigger Engines. This addresses Jira GDQMQ-132.
- Change also (Fast mon) quantity FastMon_MetaContext_OpenMode from mean to valuechange[0]. This addresses Jira GDQMQ-134.
- Change also (Fast mon) quantities FastMon_Evt_DataTransferId and FastMon_MetaContext_LATC_Master from outputnumber to valuechange[0]
- Change quantity LastDataTransferId from type outputnumber to valuechange[0]. This addresses jira GDQMQ-146.
- Added Normalized rates for FastMon GPS_OUTofLock. This addresses Jira GDQMQ-138.
- Added normalized rates for GPS_OUTofLock. This addresses Jira GDQMQ-138.
- Added rates for CTBParticleType==0 and == -1. Change range of histogram for CTBParticleType to accomodate the values -1 and 0, which were not expected according to documentation provided.
dataMonitoring/FastMon: v3r2p1
- Bug fix when calculating the phi angle of the 3D Position vector, now using safer ROOT.TVector3 methods.
- Now handling LATC_master key in LPA runs with meta.hardwareKey and in LCI runs with meta.writeCfg. Relevant jira(s): GDQMQ-145
svac/Monitor: v1r1p20
- Changes to accomodate quantity type ValChange_ into database. That addresses jira GDQMQ-146.
Complete set of tags for L1Proc 1.55
Code Versions
GlastRelease (sim/recon) v15r6p1*
ScienceTools (Level 2) : v9r5p5
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r55*
calibTkrUtil v2r2p3
dataMonitoring/AlarmsCfg: v1r1p1
dataMonitoring/FastMonCfg: v1r2p0*
dataMonitoring/DigiReconCalMeritCfg: v1r1p29*
dataMonitoring/Common: v3r1p5
dataMonitoring/FastMon: v3r2p1*
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r1p20*
svac/EngineeringModelRoot: v4r0p0*
svac/TestReport: v6r8
users/richard/pipelineDatasets: v0r4
ft2Util: v1r2p15
evtClassDefs v0r4
GPLtools: v1r10
Reason for change
Two main fixes:
- Ingestion code buggy for run based quantities
- Web front end picked up old alarm files
Both have been fixed.
Test Procedure
This version has been tested on the DEV server.
Rollback procedure
Version 1.0.0 can be easily put back in place.
CCB Jira
Details
Reason for change
New GCN notices don't appear in the Processing page as the Lat_Alert_Time is filled by the ASP tasks.
The code has been change to use the GRB_ID when the Alert time is not present.
Test Procedure
This version has been tested on the DEV server.
Rollback procedure
Version 0.1.3 could be reinstalled
CCB Jira
Reason for Change
Implement use of Oracle wallets for establishing credentials for db connections. This will enable the use of the GroupManager for ascertaining email distribution lists for GRB and Flare advocates. Bug-fix for grbASP/python/launch[Refinement,Afterglow]Stream.py.
Test Procedure
Tested in dev pipeline using ASP tables in dev space and test data in /ASP/TestData2.
Rollback Procedure
Revert to ASP v2r5. Reinstall encoded text file with credentials in afs location.
CCB JIRA
Details
- ASP/AspPolicy v0r6
- modify wrapperGenerator.py to set up environment for Oracle wallets
- ASP/pyASP v3r5
- use connection strings appropriate for Oracle wallet use
- ASP/grbASP v4r2p1
- query GroupManager tables for GRB advocate email distribution list
- import FileStager module in launchRefinementStream.py and launchAfterglowStream.py
- improve handling for filtering GCN Notice types in email ingest
- Remove encoded text file with credentials from afs location.
Reason for change
We now have a new GlastRelease that has the real onboard filter status bits available in the digi file. In combination with this we have added new monitoring quantities based on these and also updated the SVAC ntuple. Note that this in particular include the DGN bits i.e. for the first time we have monitoring of an unbiased sample.
This GR also contains the TKR alignment used through the calibration meta-database like all the other calibration files. The relevant calibration files (inter- and intra-tower) have been added to the calibration meta-database and are now being used by default in data processing in the L1 pipeline.
We have also clarified where to pick up the particle type cuts (transient etc) and made sure the cuts used in the monitoring are the same as the one used for the instrument response functions.
Note that this GR needs a new version of DFI which has to be deployed in synch. Also note that Flight Ops are keen on deploying asap as the new DFI request on their side comes in combination with needed Mission Planning updates.
Test Procedure
We have processed real data on the DEV server with this version of L1Proc. Testing has been complicated by the fact that we have needed the NIGHTLY ISOC software environment to pick up the new DFI and that we could not use our usual evt files since the format has changed.
An additional complication has been that we only could use ground data runs to test the new monitoring quantities. Ops Sim 2 or any other MC that has gone through the usual Ebf2Ccsds only contain the real Passtru bit and are useless for testing the (other) real filter status bits. I'm working with Gregg to find a way to solve this for the future. Note that the absence of a host side LPA means that even in the best of cases only one filter can run at any given time on an EBF file (I'm told there is a denied FSW CCB request that would have solved this. Oh well.). So MC files processed in the L1 pipeline will never contain a full set of 'real' filter status bits. This makes testing difficult as ground data doesn't look like on-orbit data.
Rollback procedure
L1Proc can be rolled back to the previous version (in synch with DFI).
CCB Jira
Details
L1Pipeline v1r54
- Updated to the use the new GR and newest set of monitoring tags.
- Be more paranoid about cleaning up crumb-level files LONE-61.
- PipelineStaging6 is having reliability issues, we're not using it for now.
GlastRelease v15r5:
- This GR depends on the new DFI and contains the real filter status bits.
- Systemtestsfor GR v15r5wrt GR v15r2 show no differences as the real filter bits are not part of the system tests and (for the TKR alignment) there are still no data runs in the system tests.
- RM diffwith respect to the previous production release, GR v15r2.
- A few words about testing the real filter status bits: I have verified on a handful of real data events that there is agreement between the information in the evt file and the digi file for the GAMMA, MIP and DGN filters concerning which filters passed the events. Unfortunately I have not found any events in ground data that pass the HIP-filter. As I can not use MC (for the reasons detailed above) I have not been able to explicitly verify that the HIP-filter bits are properly set. Neither has anybody else.
dataMonitoring/DigiReconCalMeritCfg: v1r1p21
- Corrected few typos in merit trending parameters. This addresses Jira GDQMQ-137.
- Added merit parameters. This addresses Jira GDQMQ-137.
- Changed normalization in quantities that relate Flight Software and Gleam filter bits (file monconfig_digi_trending.xml). This addresses Jira GDQMQ-132.
- Added a bunch of new quantities related to the real flight software filter bits. This addresses Jira GDQMQ-132.
- Added Log10 of measured energy (recon job; monconfig_recon_histos.xml). This will show all features in the energy spectra. This addresses Jira GDQMQ-128.
dataMonitoring/FastMon: v3r1p5
- Bug Fix, change modf to fmod in the routine calculating the longitude in pSCPosition.py, as done in ST astro package. Relevant jira(s): GDQMQ-133
svac/Monitor: v1r1p19
- "Add gamma->has()" MonInput_FswFilters_GammaBits.cxx
- Change definition of prescales in class MonInput_DGNGemPrescale.cxx.
- Changes in code to compile with the new glast release (GlastRelease-HEAD1.1097). Those issues address the jira GDQMQ-132.
- Changes in code to compile with the new glast release (GlastRelease-HEAD1.1096).
- Add merit parameters. Those issues address the jiras GDQMQ-137 and GDQMQ-132.
- Define Branch name for filter bits info retrieval. Changed Passed() by passed(). This address Jira GDQMQ-132.
- Added new quantities to retrieve the real flight software filter bits. This addresses Jira GDQMQ-132.
svac/EngineeringModelRoot: v4r0p0
- Added real FSW filter status information. Needs the new DFI!
calibTkrUtil: v2r2p3
- Bug fix to avoid division by zero.
Complete set of tags for L1Proc 1.54
Code Versions
GlastRelease (sim/recon) v15r5
ScienceTools (Level 2) : v9r5p4
Science Ops (task defs, scripts):
Level 1 pipeline code and applications running in L1:
svac/L1Pipeline: v1r54*
dataMonitoring/AlarmsCfg: v1r1p1
dataMonitoring/FastMonCfg: v1r1p11
dataMonitoring/DigiReconCalMeritCfg: v1r1p21*
dataMonitoring/Common: v3r1p5
dataMonitoring/FastMon: v3r1p5*
datMonitoring/IGRF: v1r0p1
svac/Monitor: v1r1p19*
svac/EngineeringModelRoot: v4r0p0*
svac/TestReport: v6r8
calibTkrUtil: v2r2p3*
users/richard/pipelineDatasets: v0r4
ft2Util: v1r2p15
evtClassDefs v0r4
GPLtools: v1r10