The page will be used to track issues arising during Infrastructure Shifts.
June 25
18:24 PDT
A new version (1.2.4) of the pipeline has been installed. See https://jira.slac.stanford.edu/browse/SSC-74
DataQualityMonitoring hanging
The DataQualityMonitoring application at some point was hanging.
The log files was spitting out several lines of:
WARNING: Waiting for response for 12 secs stat[wain019.slac.stanford.edu/172.23.2.19:1094:anonymous] Jun 25, 2008 10:25:00 AM org.glast.base.application.web.filter.ApplicationFilter doFilter
I had to restart the server as I could not get to probe.
After this all was well.
Max
TelemetryTrending application crashes
The telemetry trending application had to be restarted a couple of times in the last 24 hours. This has been caused by large memory usage when tabulating the data IOT-87@jira.
I am working on a fix.
In the meantime monitor the memory usage of tomcat12 from the Server Monitoring application.
When memory gets close to 90% try clicking on "Force GC" (If you don't see this link you need to be added to the ServerMonitoringAdmin group). If garbage collection does not reduce the memory usage a crash might be imminent.
Max
4am UTC
Tony: Waiting for data.... Registered my pager – sending to number@amsmsg.net seems to work (alias@MyAirMail.com does not)
Two outstanding (new) issues: PFE-172@jira IFO-24@jira