The page will be used to track issues arising during Infrastructure Shifts.
June 25
DataQualityMonitoring hanging
The DataQualityMonitoring application at some point was hanging.
The log files was spitting out several lines of:
WARNING: Waiting for response for 12 secs stat[wain019.slac.stanford.edu/172.23.2.19:1094:anonymous] Jun 25, 2008 10:25:00 AM org.glast.base.application.web.filter.ApplicationFilter doFilter
I had to restart the server as I could not get to probe.
After this all was well.
TelemetryTrending application crashes
The telemetry trending application had to be restarted a couple of times in the last 24 hours. This has been caused by large memory usage when tabulating the data IOT-87@jira.
I am working on a fix.
In the meantime monitor the memory usage of tomcat12 from the Server Monitoring application.
When memory gets close to 90% try clicking on "Force GC" (If you don't see this link you need to be added to the ServerMonitoringAdmin group). If garbage collection does not reduce the memory usage a crash might be imminent.
4am UTC
Tony: Waiting for data.... Registered my pager – sending to number@amsmsg.net seems to work (alias@MyAirMail.com does not)
Two outstanding (new) issues: PFE-172@jira IFO-24@jira