The page will be used to track issues arising during Infrastructure Shifts.
June 26
glast-win01 and glast-win02 became unreachable at 9:20pm. The SCCS on call technical contact has been paged. We just received the following e-mail:
I just spoke with Mike, the servers are physically up but there seems to be a network problem. Mike is working on contacting the networking team to investigate on their end.
- Had to restart PROD data crawler one time (because Nagios and http://glastlnx20.slac.stanford.edu:5080
were complaining). Looks like problem was caused by MC writing to DEV version of xrootd on glastlnx22. - Problems with Run Quality monitoring reported yesterday now fixed.
June 25
From OpsLog
Watching plots on the web is right now very slow...
**Comment by David Paneque on Thursday, June 26, 2008 5:20:39 AM UTC
It is so from both, the shifter computers and our laptops.
**Comment by David Paneque on Thursday, June 26, 2008 5:21:33 AM UTC
now it is fast again...
**Comment by Tony Johnson on Thursday, June 26, 2008 5:55:18 AM UTC
Trending plots, data quality plot, all plots? One possibility is that xrootd load slows down plotting (some plots are read using xrootd). I noticed there were some ASP and MC jobs running in the pipeline around this time which may have been slowing things down.
**Comment by Tony Johnson on Thursday, June 26, 2008 6:07:59 AM UTC
Indeed in the DataQualityMonitoring log file around this time I see lots of messages about waiting for response from xrootd.
Outstanding Issues:
ELG-18@jira OpsLog session times out immediately after login.
[] Some plots in data monitoring are inaccessible (message says plot not available), even while the same plots are accessible from another browser at the same time. Maybe this is also related to the workstations in the control room running firefox 1.5 (seems unlikely)? See discussion in OpsProbList.
[] Jim reported that several of his scripts were running and were killed when the pipeline server was restarted. We need to understand why his data catalog queries are still taking so long >10 minutes to run.
GRQ-1 Run quality status is not being updated even though change entries are made in the history table. I looked at the code but could not see anything obvious wrong, maybe I am too sleepy.
LONE-72 Attach intent as meta-data to files
LONE-71 Digi merging loses IObfStatus – results in Digi files being marked as ContentError in Data Catalog
18:24 PDT A new version (1.2.4) of the pipeline has been installed. See https://jira.slac.stanford.edu/browse/SSC-74
DataQualityMonitoring hanging
The DataQualityMonitoring application at some point was hanging.
The log files was spitting out several lines of:
WARNING: Waiting for response for 12 secs stat[wain019.slac.stanford.edu/172.23.2.19:1094:anonymous] Jun 25, 2008 10:25:00 AM org.glast.base.application.web.filter.ApplicationFilter doFilter
I had to restart the server as I could not get to probe.
After this all was well.
Max
TelemetryTrending application crashes
The telemetry trending application had to be restarted a couple of times in the last 24 hours. This has been caused by large memory usage when tabulating the data IOT-87@jira.
I am working on a fix.
In the meantime monitor the memory usage of tomcat12 from the Server Monitoring application.
When memory gets close to 90% try clicking on "Force GC" (If you don't see this link you need to be added to the ServerMonitoringAdmin group). If garbage collection does not reduce the memory usage a crash might be imminent.
Max
4am UTC
Tony: Waiting for data.... Registered my pager – sending to number@amsmsg.net seems to work (alias@MyAirMail.com does not)
Two outstanding (new) issues: PFE-172@jira IFO-24@jira