Infrastructure Shift Log

The page will be used to track issues arising during Infrastructure Shifts.

• How to do infrastructure shifts
• Weekly Shift Schedule
• Contact List
• [How To Fix]

Please check the list of known problems

July 15

20:45 (CH) There was a scheduled outage of the LSF master server causing some runs to fail.
After the server came back up some jobs were tagged as suspended (SSUSP) and it wasn't clear whether to wait or kill them and resubmit.

July 14

09:00 (KH) Same problem as at 00:30. Tomcat09 restarted.

00:30 (TJ) I restart tomcat09 since all of the pipeline-II JDBC connections were in use. The problem persisted after the restart, although the applications were still responding, then after about 15 minutes the problem seemed to go away by itself.

July 13

01:00 am Restarted tomcat07 due to Data Quality Monitoring Unresponsive.

19:30 (TG) Another instance of DQM web page failing to come up on DS terminal. Problem resolved after two restarts of Firefox. GDM-122@jira.

July 12

glastlnx12 (OpsLog and Pipeline-II PROD) crashed for unknown reasons, and was restarted by Chuck Boehiem.

DQM trending app was failing for Anders with GDM-122@jira.
Clearing the session (by restarting the browser, or in Ander's case clearing the glast-ground cookies) fixed the problem

July 11

With reference to the TelemetryTrending problem, nagios had been complaining about tomcat12 for sometime. It monitors the tomcat servers using probe's "quickcheck" feature. This was showing that all of the JDBC connections were used up. This could just be a side effect, since if the application hangs while it has a DB connection open that will soon use up all of the connections.

8:45 am I restarted the dev datacat crawler (using the button on the datacat web admin page)
8:20 am same as below. TelemetryTrending application unresponsive.
5:30 am TelemetryTrending application unresponsive. Restarting the server fixed the problem. No evidence was found in the log files.

July 10

5:30 am TelemetryTrending problem fetching data from XML-RPC

http://glastlnx24:5441: org.apache.xmlrpc.XmlRpcException: Failed to create input stream:
  Connection reset
or
  Connection refused

This is a problem with the XML-RPC python server. This problem should be brought to the attention of the FO shifter.

19:20 (Richard, for Tony)

 Batch jobs were taking a long time, apparently being slow, but were in fact failed with no log files produced. Was tracked down to DNS failures on the balis. It has been reset (reported by Neal Adama at 18:15). Unix-admin ticket [SLAC #120230].

July 5

7:10 pm I restarted tomcat12 since the monitoring programs were complaining and ServerMonitoring showed it missing - Tony

July4">July 4

6:00 pm Old DQM ingestion script put back into production. The new script worked fine for some 24 hours and then we started having "idle" sessions locking out all the following ones. There were some 60 of them waiting. Killing the first one did not solve the problem as the next one went in "idle" state. We decided to kill all the waiting sessions and put the old script back in production. The failed ingest scripts are being rolled back.

The root problem was a software crash on rtr-slb1.
I just came in after Mike called me on the phone and power cycled the machine. It has come up fine and all services should now be restored.

Update on this...
As Sean says, the servers are up. swh-servcore1 shows both ports connected vlan 1854, full 1000. They are on 7/5 and 7/6.

I called Antonio and he saw that a small router had a hiccup/crashed??
He is going to SLAC now to reboot the router.

• Had to [restart PROD data crawler] one time (because Nagios and http://glastlnx20.slac.stanford.edu:5080 were complaining). Looks like problem was caused by MC writing to DEV version of xrootd on glastlnx22.
• Problems with Run Quality monitoring reported yesterday now fixed.

June 25

From OpsLog

Outstanding Issues:

ELG-18@jira OpsLog session times out immediately after login.
[] Some plots in data monitoring are inaccessible (message says plot not available), even while the same plots are accessible from another browser at the same time. Maybe this is also related to the workstations in the control room running firefox 1.5 (seems unlikely)? See discussion in OpsProbList.
[] Jim reported that several of his scripts were running and were killed when the pipeline server was restarted. We need to understand why his data catalog queries are still taking so long >10 minutes to run.
[GRQ-1] Run quality status is not being updated even though change entries are made in the history table. I looked at the code but could not see anything obvious wrong, maybe I am too sleepy.
[LONE-72] Attach intent as meta-data to files
[LONE-71] Digi merging loses IObfStatus - results in Digi files being marked as ContentError in Data Catalog

18:24 PDT A new version (1.2.4) of the pipeline has been installed. See https://jira.slac.stanford.edu/browse/SSC-74

TelemetryTrending application crashes

The telemetry trending application had to be restarted a couple of times in the last 24 hours. This has been caused by large memory usage when tabulating the data IOT-87@jira.

I am working on a fix.

In the meantime monitor the memory usage of tomcat12 from the Server Monitoring application.
When memory gets close to 90% try clicking on "Force GC" (If you don't see this link you need to be added to the ServerMonitoringAdmin group). If garbage collection does not reduce the memory usage a crash might be imminent.

Max

4am UTC

Tony: Waiting for data.... Registered my pager - sending to number@amsmsg.net seems to work (alias@MyAirMail.com does not)

Two outstanding (new) issues: PFE-172@jira IFO-24@jira