Open Problems

Data Quality Monitoring Unresponsive

Symptoms

The DataQualityMonitoring application is unresponsive: No page is returned and the browser keeps on waiting for a response.
Probe is also unresponsive.

Cause

This seems to be due to some threads losing a socket connection with xrootd when trying to fetch root files to display histograms.
Looking at the log files for the DataQualityMonitoring application there is a proliferation of messages like:

WARNING: Waiting for response for 12 secs stat[wain019.slac.stanford.edu/172.23.2.19:1094:anonymous]
Jun 25, 2008 10:25:00 AM org.glast.base.application.web.filter.ApplicationFilter doFilter

As probe is unresponsive to look at the log files you have to log-in glast-tomcat07, cd tomcat/BASE55/glastlnx09/logs and tail the most recent DataQualityMonitoring log file.

Solution

Restart glast-tomcat07 : from the ServerMonitoring application restart the glast-tomcat07 server by clicking on the recycle icon. If you don't see this icon it means you need to be added to the administrators list.

History and Exceptions

Follow this link for a History of the Exceptions

OpsLog, DataQualityMonitoring, TelemetryTrending losing sessions

Symptoms

There are many symptoms linked to this problem. All seem to be due to the fact that the session is expired.

The very strange thing is that these problems seem to be correlated across applications.

OpsLog

Some users have been timed-out of the OpsLog with the following message:

Your session has timed out.

If there is no activity from the browser in 3 hours the server releases the session. 
We limit the length of the session to control the load on the server. 
The server has no way of telling whether the browser is still connected other than to monitor its activity. 
If the sessions are too long the server keeps abandoned session open for an infinite time. 
They accumulate quickly and consume the memory.

Please return to the index and log in again.

TelemetryTrending

The following exception in an indication that the session has expired:

java.lang.NullPointerException
   org.freehep.webutil.tree.TreeUtils.nodesForPath(TreeUtils.java:17)
   org.glast.base.application.web.filter.ApplicationFilter.defaultApplicationFilter(ApplicationFilter.java:248)
   org.glast.base.application.web.filter.ApplicationFilter.doFilter(ApplicationFilter.java:173)

DataQualityMonitoring

Some users have noticed that when selecting one or more runs from the main table, their selection was not taken into account when pushing "Update". After a while the server would return a table with all the runs selected.

This is an indication of an expired session.

Cause

Unknown.

There is some speculation that something is happening at the cookies level.
The shifters have been instructed to add more details when reporting such problems.

Solution

Close all browsers and start from scratch.

  • No labels