You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

Open Problems

Data Quality Monitoring Unresponsive

Symptoms

The DataQualityMonitoring application is unresponsive: No page is returned and the browser keeps on waiting for a response.
Probe is also unresponsive.

Cause

This seems to be due to some threads losing a socket connection with xrootd when trying to fetch root files to display histograms.
Looking at the log files for the DataQualityMonitoring application there is a proliferation of messages like:

WARNING: Waiting for response for 12 secs stat[wain019.slac.stanford.edu/172.23.2.19:1094:anonymous]
Jun 25, 2008 10:25:00 AM org.glast.base.application.web.filter.ApplicationFilter doFilter

As probe is unresponsive to look at the log files you have to log-in glast-tomcat07, cd tomcat/BASE55/glastlnx09/logs and tail the most recent DataQualityMonitoring log file.

Solution

Restart glast-tomcat07 : from the ServerMonitoring application restart the glast-tomcat07 server by clicking on the recycle icon. If you don't see this icon it means you need to be added to the administrators list.

History and Exceptions

Follow this link for a History of the Exceptions

OpsLog, DataQualityMonitoring, TelemetryTrending losing sessions

Symptoms

There are many symptoms linked to this problem. All seem to be due to the fact that the session is expired.

The very strange thing is that these problems seem to be correlated across applications.

OpsLog

Some users have been timed-out of the OpsLog with the following message:

Your session has timed out.

If there is no activity from the browser in 3 hours the server releases the session. 
We limit the length of the session to control the load on the server. 
The server has no way of telling whether the browser is still connected other than to monitor its activity. 
If the sessions are too long the server keeps abandoned session open for an infinite time. 
They accumulate quickly and consume the memory.

Please return to the index and log in again.

TelemetryTrending

The following exception in an indication that the session has expired:

java.lang.NullPointerException
   org.freehep.webutil.tree.TreeUtils.nodesForPath(TreeUtils.java:17)
   org.glast.base.application.web.filter.ApplicationFilter.defaultApplicationFilter(ApplicationFilter.java:248)
   org.glast.base.application.web.filter.ApplicationFilter.doFilter(ApplicationFilter.java:173)
  • No labels