You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Open Problems

Data Quality Monitoring Unresponsive

Symptoms

The DataQualityMonitoring application is unresponsive: No page is returned and the browser keeps on waiting for a response.
Probe is also unresponsive.

Cause

This seems to be due to some threads losing a socket connection with xrootd when trying to fetch root files to display histograms.
Looking at the log files for the DataQualityMonitoring application there is a proliferation of messages like:

WARNING: Waiting for response for 12 secs stat[wain019.slac.stanford.edu/172.23.2.19:1094:anonymous]
Jun 25, 2008 10:25:00 AM org.glast.base.application.web.filter.ApplicationFilter doFilter

As probe is unresponsive to look at the log files you have to log-in glast-tomcat07, cd tomcat/BASE55/glastlnx09/logs and tail the most recent DataQualityMonitoring log file.

Solution

Restart glast-tomcat07 : from the ServerMonitoring application restart the glast-tomcat07 server by clicking on the recycle icon. If you don't see this icon it means you need to be added to the administrators list.

History and Exceptions

  • Apr 21, 2008 12:19:50 PM
  • May 14, 2008 11:38:13 AM
  • Jun 25, 2008 10:15:40 PM
  • Jun 25, 2008 10:19:35 PM
hep.io.root.daemon.xrootd.Multiplexor handleSocketException
WARNING: [wain017.slac.stanford.edu/172.23.2.17:1094:anonymous] Unexpected IO exception on socket
java.io.IOException: [wain017.slac.stanford.edu/172.23.2.17:1094:anonymous] No handler found for handle 0 (status=0)
        at hep.io.root.daemon.xrootd.Multiplexor.run(Multiplexor.java:270)
        at java.lang.Thread.run(Thread.java:619)
  • Jun 25, 2008 9:39:55 AM
  • Jun 26, 2008 9:20:18 AM 
     hep.io.root.daemon.xrootd.Multiplexor handleSocketException
WARNING: [wain019.slac.stanford.edu/172.23.2.19:1094:anonymous] Unexpected IO exception on socket
java.io.IOException: [wain019.slac.stanford.edu/172.23.2.19:1094:anonymous] No handler found for handle 1 (status=0)
        at hep.io.root.daemon.xrootd.Multiplexor.run(Multiplexor.java:270)
        at java.lang.Thread.run(Thread.java:619)
  • Jun 29, 2008 2:39:41 PM 
    hep.io.root.daemon.xrootd.Multiplexor handleSocketException
WARNING: [wain018.slac.stanford.edu/172.23.2.18:1094:anonymous] Unexpected IO exception on socket
java.io.IOException: [wain018.slac.stanford.edu/172.23.2.18:1094:anonymous] No handler found for handle 2 (status=0)
        at hep.io.root.daemon.xrootd.Multiplexor.run(Multiplexor.java:270)
        at java.lang.Thread.run(Thread.java:619)
  • Jun 27, 2008 11:03:46 AM 
    hep.io.root.daemon.xrootd.Multiplexor handleSocketException
WARNING: [wain019.slac.stanford.edu/172.23.2.19:1094:anonymous] Unexpected IO exception on socket
java.io.IOException: [wain019.slac.stanford.edu/172.23.2.19:1094:anonymous] No handler found for handle 7 (status=0)
        at hep.io.root.daemon.xrootd.Multiplexor.run(Multiplexor.java:270)
        at java.lang.Thread.run(Thread.java:619)
  • Apr 23, 2008 3:05:02 PM
  • May 21, 2008 many times between 11 AM and 8 PM
  • May 22, 2008 a few times between 5:40 AM and 10:30 AM
hep.io.root.daemon.xrootd.Multiplexor handleSocketException
WARNING: [wain017.slac.stanford.edu/172.23.2.17:1094:anonymous] Unexpected IO exception on socket
java.io.EOFException
        at java.io.DataInputStream.readShort(DataInputStream.java:298)
        at hep.io.root.daemon.xrootd.Multiplexor$Response.read(Multiplexor.java:391)
        at hep.io.root.daemon.xrootd.Multiplexor$Response.access$200(Multiplexor.java:379)
        at hep.io.root.daemon.xrootd.Multiplexor.run(Multiplexor.java:211)
        at java.lang.Thread.run(Thread.java:619)

sometimes followed by 

org.apache.catalina.core.StandardWrapperValve invoke
INFO: *** Clearing plot reigstry for session 987A5B6A934C63970BA2E35F92EB817B
java.io.IOException: Xrootd error 3006: Invalid request; user not logged in
        at hep.io.root.daemon.xrootd.Session.waitForResponse(Session.java:371)
        at hep.io.root.daemon.xrootd.Session.stat(Session.java:194)
        at hep.io.root.daemon.xrootd.XrootdURLConnection.connect(XrootdURLConnection.java:95)
        at hep.io.root.daemon.xrootd.XrootdURLConnection.getInputStream(XrootdURLConnection.java:46)
        at hep.io.root.RootFileReader.<init>(RootFileReader.java:95)
        at hep.io.root.RootFileReader.<init>(RootFileReader.java:118)
        at hep.aida.ref.root.XrootdStore.read(XrootdStore.java:73)
        at hep.aida.ref.tree.Tree.init(Tree.java:1041)
        at hep.aida.ref.tree.TreeFactory.createTree(TreeFactory.java:183)
        at hep.aida.ref.tree.TreeFactory.create(TreeFactory.java:115)
        at hep.aida.web.taglib.util.TreeUtils.getTree(TreeUtils.java:80)
        at org.glast.dataqualitymonitoring.application.web.DataMonitoringWebApplication.getDataTree(DataMonitoringWebApplication.java:218)
  • Apr 21, 2008 1:16:37 PM
  • May 15, 2008 12:21:56 PM
  • May 15, 2008 1:11:59 PM
  • May 15, 2008 2:02:01 PM
  • May 15, 2008 2:52:03 PM
  • May 19, 2008 5:28:26 AM
  • May 21, 2008 many times between 11 AM and 8 PM
  • Jun 25, 2008 10:24:45 AM
  • Jun 25, 2008 11:03:06 PM
  • Jun 25, 2008 11:53:08 PM
  • Jun 26, 2008 12:43:11 AM
  • Jun 26, 2008 1:33:13 AM
  • Jun 26, 2008 2:23:16 AM
  • Jun 26, 2008 3:13:18 AM
  • Jun 27, 2008 11:25:22 AM
  • Jun 29, 2008 3:01:59 PM 
    hep.io.root.daemon.xrootd.Multiplexor handleSocketException
WARNING: [glast-rdr.slac.stanford.edu/134.79.200.91:-1:anonymous] Unexpected IO exception on socket
java.io.IOException: Socket closed
        at hep.io.root.daemon.xrootd.Multiplexor.sendMessage(Multiplexor.java:365)
        at hep.io.root.daemon.xrootd.Session$3.sendMessage(Session.java:189)
        at hep.io.root.daemon.xrootd.Session.redirectConnection(Session.java:104)
        at hep.io.root.daemon.xrootd.ResponseHandler.handleRedirect(ResponseHandler.java:19)
        at hep.io.root.daemon.xrootd.Multiplexor.run(Multiplexor.java:292)
        at java.lang.Thread.run(Thread.java:619)

    hep.io.root.daemon.xrootd.Multiplexor handleSocketException
WARNING: [glast-rdr.slac.stanford.edu/134.79.200.92:-1:anonymous] Unexpected IO exception on socket
java.io.IOException: Socket closed
        at hep.io.root.daemon.xrootd.Multiplexor.sendMessage(Multiplexor.java:398)
        at hep.io.root.daemon.xrootd.Session$3.sendMessage(Session.java:189)
        at hep.io.root.daemon.xrootd.Session.redirectConnection(Session.java:104)
        at hep.io.root.daemon.xrootd.ResponseHandler.handleRedirect(ResponseHandler.java:19)
        at hep.io.root.daemon.xrootd.Multiplexor.run(Multiplexor.java:321)
        at java.lang.Thread.run(Thread.java:619)

SEVERE: [wain017.slac.stanford.edu/172.23.2.17:1094:anonymous] multiplexor thread dead!
java.lang.ArrayIndexOutOfBoundsException
        at java.net.SocketInputStream.read(SocketInputStream.java:121)
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at hep.io.root.daemon.xrootd.Session$8.handleResponse(Session.java:311)
        at hep.io.root.daemon.xrootd.Multiplexor.run(Multiplexor.java:337)
        at java.lang.Thread.run(Thread.java:619)
  • No labels