Versions Compared

Old Version 1

changes.mady.by.user Unknown User (jrock)

Saved on

compared with

New Version Current

changes.mady.by.user Unknown User (jrock)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin

...

Panel

Table of Contents

Table of Contents

Overview 

VNC = Virtual Network Computing http://en.wikipedia.org/wiki/VNC enables remote connection you to set up a connection from a remote host (e.g. a PC offsite) to a session on a SLAC Linux host.

iocConsole EPICS Extension doc: http://www.slac.stanford.edu/grp/cd/soft/epics/extensions/iocConsole/index.html

contacts:

  • Judy Rock
  • Jingchen Zhou

technical details:
an iocConsole session has several moving parts:

  • perl scripts that handle the session (start, stop, cleanup, etc.)
  • Linux screen process that runs the console
  • pipe file (socket) that associates the ioc name with the screen process id

basic commands
Once your IOC is configured in the system (see Configuration below) you can start an iocConsole session like this:
iocConsole iocname
e.g.
iocConsole sioc-sys0-al00

note: doing iocConsole to a soft IOC that is down will start it!

  • (snip from the iocConsole EPICS Extension doc referenced above):
    In the iocConsole session, the most important screen commands are ("C-" below indicates the ctrl key)
    • C-a d : Detach the screen session from this terminal (C-a C-d also works).
    • C-a ? : Show all screen commands.
    • C-a * : Show a list of all users attatched to this screen session.
    • Wiki Markup
      C-a \[ : Enter copy/scrollback mode (C-a C-\[ or C-a <esc> also works). In copy/scrollback mode you can use 'vi' or 'emacs' commands to move around the scrollback history. Some useful scrollback mode commands are:
      • Arrow keys move line by line or column by column.
      • h, j, k, l move line by line or column by column (vi-style).
      • + - move one line up/down.
      • C-u C-d move up/down one half a screen.
      • C-b C-f move up/down a full screen.
      • g moves to the beginning of the scrollback history.
      • % preceded by a number jumps to that percentage of the scrollback.
      • / or ? followed by some characters followed by a return searches forward or backwards for those characters.
    • <esc> exits copy/scrollback mode.
    • Important: do not type "exit", or do ctrl-C from within a soft iocConsole session: this will terminate the soft IOC process!  Instead use C-a d to detach from the screen session.
  • iocConsole iocname -cleanup
    as mentioned above, iocConsole has a couple of components that need to be in sync in order to work properly. iocConsole iocname -cleanup can help (see troubleshooting section belo)
    • please note: doing iocConsole -cleanup will kill any concurrent iocConsole sessions - i.e. another colleague interacting with the IOC.
    • please note: doing iocConsole -cleanup for a running soft IOC will kill the IOC process - please use with care!

Configuration

screeniocs
To use iocConsole, your IOC must be configured in the system screeniocs file (contact Judy Rock to add your IOC to screeniocs in production)
When iocConsole starts up a session, it connects to the IOC using info found in the screeniocs file.  screeniocs is CVS-ed.

  • screeniocs location :
    $IOC/All/Prod/screeniocs or facet_screeniocs
  • screeniocs contents:
    • RTEMS IOC:
      • ioc name
      • terminal server
      • port
      • screen host (where the shared screen session runs)
    • Soft IOC
      • ioc name
      • executable
      • screen host (where the shared screen session and the soft IOC process run)

screen logging
logging turned on is the default (not need for action on your part to make it happen)

  • default screen logging parameters are found in the $IOC/Prod/All/screenrc file
    • to override defaults, copy screenrc into your own ioc's $IOC/iocname directory and edit it
    • for example, to turn off screen logging, in your own local copy comment out the "deflog on" line: "#deflog on"

Troubleshooting

iocConsole's error messages and conditions are not always 100% clear! 
Feel free to call Judy Rock (or Jingchen Zhou) for assistance.

Here are a few situations you may encounter at some point:

  • Wiki Markup
    *Your IOC hasn't been added to screeniocs yet:*
looks like this:
\[softegr@lcls-builder script\]$ iocConsole xyz
Error: iocConsole is terminating...ioc was not included in screeniocs file
  • Unsuccessful attempt to start an iocConsole session, with no socket found error message from iocConsole:
    looks like this:
    screen writes out a message with something like "blah blah blah no socket found blah blah blah" and returns to the Linux prompt.
    <when a real example arises, I will cut/paste it in here!>
    • Possible causes: soft iocs - for a soft ioc, this error condition often indicates that the soft IOC cannot start due to some IOC software or configuration problem, or has hung in some way.
      • the IOC process has crashed.
      • rare: the IOC process has hung, but is still "running"
      • the IOC is incorrectly entered in screeniocs (e.g. wrong directories or executable)
    • Things to try soft iocs:
      • check screeniocs to make sure the data for your IOC are correct
      • have a look at the screenlog file: $IOC_DATA/iocname/screenlog.0. There are often error messages from the IOC indicating why it's having problems. Problems previously observed in screenlog.0 include:
      • $IOC/iocname bin directory or executable name not found (problem with make)
      • segmentation fault - IOC executable cannot run
      • for a hung but running soft ioc: as <adminuser> on <softIOCmachine>:  ps -ef | grep <iocname>  
      • If a process exists, but you can't iocConsole to it, this indicates a hang in the process. You'll need to kill the process from the <adminuser>@softIOCmachine command line. Then try iocConsole again.
    • Possible causes: hard iocs
      • the IOC or terminal server are powered down, or not connected to the network
      • there is a hardware problem with the terminal server or IOC
      • the IOC is incorrectly entered in screeniocs (e.g. wrong port)
      • the terminal server port needs resetting.
      • prior iocConsole session needs cleaning up
    • Things to try: hard iocs
      • check screeniocs to make sure the data for your IOC are correct
      • check to make sure the IOC and terminal server are up, and connected properly to the network (ping, etc.)
      • try resetting the terminal server port for the IOC:
        iocTSmgr iocname
      • try iocConsole -cleanup
        (you must be logged in as iocegr or fiocegr to run iocTSmgr)
      • iocConsole iocname -cleanup
  • Unsuccessful attempt to start an iocConsole session, with dead screen error message from iocConsole:
    looks like this:
    ...
    iocScreen: /home/screen/bin/screen -r -S laci/ioc-li17-ky00
    There is a screen on:
    10660.ioc-li17-ky00 (Dead ???)
    Remove dead screens with 'screen -wipe'.
    There is no screen to be attached matching ioc-li17-ky00.
    ...
    • Cause:
      • a previously running SCREEN session was killed without deleting its socket pipe file.
    • To fix:
      • iocConsole <iocname> -cleanup
      • please note: this will kill a soft IOC that is running. (unlikely that the ioc can run under these circumstances though)
  • Hard IOC console session starts and connects to the terminal server, but there is no EPICS prompt or response from the IOC
    looks like this:
    Trying 172.27.68.102...
    Connected to ts-li20-nw02.slac.stanford.edu (172.27.68.102).
    Escape character is '^]'.
    < then no response to a carriage return or other IOC command >
    • Possible causes:
      • the IOC is powered down, or not connected properly to the terminal server
      • the IOC is having hardware or software problems and needs restarting, or other application-specific action
      • the IOC is incorrectly entered in screeniocs (e.g. wrong port)
      • the terminal server port needs resetting.
    • Things to try:
      • check screeniocs to make sure the data for your IOC are correct
      • check to make sure the IOC and terminal server are up, and connected properly to the network (ping, etc.)
      • try resetting the terminal server port for the IOC
        iocTSmgr iocname
        (you must be logged in as iocegr or fiocegr to run iocTSmgr)
      • Reboot or power cycle your IOC
  • IOC console session stops logging (also see the maintence section below - logging is monitored daily by a cron job)
    looks like this: screenlog.0 is missing OR screenlog.0 is not growing as expected
    There are a couple of possible causes:
    • your IOC has its own copy of screenrc with "deflog on" commented out
      • check to see of there's a $IOC/iocname/screenrc file
      • if so, make sure the "deflog on" is not commented out ("#deflog on")
    • the iocConsole session has terminated or is in an inconsistent state
      • for HARD IOC only do iocConsole -cleanup:
      • First do
        iocConsole iocname -cleanup
      • then
        iocConsole iocname
    • the screen process logging function has hung up somehow (this happens most commonly for soft IOCs, but can probably occur for all IOCs)
      • if screenlog.0 exists, make a backup copy
        cd $IOC_DATA/iocname
        cp screenlog.0 screenlog.0SAVE
      • iocConsole iocname
      • within the iocConsole session, toggle logging off and then on again:
        ctrl-a H
        ctrl-a H
        then exit iocConsole with ctrl-a d

Maintenance and monitoring

iocConsole hosts and sessions

  • LCLS
    • All LCLS hard and soft IOC processes and screen sessions run on lcls-daemon1 EXCEPT:
    • The klystron support soft IOC ?processes and screen sessions run on lcls-daemon0 (due to high loads)
    • The process and screen session host is configured by the entry in the screeniocs file "Host where screen is run" column (the last column)
  • FACET
    • All FACET hard and soft IOC processes and screen sessions run on facet-daemon1.

screenlogd.bash
For each accelerator facility (LCLS and FACET) a nightly cron job running the screenlogd.bash script handles screenlog.0 file cleanup and maintenance of screen logging iocConsole sessions:

  • manages screenlog backups
    • screenlog.0 files larger than 1MB are cycled into a series of backup copies, with timestamp extensions. 5 backup copies are kept, with the oldest deleted first. If you would like to keep more than 5 copies, please copy them to another directory.
  • manages iocConsole sessions to maintain screenlogging
    • "hard" IOCs
      • if a hard IOC's iocConsole screen session is found to be absent, screenlogd.bash attempts to restart an iocConsole session. If the restart fails, screenlogd.bash tries to reset the IOC's terminal server port using iocTSmgr, and then retries establishing an iocConsole screen session.
    • "soft" IOCs
      • screenlogd.bash does NOT restart iocConsole screen sessions for soft IOCs, since this will restart the IOC in production.
      • screenlogd.bash reports soft IOCs that are missing screenlog.0 files (by e-mail to Judy Rock and Jingchen Zhou)
      • to re-establish logging for a soft IOC missing screenlog.0:
        • iocConsole to the soft IOC
        • in the IOC's console session:
          ctrl-a H to stop screen logging
          ctrl-a H again to restart screen logging
          ctrl-a d to exit from the iocConsole session

iocConsole mass restart scripts
If a server that supports iocConsole (lcls-daemon1, lcls-daemon0, facet-daemon1) goes down for some reason, iocConsole sessions are often left in an inconsistent state.
When the servers come back up again:

  For access to control system components such as edm displays or matlab programs, VNC often provides the best performance.
The elements are:

  • The vncserver runs a KDE session on a linux host.
  • The VNC Viewer runs on your PC, and connects to the Linux session, allowing you to do work on the Linux host.

This document contains instructions for setting up a connection from a Windows PC to a VNC session on a public Linux host at SLAC.  

IMPORTANT, PLEASE NOTE:

  • Due to timeouts associated with things like afs tokens, kerberos tickets, vpn, and ssh tunnelling, vncserver sessions left running can become inaccessible to VNC Viewers, and will to be restarted. Troubleshooting section below has some possibilities for getting around the problem, but they are not always successful. In any case, please do not leave unsaved work in your session.
  • The iris group of SCCS machines are equipped with fonts necessary for running edm displays like lclshome. Other machines like noric or yakut do not have these fonts.

Set up your PC (one time)

Configure your vncserver (one time)

  • Using putty (or XWin-32 if you prefer) log into iris01.slac.stanford.edu, providing your unix username and password
  • Create your vnc password:
    • > vncpasswd
      Provide a password following the usual SLAC password guidelines.
      The password will be stored in ~/.vnc/passwd
      To reset the password, you can run vncpasswd again.
  • Protect the password:
    • > fs setacl ~/.vnc system:slac none
    • > fs setacl ~/.vnc system:authuser none
  • Start vncserver to create your xstartup file:
    • > vncserver -localhost -nolisten tcp -geometry 1440x900
    • Make note of the display number that is returned, for example:
      jrock@iris01> vncserver -localhost -nolisten tcp -geometry 1440x900
      New 'iris01:3 (jrock)' desktop is iris01:3
      Starting applications specified in /u/cd/jrock/.vnc/xstartup
      Log file is /u/cd/jrock/.vnc/iris01:3.log
      In this case, 3 is the display number, which is a unique ID for your particular vncserver session.
      In the following instructions, the display number is indicated by displaynum
  • Kill the server:
    vncserver -kill :displaynum
    for example
    vncserver -kill :3
  • Edit your vnc xstartup file to invoke kde at startup:
    • > emacs ~/.vnc/xstartup
      change last line from "twm" & to "startkde&", save and close

Run your vnc server (as needed)

As needed.

The session will continue to run until you -kill it, or until SCCS does.
Warning - tokens, kerberos tickets, vpn, and ssh tunnelling can all conspire to prevent you from reconnecting to an existing vncserver session. See troubleshooting section below. The safest thing is to re-start a vncserver for every use, and -kill it when done.

Use VNC Viewer to connect from Windows (every time you connect)

Every time you need a new session.

  • Invoke a command window on your PC:
    • click the Start Button in the lower lefthand corner of your screen
    • enter Command in the search box
    • select "Command line tools" from the list
  • In the command window, use putty to create a secure scp tunnel to the vncserver host and vncserver port:
    • putty -ssh -L pcport:localhost:59displaynum vncserverhost.slac.stanford.edu
      • pcport = port on your pc, e.g. 5902
      • displaynum = vncserver session as noted when you started the vncserver
      • vncserverhost = host where vncserver is running
    • for example
      putty -ssh -L 5902:localhost:5903 iris01.slac.stanford.edu
      or
      putty -ssh -L 5902:localhost:5912 iris02.slac.stanford.edu
      **Enter your unix username and password.
  • Now run the VNC Viewer (click the shortcut)
    • In the Server box, enter localhost:pcport
      for example localhost:5902
    • click OK
    • The vnc viewer authentication popup will appear; this can take some time.
    • Enter your vnc server password (you don't need to enter username in the popup).
  • Your linux desktop should appear.
  • When you're ready to close the vnc client, simply click the window close icon, "X", at the upper righthand corner of the display.

Troubleshooting

Here are some VNC Viewer errors you may encounter.

  • "No password configured for VNC Auth"
    • this means that your session has lost the ability to read the vnc password file, or can't find the pw file. Here are some things to try:
      • Verify you are still connected to SLAC via vpn
      • Verify you're putty-ed into the right host and port for your vncserver, and you've specified the right pc port in the vnc client popup.
      • log into the vncserver host and verify your that vncserver session exists:
        ps -ef | grep vnc | grep yourusername
      • If the process is not running, then restart it as detailed above and try connecting again.
      • If the process is running, you can try to refresh the vnc password (use vncpasswd command, as detailed above)  You can use the same password, or a different one - the goal is to touch the password file. Close your putty ssh sessions, and re-establish the tunnel to the vncserver port.
    • If none of these steps are successful, then you will have to -kill the vncserver, restart it, and then reconnect. See above for instructions.
  • "Unable to connect to host: Connection refused (10061)"
    • You don't have an ssh tunnel established to the host and port.
  • "The connection closed unexpectedly" (at VNC Viewer startup)
    • You're logged into the wrong displaynum port corresponding to your vncserver.
  • edm displays (e.g. lclshome) come up without text
    • Not all SCCS machines have the fontset for edm. Please use the iris machines.

References 

SLAC Linux VNC page: http://www.slac.stanford.edu/comp/unix/vnc.html

contacts:

  • Judy Rock (x3639)
  • Mike Zelazny (x3673)
  • Jingchen Zhou (x4661)
  • soft IOCs are automatically restarted upon system startup, which includes an automatic cleanup and restart of their iocConsole sessions.
  • hard IOCs require manual intervention to cleanup and restart their iocConsole sessions (for screenlogging). Here are the scripts that run the cleanup and iocConsole/screenlogging restart en masse, that should be run after system restart.  They start iocConsole console sessions and will execute iocTSmgr to reset terminal server ports that need it.
  • LCLS
    /usr/local/lcls/tools/script/restartALLIOCLogging.bash
    • FACET
    /usr/local/facet/tools/script/restartALLIOCLogging_facet.bash

...