UNDER CONSTRUCTION!

Overview and operation

• iocConsole is a tool (an EPICS Extension) that creates a persistent console session into an IOC, EIOC or SIOC which can be accessed by multiple users simultaneously. The session can also optionally log to a file: $IOC_DATA/iocname/screenlog.0.
• iocConsole connects to RTEMS or other "hard" IOCs by telnet-ing to the IOC's terminal server port.  iocConsole starts up a soft IOC process, and establishes a console session for that process.
• iocConsole uses the Linux screen program (http://www.slac.stanford.edu/grp/cd/soft/epics/extensions/iocConsole/screen.1.html) to create and manage the console sessions and logging.

iocConsole EPICS Extension doc: http://www.slac.stanford.edu/grp/cd/soft/epics/extensions/iocConsole/index.html

technical details: iocConsole has several moving parts:

• perl scripts that handle the console sessions (start, stop, cleanup, etc.)
• Linux screen sessions that runs the console
• pipe files (sockets) that associates the ioc name with the screen process id

basic commands

• Once your IOC is configured in the system (see Configuration below) you can start an iocConsole session like this:
  iocConsole iocname
  e.g.
  iocConsole sioc-sys0-al00

• (snip from the iocConsole EPICS Extension doc referenced above):
  In the iocConsole session, the most important screen commands are ("C-" below indicates the ctrl key)

• C-a d : Detach the screen session from this terminal (C-a C-d also works).
• C-a ? : Show all screen commands.
• C-a * : Show a list of all users attatched to this screen session.

• C-a [ : Enter copy/scrollback mode (C-a C-[ or C-a <esc> also works). In copy/scrollback mode you can use 'vi' or 'emacs' commands to move around the scrollback history. Some useful scrollback mode commands are:
  o Arrow keys move line by line or column by column.
  o h, j, k, l move line by line or column by column (vi-style).
  o + - move one line up/down.
  o C-u C-d move up/down one half a screen.
  o C-b C-f move up/down a full screen.
  o g moves to the beginning of the scrollback history.
  o % preceded by a number jumps to that percentage of the scrollback.
  o / or ? followed by some characters followed by a return searches forward or backwards for those characters.
  o <esc> exits copy/scrollback mode.

• Important: do not type "exit", or do ctrl-C from within a soft iocConsole session: this will terminate the soft IOC process!  Instead use C-a d to detach from the screen session.

• iocConsole -cleanup*
  • as mentioned above, iocConsole has a couple of components that need to be in sync in order to work properly:
    • screen process
    • pipe file (socket)
      Occasionally the screen process disappears, and the socket is left due to a system failure (e.g. reboot of lcls-daemon1) or other malfunction. The remedy is:
  • iocConsole iocname -cleanup
    This will clean up the inconsistent state, and when you
  • iocConsole iocname again, it should work.
  • please note: doing iocConsole -cleanup for a soft IOC will restart the IOC process - please use with care!

Configuration

screeniocs
To use iocConsole, your IOC must be configured in the system screeniocs file (contact Judy Rock to add your IOC to screeniocs)
When iocConsole starts up a session, it connects to the IOC using info found in the screeniocs file.

• screeniocs location :
  $IOC/All/Prod/screeniocs or facet_screeniocs

• screeniocs contents:
  • RTEMS IOC:
    • ioc name
    • terminal server
    • port
    • screen host (where the shared screen session runs)
  • Soft IOC
    • ioc name
    • executable
    • screen host (where the shared screen session and the soft IOC process run)

screen logging

• default screen logging parameters are found in the $IOC/Prod/All/screenrc file
• logging turned on is the default (not need for action on your part to make it happen)
  • to override defaults, copy screenrc into your own ioc's $IOC/iocname directory and edit it
  • for example, to turn off screen logging, in your own local copy comment out the "deflog on" line: "#deflog on"

Troubleshooting

iocConsole's error messages and conditions are not always 100% clear!  Here are a few situations you may encounter at some point:

• Your IOC hasn't been added to screeniocs yet:
  looks like this:
  [softegr@lcls-builder script]$ iocConsole xyz
  Error: iocConsole is terminating...ioc was not included in screeniocs file

• Unsuccessful attempt to connect to the terminal server for your hard IOC, with error message from iocConsole:
  looks like this:
  <when a real example arises, I will cut/paste it in here!>
  screen writes out a message with something like "blah blah blah no socket found blah blah blah"
  • Possible causes:
    • the IOC or terminal server are powered down, or not connected to the network
    • there is a hardware problem with the IOC
    • the IOC may be incorrectly entered in screeniocs (e.g. wrong port)
    • the terminal server port may need resetting.
  • Things to try:
    • check screeniocs to make sure the data for your IOC are correct
    • check to make sure the IOC and terminal server are up, and connected properly to the network (ping, etc.)
    • try iocConsole cleanup (see overview above for more info on cleanup):
      iocConsole iocname -cleanup
    • try resetting the terminal server port for the IOC:
      iocTSmgr iocname
      (you must be logged in as iocegr or fiocegr to run iocTSmgr)

• Hard IOC console session starts and connects to the terminal server, but there is to EPICS prompt or response from the IOC
  looks like this:
  Trying 172.27.68.102...
  Connected to ts-li20-nw02.slac.stanford.edu (172.27.68.102).
  Escape character is '^]'.
  < then no response to a carriage return or other IOC command >
  • Possible causes:
    • the IOC is powered down, or not connected properly to the terminal server
    • the IOC is having hardware or software problems and needs restarting, or other application-specific action
    • the IOC may be incorrectly entered in screeniocs (e.g. wrong port)
    • the terminal server port may need resetting.
  • Things to try:
    • check screeniocs to make sure the data for your IOC are correct
    • check to make sure the IOC and terminal server are up, and connected properly to the network (ping, etc.)
    • try resetting the terminal server port for the IOC (unlikely to help - but worth a try!)
      iocTSmgr iocname
      (you must be logged in as iocegr or fiocegr to run iocTSmgr)
    • Reboot or power cycle your IOC
• IOC console session stops logging (also see the maintence section below - logging is monitored daily by a cron job)
  looks like: screenlog.0 is missing OR screenlog.0 is not growing as expected
  There are a couple of possible causes:
• your IOC has its own copy of screenrc with "deflog on" commented out
  • check to see of there's a $IOC/iocname/screenrc file
  • if so, make sure the "deflog on" is not commented out ("#deflog on")
• the iocConsole session has terminated or is in an inconsistent state
  • for HARD IOC only do iocConsole -cleanup:
  • First do
    iocConsole iocname -cleanup
  • then
    iocConsole iocname
• the screen process logging function has hung up somehow (this happens most commonly for soft IOCs, but can probably occur for all IOCs)
  • if screenlog.0 exists, make a backup copy
    cd $IOC_DATA/iocname
    cp screenlog.0 screenlog.0SAVE
  • iocConsole iocname
  • within the iocConsole session, toggle logging off and then on again:
    ctrl-a H
    ctrl-a H
    then exit iocConsole with ctrl-a d

Maintenance and monitoring

A nightly cron job running the screenlogd.bash script handles screenlog.0 file cleanup and maintenance of screen logging iocConsole sessions:

• manages screenlog backups
  • screenlog.0 files larger than 1MB are cycled into a series of backup copies, with timestamp extensions. 5 backup copies are kept, with the oldest deleted first. If you would like to keep more than 5 copies, please copy them to another directory.

• manages iocConsole sessions to maintain screenlogging
  • "hard" IOCs
    • if a hard IOC's iocConsole screen session is found to be absent, screenlogd.bash attempts to restart an iocConsole session. If the restart fails, screenlogd.bash tries to reset the IOC's terminal server port using iocTSmgr, and then retries establishing an iocConsole screen session.
  • "soft" IOCs
    • screenlogd.bash does NOT restart iocConsole screen sessions for soft IOCs, since this will restart the IOC in production.
    • screenlogd.bash does report soft IOCs that are missing screenlog.0 files.
    • to re-establish logging for a soft IOC missing screenlog.0:
      • iocConsole to the soft IOC
      • in the IOC's console session:
        ctrl-H to stop screen logging
        ctrl-H again to restart screen logging
        ctrl-a d to exit from the iocConsole session