UNDER CONSTRUCTION!

Table of Contents

Overview and operation

iocConsole is an EPICS Extension that creates a persistent console session into an IOC, EIOC or SIOC which can be accessed by multiple users simultaneously from their workstations. The session can also optionally log to a file: $IOC_DATA/iocname/screenlog.0.
iocConsole connects to RTEMS or other "hard" IOCs by telnet-ing to the IOC's terminal server port.
iocConsole starts up a soft IOC process, and establishes a console session for that process.
iocConsole uses the Linux screen program (http://www.slac.stanford.edu/grp/cd/soft/epics/extensions/iocConsole/screen.1.html) to create and manage the console sessions and logging.

iocConsole EPICS Extension doc: http://www.slac.stanford.edu/grp/cd/soft/epics/extensions/iocConsole/index.html

contacts:

Judy Rock
Jingchen Zhou

technical details: an iocConsole session has several moving parts:

perl scripts that handle the session (start, stop, cleanup, etc.)
Linux screen process that run the console
pipe file (socket) that associates the ioc name with the screen process id

basic commands

Once your IOC is configured in the system (see Configuration below) you can start an iocConsole session like this:
iocConsole iocname
e.g.
iocConsole sioc-sys0-al00

(snip from the iocConsole EPICS Extension doc referenced above):
In the iocConsole session, the most important screen commands are ("C-" below indicates the ctrl key)

C-a d : Detach the screen session from this terminal (C-a C-d also works).
C-a ? : Show all screen commands.
C-a * : Show a list of all users attatched to this screen session.

C-a \[ : Enter copy/scrollback mode (C-a C-\[ or C-a <esc> also works). In copy/scrollback mode you can use 'vi' or 'emacs' commands to move around the scrollback history. Some useful scrollback mode commands are:

Arrow keys move line by line or column by column.
h, j, k, l move line by line or column by column (vi-style).
+ - move one line up/down.
C-u C-d move up/down one half a screen.
C-b C-f move up/down a full screen.
g moves to the beginning of the scrollback history.
% preceded by a number jumps to that percentage of the scrollback.
/ or ? followed by some characters followed by a return searches forward or backwards for those characters.

<esc> exits copy/scrollback mode.
Important: do not type "exit", or do ctrl-C from within a soft iocConsole session: this will terminate the soft IOC process! Instead use C-a d to detach from the screen session.

iocConsole iocname -cleanup
For hard IOCs only as mentioned above, iocConsole has a couple of components that need to be in sync in order to work properly. iocConsole -cleanup can help (see troubleshooting section below)
- please note: doing iocConsole -cleanup for a soft IOC will kill the IOC process - please use with care!

Configuration

screeniocs
To use iocConsole, your IOC must be configured in the system screeniocs file (contact Judy Rock to add your IOC to screeniocs in production)
When iocConsole starts up a session, it connects to the IOC using info found in the screeniocs file.

screeniocs location :
$IOC/All/Prod/screeniocs or facet_screeniocs

screeniocs contents:
- RTEMS IOC:
  - ioc name
  - terminal server
  - port
  - screen host (where the shared screen session runs)
- Soft IOC
  - ioc name
  - executable
  - screen host (where the shared screen session and the soft IOC process run)

screen logging

default screen logging parameters are found in the $IOC/Prod/All/screenrc file
logging turned on is the default (not need for action on your part to make it happen)
- to override defaults, copy screenrc into your own ioc's $IOC/iocname directory and edit it
- for example, to turn off screen logging, in your own local copy comment out the "deflog on" line: "#deflog on"

Troubleshooting

iocConsole's error messages and conditions are not always 100% clear! Here are a few situations you may encounter at some point:

*Your IOC hasn't been added to screeniocs yet:*
looks like this:
\[softegr@lcls-builder script\]$ iocConsole xyz
Error: iocConsole is terminating...ioc was not included in screeniocs file

Unsuccessful attempt to start an iocConsole session, with error message from iocConsole:
looks like this:
<when a real example arises, I will cut/paste it in here!>
screen writes out a message with something like "blah blah blah no socket found blah blah blah"
- Possible causes: hard iocs
  - the IOC or terminal server are powered down, or not connected to the network
  - there is a hardware problem with the IOC
  - the IOC may be incorrectly entered in screeniocs (e.g. wrong port)
  - the terminal server port may need resetting.
  - prior iocConsole session may need cleaning up
- Things to try:
  - check screeniocs to make sure the data for your IOC are correct
  - check to make sure the IOC and terminal server are up, and connected properly to the network (ping, etc.)
  - try resetting the terminal server port for the IOC:
    iocTSmgr iocname
  - try iocConsole -cleanup
    (you must be logged in as iocegr or fiocegr to run iocTSmgr)
    iocConsole iocname -cleanup
- Possible causes: soft iocs
  - the IOC may be incorrectly entered in screeniocs (e.g. wrong executable or host)
  - prior iocConsole session may need cleaning up
- Things to try:
  - check screeniocs to make sure the data for your IOC are correct
  - try iocConsole -cleanup
    iocConsole iocname -cleanup
  - try resetting the terminal server port for the IOC:
    iocTSmgr iocname
    (you must be logged in as iocegr or fiocegr to run iocTSmgr)

Hard IOC console session starts and connects to the terminal server, but there is to EPICS prompt or response from the IOC
looks like this:
Trying 172.27.68.102...
Connected to ts-li20-nw02.slac.stanford.edu (172.27.68.102).
Escape character is '^]'.
< then no response to a carriage return or other IOC command >
- Possible causes:
  - the IOC is powered down, or not connected properly to the terminal server
  - the IOC is having hardware or software problems and needs restarting, or other application-specific action
  - the IOC may be incorrectly entered in screeniocs (e.g. wrong port)
  - the terminal server port may need resetting.
- Things to try:
  - check screeniocs to make sure the data for your IOC are correct
  - check to make sure the IOC and terminal server are up, and connected properly to the network (ping, etc.)
  - try resetting the terminal server port for the IOC (unlikely to help - but worth a try!)
    iocTSmgr iocname
    (you must be logged in as iocegr or fiocegr to run iocTSmgr)
  - Reboot or power cycle your IOC

IOC console session stops logging (also see the maintence section below - logging is monitored daily by a cron job)
looks like: screenlog.0 is missing OR screenlog.0 is not growing as expected
There are a couple of possible causes:
- your IOC has its own copy of screenrc with "deflog on" commented out
  - check to see of there's a $IOC/iocname/screenrc file
  - if so, make sure the "deflog on" is not commented out ("#deflog on")
- the iocConsole session has terminated or is in an inconsistent state
  - for HARD IOC only do iocConsole -cleanup:
  - First do
    iocConsole iocname -cleanup
  - then
    iocConsole iocname
- the screen process logging function has hung up somehow (this happens most commonly for soft IOCs, but can probably occur for all IOCs)
  - if screenlog.0 exists, make a backup copy
    cd $IOC_DATA/iocname
    cp screenlog.0 screenlog.0SAVE
  - iocConsole iocname
  - within the iocConsole session, toggle logging off and then on again:
    ctrl-a H
    ctrl-a H
    then exit iocConsole with ctrl-a d

Maintenance and monitoring

A nightly cron job running the screenlogd.bash script handles screenlog.0 file cleanup and maintenance of screen logging iocConsole sessions:

manages screenlog backups
- screenlog.0 files larger than 1MB are cycled into a series of backup copies, with timestamp extensions. 5 backup copies are kept, with the oldest deleted first. If you would like to keep more than 5 copies, please copy them to another directory.

manages iocConsole sessions to maintain screenlogging
- "hard" IOCs
  - if a hard IOC's iocConsole screen session is found to be absent, screenlogd.bash attempts to restart an iocConsole session. If the restart fails, screenlogd.bash tries to reset the IOC's terminal server port using iocTSmgr, and then retries establishing an iocConsole screen session.
- "soft" IOCs
  - screenlogd.bash does NOT restart iocConsole screen sessions for soft IOCs, since this will restart the IOC in production.
  - screenlogd.bash reports soft IOCs that are missing screenlog.0 files (by e-mail to Judy Rock and Jingchen Zhou)
  - to re-establish logging for a soft IOC missing screenlog.0:
    - iocConsole to the soft IOC
    - in the IOC's console session:
      ctrl-a H to stop screen logging
      ctrl-a H again to restart screen logging
      ctrl-a d to exit from the iocConsole session