Page History

...

Maintenance and monitoring

screenlogd.bash
A nightly cron job running the screenlogd.bash script handles screenlog.0 file cleanup and maintenance of screen logging iocConsole sessions:

...

• manages iocConsole sessions to maintain screenlogging
  • "hard" IOCs
    • if a hard IOC's iocConsole screen session is found to be absent, screenlogd.bash attempts to restart an iocConsole session. If the restart fails, screenlogd.bash tries to reset the IOC's terminal server port using iocTSmgr, and then retries establishing an iocConsole screen session.
  • "soft" IOCs
    • screenlogd.bash does NOT restart iocConsole screen sessions for soft IOCs, since this will restart the IOC in production.
    • screenlogd.bash reports soft IOCs that are missing screenlog.0 files (by e-mail to Judy Rock and Jingchen Zhou)
    • to re-establish logging for a soft IOC missing screenlog.0:
      • iocConsole to the soft IOC
      • in the IOC's console session:
        ctrl-a H to stop screen logging
        ctrl-a H again to restart screen logging
        ctrl-a d to exit from the iocConsole session

iocConsole mass restart scripts
If a server that supports iocConsole (lcls-daemon1, lcls-daemon0, facet-daemon1) goes down for some reason, iocConsole sessions are often left in an inconsistent state.
When the servers come back up again:

• soft IOCs are automatically restarted, which includes an automatic cleanup of their iocConsole sessions.
• hard IOCs require manual intervention to cleanup and restart their iocConsole sessions (for screenlogging). Here are the scripts that run the cleanup and restart en masse, that should be run after system restart:
  • LCLS
    /usr/local/lcls/tools/script/restartALLIOCLogging.bash
  • FACET
    /usr/local/facet/tools/script/restartALLIOCLogging_facet.bash

...