LCLS Alarm Handling Configuration

References

• LCLS Use of Alarm Handler
• EPICS Extensions at LCLS
• LCLS Naming Convention

Subsystems

• Beam Containment System Alarms (BCS)
• Magnets Alarms
• Machine Protection System Alarms (MPS)
• Personnel Protection System Alarms (PPS)
• Vacuum Alarms
• Profile Monitor Status (see Please Note section above)
• Laser Alarms
• RF Alarms
• Utilities Alarms
• Temperature Alarms

Display

• Prototypes
• Alarm Handler Display
• Subsystem Alarm Handler Display
• EDM
  • LCLS Home Alarm Grid
  • LCLS Alarm Grid CUD Display (same layout but bigger)

CVS for ALH Configuration Files Reference directory: /afs/slac/g/lcls/tools/alh/config

Guidelines for adding PVs to the alarm handler

• How to decide which PVs should be in the alarm handler: Think of the alarm handler as a tool to help operators and physicists keep the accelerator running and producing good quality beam. It should be a list of things that they care about. Try to keep the list short. Instead of adding every PV associated with a particular device, just add the PVs that are important to operations.

• How the PVs you put in the alarm handler should behave (color/severity standards): All the PVs in the alarm handler must have meaningful severity defined in the IOC databases. Think of the severity from the point of view of the operators and physicists who are running the accelerator.

• • Red = Major Alarm. Fix it immediately.
    • Something is holding the beam off (vacuum valve closed, stopper inserted, safety system interlock)
    • There is a problem that seriously affects beam quality (magnet far out of tolerance, critical feedback loop not running, cooling water outside of allowed temperature range, undulator beam orbit outside allowed aperture)
  • Yellow = Minor Alarm. Take notice.
    • Be aware of unusual operating conditions (profile monitor inserted, wire scanner in the beam)
    • There is a problem that does not affect beam quality but should be fixed (light bulb burned out on profile monitor)
    • A device is near a threshold for a red alarm (magnet drifting out of tolerance, water heating up)
  • Green = No Alarm.
    • Everything is fine for running the beam.

Info needed to define a subsystem Alarm Handler Tree

• Subsystem name
• PVs: can supply a list of PVs, or for IOCs that are being crawled into the IRMIS database (should be all LCLS IOCs), can use wildcards.
• For each PV, to log or not to log to cmlog (logging will be the default)
• Area/beam region, device type and device level PV groupings; other groupings if appropriate
• Associated edm display names at any level

see below for more details about each aspect

Alarm Handler Tree Structures
There will be different tree structures for different purposes. In at least one of the available tree structures, the ALH alarm status at the area/region level will need to match the alarm status on the CUD display. A PV or subtree can be present in more than one tree. However, please note that a PV should only log changes once to cmlog, i.e. logging should be turned off for all but one entry in the Alarm Tree.

Tree structure I: device-level PVs
Useful for debugging device failure.

• LCLS
  • Mode*
    • Area/beam region*
      • Subsystem
        • Device type
          • Device unit (subsystem dependent)
            • Device Attribute PVs

Possibly, Tree structure II: summary PVs
Useful for identifying which devices or subsystems are failing.

• LCLS
  • Mode*
    • Area/beam region* with summary PVs
      • Device type with device level summary PVs

*

IN20, LI21 and BSY Beam Regions
For Alarming purposes, IN20, LI21 and BSY both contain multiple beam regions. Otherwise, beam regions correspond to previously-defined areas .
Here are the Beam Regions, their start and end points as locations in the MAD deck, and their abbreviations for the alhConfig filenames (for example, mgnt_in20Gr_bend.alhConfig)

wiki page listing MAD names + control system names for IN20 which may be helpful.
Other regions in the alarm handler and lclshome correspond to the usual set of AREAs (e.g. LI23, LI30)

*

Modes
On the alarm handler display, the operators will be able to see the status/readiness of various operational modes. Under each mode are the alarm sub-trees for devices required. Alarm sub-trees may be present in multiple modes. The modes are:

• All LCLS Devices
• Gun Spectrometer Mode
• Injection Spectrometer Mode
• TD11 Mode
• BSY Mode

The alh logging instance uses the "All LCLS Devices" mode tree only. All PVs should be present in the All LCLS Devices mode tree.

Acknowledging Alarms
ALH instances started from control room OPIs will run in global mode, which allows globally visible alarm acknowledgement. Other ALH instances will be run in local passive mode, and not allow acknowledgement. See LCLS use of Alarm Handler Alarm Handler Processes for running in these modes.

Configuration File Naming Convention
Production alhConfig files are located in /usr/local/lcls/tools/alh/config (and in CVS under tools/alh/config)

In general:

• Top level: LCLS_<function>.alhConfig (e.g. LCLS_all.alhConfig)
• Area level: <area>.alhConfig (li22.alhConfig)
• Subsystem level: <subsystem>_<area>.alhConfig (mgnt_li22.alhConfig)
• Device type level: <subsystem><area><dev type>.alhConfig (mgnt_li22_bend.alhConfig)

• Below this level, it's a choice of:
  • make further levels of separate config files
  • combine alarm handler groups in a single config file (e.g. see mgnt_li21L1Bc1.alhConfig)

Summary PVs and Alarm Group Naming Conventions

• Summary PV databases are created by perl scripts from alhConfig files, with a summary PV for each node in the tree (group).
• The summary PVs are calcs that return 0, with maximize severity for INPs (standard summary method).
• !! !Since Summary PV definitions are derived from the alhConfig GROUP names, it's best to adhere to the naming convention below for GROUP names!!!
• Summary PV values
  • ALH group sevr value 0 = NO_ALARM
  • ALH group sevr value 1 = MINOR
  • ALH group sevr value 2 = MAJOR
  • ALH group sevr value 3 = INVALID
  • ALH group sevr value 4 = INVALID
• Summary PVs are booted into IOCs sioc-sys0-al00 and sioc-sys0-al01 (prod, on lcls-daemon1); sioc-sys0-al0d (on lclsdev) can be used for development.

• This naming convention covers summary PV names and corresponding alarm group names. The alarm group name used in the .alhConfig files is used to autogenerate Summary PV names; however, more friendly aliases may be used in the alh display.

• 1. Device (related measurements) means PVs associated with a device, but part of a different subsystem, for example, temperatures of magnets or RF devices. In the alh GUI, the device-level group is aliased to show the device name. But the group and Summary PV names are uniquely identified by appending the subsystem abbreviation to the device type. See temp_*.alhConfig files for examples.

Device level status PV

• In addition, a device level status string to be written to cmlog on change (by Channel Watcher):
  • <device type>:<area>:<position>:STATMSG
  • For example BEND:LI21:215:STATMSG

SUMMARIZED PVs and the linkage between alh trees and lclshome: DP PVs
Enabling/disabling of alarm trees is propagated to the lclshome alarm summaries by this mechanism (and see below for details on enable/disable):
The PVs that participate in the summaries are derived from the original subsystem PVs, with DP appended to the name e.g. VGPR:IN20:STATUSDP.
The DP PV alarm statuses are enabled or disabled depending on the value of the corresponding FORCEPVs (using the DISV, SDIS, DISS fields). If a FORCEPV is set to 1 (disabled) the DP PV alarm state is set to NO_ALARM, and it effectively drops out of the summaries above. When the FORCEPV is set to 0 (enabled) the DP PV alarm state reflects the alarm state of the original source PV.

• Naming Convention
  • <PV name>DP e.g. BEND:LI22:215:BACTDP

FORCEPVs
The config file $FORCEPV option allows segments of the alarm tree to be temporarily disabled/enabled. I.e., the PVs in the tree are eliminated on the fly from contributing to the alarm display and from logging to cmlog. A FORCE PV triggers a change of the ALH channel. Due to ALH gui limitations, FORCEPVs will be defined only on the individual PV level.

• Naming Convention
  • <PV name>FP e.g. BEND:LI22:215:BACTFP

• Details
  • FORCE PV record definition is in alhFORCEPV.db
  • FORCE PVs are BI records, with value 1 disabling alarming.
  • FORCE PV templates to instantiate the PVs will be autogenerated from alhConfig files.
  • FORCE PVs area be booted into AL00/AL0D.
  • FORCE PVs are save/restored using Channel Watcher, so that if the FORCE PV soft IOC is rebooted, the state of the alarm tree is preserved.
  • FORCE PV values can be put using the disableALHGroup and enableALHGroup shell scripts
  • Scripts and processes external to the alarm system may make use of the FORCE PVs to selectively disable alarming and logging.

FORCEPVs and DP PVs are generated from the same database file, alhFORCEPV.db

FORCEPV summaries: STATSUMYFP PVs
FORCEPV alarm severities are:

• enabled: NO_ALARM
• disabled: MINOR

FORCEPV severities are summarized at each level of the tree, with a STATSUMYFP PV for each STATSUMY PV in the system. On the lclshome display, STATSUMYFP for each subsystem/area (colored box) is used to show that an alarm sub-tree in the corresponding subsystem/area has been disabled: an asterisk is displayed. The asterisk disappears when the sub-tree is enabled. To track exactly what has been disabled, use the alh gui.

enable/disable scripts: disabling/enabling alarming and logging on the fly
To disable/enable alarming for an alh group:

• disableALHGroup groupname
• enableALHGroup groupname

Groupname can be found in the "G"uidance popup for each group, or for devices, simply use the device name if no "G" button is present. You can also specify an individual PV name.
The script will prompt for confirmation; enter Y to confirm.

After running disableALHGroup you will see:

• "*" appears in the corresponding lclshome subsystem/area alarm color box (use ALH to drill down and see the level that was disabled)
• -DATL mask appears for the tree (no alarming, no logging), and up the tree above
• group no longer shows alarms, and group alarms no longer sum up the tree
• a Channel Watcher cmlog entry for each PV that was disabled by the operation
• alarm logging to cmlog stops

After running enableALHGroup you will see:

• "*" in the corresponding lclshome subsystem/area alarm color box disappears
• ALH masks return to their normal state, as configured in ALH.
• group will show alarms, with group alarms summing up the ALH tree
• logging to cmlog resumes

ALH disable/enable operations are propagated to the lclshome alarm colors: see the "SUMMARIZED PVs: linkage between alh trees and lclshome" section above for details.

(A new version of ALH from SNS will be put into production soon that allows multiple options from the "P" button. The enable and disable scripts will be added to the dropdown list for each group...coming eventually!)

Related edm displays
Clicking the P button at any level of the alarm tree will launch the appropriate associated edm display.
Soon a new version of ALH will be brought in from SNS that allows dropdown lists from the P button. To see the related edm display, select "edm display" from the dropdown.

ALH Scripts

Adding to the alarm tree, and useful scripts
Some useful scripts to assist in adding subsystems to the alarm tree can be found in /afs/slac/g/lcls/tools/alh/script. README describes the scripts, plus what needs to be done to add elements to the tree. These helper scripts are not necessarily production-quality at the moment; they are made availabe as-is - please inspect output before putting into the production alarm tree! (-:

changeMasks.pl changes masks for Channels that match a wildcard. Useful for, e.g., changing all magnet
"ACT" channels to turn of logging.

Meeting Notes

• January 24,2007
• February 1, 2007 Controls meeting: Alarming topic - see Attachment 1 to this wiki page