Learning Objective: This module is intended to teach operators how to use the Cavity Summary Display to monitor and diagnose issues with cavities.
Skills:
- Know how to launch the GUI
- Understand the information presented on the GUI
- Know how to interpret faults and see the status of all potential faults associated with a cavity
- Understand how to restart backend if the display stops updating
Introduction:
Figure 1: Overview of Cavity Display GUI
The Cavity Summary Display is similar to the Klystron displays used in LCLS and FACET, but instead of displaying klystron information, the Cavity Summary Display shows cavity faults for all cavities in cryomodules 01 - 35. The GUI also displays whether a cavity's SSA is on and whether or not is has RF.
How to Launch the GUI:
The easiest way to launch the GUI is by typing "SC Cavity Summary Display" into Launchpad.
Alternatively, you can manually launch the GUI from SRF Home:
- Open a terminal
- Type in set_profile srf
- Type srfhome. This will launch a PyDM version of SRF Home
- Click "Cavity Summary GUI."
Interpreting Information on the GUI:
Cavity Shapes and Colors:
Now that we've launched the GUI, we want to use it to help us troubleshoot and keep an eye on the cavities. The GUI is broken up into the four linac regions (L0B, L1B, L2B, and L3B) each with their own cryomodules (abbreviated as CM). Each of these cryomodules has 8 cavities. If a cavity is faulted, it will display a Three Letter Code (TLC) instead of its cavity number.
The different cavity statuses are as follows:
- Green diamond with cavity number - This indicates that the cavity is healthy and has no faults.
- Yellow triangle with TLC - Cavity has a minor fault.
- Red hexagon with TLC - Cavity has a major fault.
- Gray octagon with TLC - Indicates cavity hardware mode (maintenance, maintenance done, or ready)
- Transparent decagon - Cavity hardware mode has been set to Offline
- Purple circle with TLC - The PV associated with the displayed fault code is invalid.
The different shapes used for these faults were chosen with the purpose of making the GUI red/green color blind accessible.
Figure 2: Various cavity statuses
Cavity Status Bars:
Below each cavity are two status bars. The left status bar indicates whether the SSA is On or Off, while the right status bar indicates whether the RF is On or Off. If the status bar is not present, then it means that status is off (i.e. no blue RF status bar? The RF is off for that cavity).
Figure 3: Example of a cavity with its SSA on and RF on
There is a known state where the RF status bar will appear pink/purple, but it's not disconnected in the way that we typically think about it. The RF status bar could could appear to be disconnected, but caget ACCL:L{LINAC}:{CM}{CAVITY}0:RFSTATE of the PV will yield "OFF" instead of the typical "Channel connect timed out" message. If you caget the PV again with .SEVR added to the end of it, it should yield a severity of INVALID which is what's causing the status bar to fault and indicate disconnected. This state occurs when the RF chassis is non-communicative or when the low level RF system otherwise can't trust the readout.
Figure 4: Disconnected RF status bar when the RF chassis is non-communicative. This can be proven by caget-ing the PV's severity.
Understanding Cavity Faults
Three Letter Code (TLC) Decoder:
As mentioned earlier, a faulted cavity will appear on the GUI as a red hexagon with a three letter code (TLC). To figure out what fault the TLC corresponds to, we can click on the "Three Letter Codes" button in the upper right corner of the Cavity Display GUI. This will launch a Cavity Fault Three Letter Codes panel with an alphabetized list of all the potential cavity faults as well as a short description of what each fault means.
Cavity Faults Panel:
Although cavities may have multiple faults at one time, the display can only show one fault at a time and it is designed to show the highest priority fault (aka the most severe/concerning fault) first. To see what other faults may be affecting a specific cavity, we can open a cavity fault panel for the cavity by clicking directly on the cavity widget. The resulting fault panel will display all the possible faults in order of decreasing priority. To see the PV associated with each fault, we can middle click the fault status box (which will display "OK", "FAULTED", or "INVALID").
Figure 5: Cavity fault display panel showing all faults for a given cavity
Next to each fault status is a button that will act as a direct link to the relevant EDM panel associated with a given fault. This feature was implemented to make troubleshooting specific faults easier. Most of the display links are functional, but additional effort is still being made to integrate the panels that take non standard handling (i.e. cryo, pydm, and 3.9GHz screens). The right most column shows the recommended corrective action to take in response to the current fault. These were written by the SRF group to aid operators in troubleshooting and responding to problems, but operators can call the on-call SRF person for additional assistance if needed.
GUI Backend
The Cavity Display GUI has a watcher status in the upper left hand corner to communicate whether the backend code is running. For a little background on how our watcher works: the backend code has an incrementing PV (PHYS:SYS0:1:SC_CAV_FAULT_HEARTBEAT) that will continuously update as long as the backend code is running. If this incrementing PV stops incrementing, then the watcher PV (ALRM:SYS0:SC_CAV_FAULT:ALHBERR) watching it will change its status from "RUNNING" to "NOT RUNNING."
If you notice the watcher status indicates "NOT RUNNING", feel free to restart the watcher by launching an SRF Home via Launchpad and clicking "Start Backend." After a minute or two, the backend should be working again. There is a known bug specifically with the cavity display GUI on the Large Monitor where the display occasionally freezes. If the GUI works locally on an OPI but is frozen only on the Large Monitor, the Large Monitor GUI might just need to resized ever so slightly to wake it up (this is a very weird and particular bug that occurs only on the Large Monitors).
Extra bonus information: Faults are read in from a .csv file linked below and is known as the Fault Spreadsheet. It contains information regarding all relevant fault pv names, fault level (cavity, cryomodule, rack, etc.), severity, fault conditions etc. necessary for the backend to run. If a brand new fault PV needs to be checked, the spreadsheet is where we'll add it.