...
- Know how to launch the GUI
- Understand the information presented on the GUI
- Know how to interpret faults and see the status of all potential faults associated with a cavity
- Understand how to restart backend if the display stops updating
Introduction:
Figure 1: Overview of Cavity Display GUI
...
Table of Contents |
---|
How to Launch the GUI:
To The easiest way to launch the GUI is by typing "SC Cavity Summary Display locally at an OPI, follow the instructions below" into Launchpad.
Alternatively, you can manually launch the GUI from SRF Home:
- Open a terminal
- Type in set_profile srf
- Type srfhome. This will launch a PyDM version of SRF Home
- Click "Cavity Summary GUI." This is a PyDM GUI and typically takes 2-3 minutes to launch, so be patient!
Alternatively, you can use Launchpad to open the GUI by typing in "SC Cavity Summary Display."
Interpreting Information on the GUI:
...
- Green diamond with cavity number - This indicates that the cavity is healthy and has no faults.
- Yellow triangle with TLC - Cavity has a minor fault.
- Red hexagon with TLC - Cavity has a major fault.
- Gray octagon with TLC - Indicates cavity hardware mode (maintenance, maintenance done, or ready)
- Transparent octagon with 'OFF' TLC decagon - Cavity hardware mode has been set to Offline
- Purple circle with TLC - The fault PV associated with the cavity displayed fault code is disconnectedinvalid.
The different shapes used for these faults were chosen with the purpose of making the GUI red/green color blind accessible.
Figure 2: Various cavity statuses
...
Below each cavity are two status bars. The left status bar indicates whether the SSA is On or Off, while the right status bar indicates whether the RF is On or Off. If the status bar is not present, then it means that status is off (i.e. no blue RF status bar? The RF is off for that cavity).
Figure 3: Example of a cavity with its SSA on and RF on
There is a known state where the RF status bar will appear pink/purple, but it's not disconnected in the way that we typically think about it. The RF status bar could could appear to be disconnected, but caget ACCL:L{LINAC}:{CM}{CAVITY}0:RFSTATE of the PV will yield "OFF" instead of the typical "Channel connect timed out" message. If you caget the PV again with .SEVR added to the end of it, it should yield a severity of INVALID which is what's causing the status bar to fault and indicate disconnected. This state occurs when the RF chassis is non-communicative or when the low level RF system otherwise can't trust the readout.
Figure 4: Disconnected RF status bar when the RF chassis is non-communicative. This can be proven by caget-ing the PV's severity.
...
Although cavities may have multiple faults at one time, the display can only show one fault at a time and it is designed to show the highest priority fault (aka the most severe/concerning fault) first. To see what other faults may be affecting a specific cavity, we can open a cavity fault panel for the cavity by clicking directly on the cavity widget. The resulting fault panel will display all the possible faults in order of decreasing priority. To see the PV associated with each fault, we can middle click the fault status box (which will display "OK", "FAULTED", or "INVALID").
Figure 5: Cavity fault display panel showing all faults for a given cavity
Next to each fault status is a button that will act as a direct link to the relevant EDM panel associated with a given fault. This feature was implemented to make troubleshooting specific faults easier. Most of the display links are functional, but additional effort is still being made to integrate the panels that take non standard handling (i.e. cryo, pydm, and 3.9GHz screens). The right most column shows the recommended corrective action to take in response to the current fault. These were written by the SRF group to aid operators in troubleshooting and responding to problems, but operators can call the on-call SRF person for additional assistance if needed.
GUI Backend
The Cavity Display GUI has a watcher status in the upper left hand corner to communicate whether the backend code is running. For a little background on how our watcher works: the backend code has an incrementing PV (PHYS:SYS0:1:SC_CAV_FAULT_HEARTBEAT) that will continuously update as long as the backend code is running. If this incrementing PV stops incrementing, then the watcher PV (ALRM:SYS0:SC_CAV_FAULT:ALHBERR) watching it will change its status from "RUNNING" to "NOT RUNNING."
If you notice the watcher status indicates "NOT RUNNING", feel free to restart the watcher by launching an SRF Home via Launchpad , and then clicking "Stop Backend" followed by "Start Backend." After a minute or two, the backend should be working again. There is a known bug specifically with the cavity display GUI on the Large Monitor where the display occasionally freezes. If the GUI works locally on an OPI but is frozen only on the Large Monitor, the Large Monitor GUI might just need to resized ever so slightly to wake it up (this is a very weird and particular bug that occurs only on the Large Monitors).
Extra bonus information: Faults are read in from a .csv file linked below and is known as the Fault Spreadsheet. It contains information regarding all relevant fault pv names, fault level (cavity, cryomodule, rack, etc.), severity, fault conditions etc. necessary for the backend to run. If a brand new fault PV needs to be checked, the spreadsheet is where we'll add it.