Overview
Camera Watchdog monitors the statuses of the FACET Profile Monitors, SIOCs, and POE Hubs, and attempts to revive cameras that become disconnected.
Bugs/Issues
Sometimes after an SIOC goes down, even if it comes back, the cameras on that SIOC get stuck on State 5 (purple lamps). Jiggling or rebooting cameras doesn't seem to help. Restarting the Watchdog usually fixes this issue.
Key Features
- The Camera Watchdog creates Camera objects in MATLAB for each FACET Profile Monitor. Each Camera object stores data such as the Camera's name, location, and the number of times a reboot has been attempted.
- The Camera Watchdog also creates SIOC objects and POEHub objects for the relevant SIOCs and POE Hubs.
- Each Camera assigns itself a State based on its Connection status, Acquisition status (Acquiring or Idle), Trigger Mode (Sync in 1 or Free Run), and Array Rate (zero or nonzero). The State is updated any time one of these PVs changes.
- If the Camera is in a disconnected State, it attempts to revive itself and the Reboot Count goes up by 1.
- Once a day, the Camera Watchdog creates a report and saves it to a MAT file.
States
The possible States for the Cameras are:
0 - The Camera is connected, acquiring, the array rate is nonzero, and the trigger mode is set to 'Sync in 1'.
1 - The Camera is disconnected.
2 - The Camera is connected and acquiring, but the array rate is zero.
3 - The Camera is connected, but it is idle.
4 - The Camera is connected, acquiring, and the array rate is nonzero, but the trigger mode is set to 'Free Run.'
5 - The Camera has an Alarm or its POE Hub is down.
The Camera object is periodically checking for updates from EPICS. If one of the relevant PVs changes, it updates the State. All State updates are recorded in a log file.
Rebooting
If a Camera is in State 1, it will attempt to reboot itself by calling its PowerCycle PV. It will only attempt to revive itself once; if the camera does not come back, it will not keep trying to reboot it. However, if the camera is reconnected, and then later gets disconnected again, it will attempt to revive it then.
If revived, the Camera Watchdog will also jiggle the camera.
Each time a reboot is attempted, the Reboot Count for the camera goes up by 1. The Reboot Count of any camera can be called with "caget" (see below), but it is also stored in the Archiver:
[fphysics@facet-srv01 ~/shperez ]$ caget CAMR:LI20:100:REBOOTCOUNT |
Daily Report
Once a day at midnight, the Camera Watchdog creates a report of camera data for all of the FACET profile monitors. It captures the following data in a table and saves it to a MAT file:
Name, Model, PV, X Orientation, Y Orientation, Resolution, Exposure, Connection, Serial Number, and Number of Reboots
The daily reports and log files can be found here:
|
PyDM GUI
State indicators were added to the PyDM GUI in FACET Home to visually display the state of each profile monitor.
The colors for the indicators are:
State 0 - Green
State 1 - Grey
States 2, 3, or 4 - Yellow
State 5 - Purple
An additional column of indicators was also added to display the state of the POE for each camera.
The colors for these indicators are:
Green - Power on
Grey - Power off
Purple - Alarm status, POE is down
