Call Taking Process
- Confirm with ACR that they contacted Bill Schlotter before reaching out on the FEE support line.
- Write down relevant details of Call; caller name, basic problem
- Attempt to troubleshoot problem using resources listed below (optional)
- Escalate problem to Subject Matter Expert (SME)
- Document the problem in the Operations Call Log
Troubleshooting Process
It is rare that problems in the FEE ever repeat themselves, so this guide will possibly be of minimum use. However, it is still useful to know general principles with respect to troubleshooting problems:
- Understand the problem (What are they trying to do that isn't working? Is the logic for something not working as expected? Is something missing? What have the scientists already tried?)
- Determine the general scope of the problem... Is this problem with a PV? Is this a problem with hutch python? Is this a problem with the DAQ?
- Is this a problem with a screen?
- Screens use PVs at their very foundation, if a screen is not showing a PV properly, it might be worth checking that the PV is connected
- Is the PV "caget-able" (check the IOC)? Does the PV show from different subnets (if not, this points to a gateway issue)?
- Is this a problem with hutch-python?
- If it is an issue with hutch-python, the root of the problem could be related to the experiment / hutch scripts.
- This would likely require intervention from the POC (if they haven't already been called)
- Is this a problem with motion?
- Motion systems in the FEE are controlled (mostly) via PLC. This could be related to a number of problems, but the most easily checked would be the health of the PLC IOC. If the PLC IOC is generating PVs normally, the problem could be more fundamental to the system.
- If you are properly trained in PLCs, it would be okay to check on the PLC side, and see if there are fatal errors on the PLC, and other things. This would certainly point to involvement on the SME's part. Even if you can pinpoint the problem, any changes to this system should be done by an SME
- Is this a problem with vacuum?
- As with Motion, vacuum systems are also controlled via PLC in the FEE. It could be useful to look into the PLC and see if the logic of interlocks are working appropriately. Even if you can pinpoint the problem, any changes to this system should be done by an SME
FEE Support Line Scope of Work
Things we are responsible for
Things we aren't responsible for
General References
Controls User & Troubleshooting Guide
...
SLAC Speak, common acronyms
GIS Mapping Site
Troubleshooting
General Troubleshooting Resources
Controls User and Troubleshooting Guide
...
Convention Constituent Component Naming Convention
Area Specific Troubleshooting
FEE
FEE Operation and Troubleshooting
FEE Systems Link Cheat Sheet
FEE Gateway Issues
Device/Topic Specific Troubleshooting
PLCs
PLC Help
Switches and Escrow
Switch Tool Documentations
Escrow Access
Hutch Python/Happi
github documentation
list of happi devices
EPICS and IOC/Manager
IOC Manager Staff guide
Novices Guide to accessing epics
Using PyCA (caget, caput, etc.)
In-depth Epics Resources
IPIMB/Wave8
Troubleshooting for Controls IPIMB and Wave8s
Cameras
Fee Imager Camera Debugging
MPS
MPS Troubleshooting and resources
PMPS
FEE PMPS Troubleshooting Information
...
- look under pulldown menu of this link for extensive PMPS reference
Attenuators
AT1K0
Mirrors/Optics
HOMS Troubleshooting
X/GMD
GMD and XGMD Troubleshooting
GEM
GEMs
Spectrometers
HXRSSS
DAQ
LCLS-1 DAQ TIer-1
AMI (LCLS1 and LCLS2)
...