Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Call Taking Process

  • Confirm with ACR that they contacted Bill Schlotter before reaching out on the FEE support line.
  • Write down relevant details of Call; caller name, basic problem
  • Attempt to troubleshoot problem using resources listed below (optional)
  • Escalate problem to Subject Matter Expert (SME) 
  • Document the problem in the Operations Call Log


Troubleshooting Process

It is rare that problems in the FEE ever repeat themselves, so this guide will possibly be of minimum use. However, it is still useful to know general principles with respect to troubleshooting problems:

  1. Understand the problem (What are they trying to do that isn't working? Is the logic for something not working as expected? Is something missing? What have the scientists already tried?)
  2. Determine the general scope of the problem... Is this problem with a PV? Is this a problem with hutch python? Is this a problem with the DAQ?
    1. Is this a problem with a screen?
      1. Screens use PVs at their very foundation, if a screen is not showing a PV properly, it might be worth checking that the PV is connected
      2. Is the PV "caget-able" (check the IOC)? Does the PV show from different subnets (if not, this points to a gateway issue)?
    2. Is this a problem with hutch-python?
      1. If it is an issue with hutch-python, the root of the problem could be related to the experiment / hutch scripts. 
      2. This would likely require intervention from the POC (if they haven't already been called)
    3. Is this a problem with motion?
      1. Motion systems in the FEE are controlled (mostly) via PLC. This could be related to a number of problems, but the most easily checked would be the health of the PLC IOC. If the PLC IOC is generating PVs normally, the problem could be more fundamental to the system.
      2. If you are properly trained in PLCs, it would be okay to check on the PLC side, and see if there are fatal errors on the PLC, and other things. This would certainly point to involvement on the SME's part. Even if you can pinpoint the problem, any changes to this system should be done by an SME
    4. Is this a problem with vacuum?
      1. As with Motion, vacuum systems are also controlled via PLC in the FEE. It could be useful to look into the PLC and see if the logic of interlocks are working appropriately. Even if you can pinpoint the problem, any changes to this system should be done by an SME


FEE Support Line Scope of Work

Things we are responsible for

Things we aren't responsible for

General References

Controls User & Troubleshooting Guide

...

SLAC Speak, common acronyms

GIS Mapping Site

Troubleshooting

General Troubleshooting Resources

Controls User and Troubleshooting Guide

...

Convention Constituent Component Naming Convention

Area Specific Troubleshooting

FEE

FEE Operation and Troubleshooting

FEE Systems Link Cheat Sheet

FEE Gateway Issues

Device/Topic Specific Troubleshooting

PLCs

PLC Help

Switches and Escrow

Switch Tool Documentations

Escrow Access

Hutch Python/Happi

github documentation

list of happi devices

EPICS and IOC/Manager

IOC Manager Staff guide

Novices Guide to accessing epics

Using PyCA (caget, caput, etc.)

In-depth Epics Resources

IPIMB/Wave8

Troubleshooting for Controls IPIMB and Wave8s

Cameras

Fee Imager Camera Debugging

MPS

MPS Troubleshooting and resources

PMPS

FEE PMPS Troubleshooting Information

...

  • look under pulldown menu of this link for extensive PMPS reference

Attenuators

AT1K0

Mirrors/Optics

HOMS Troubleshooting

X/GMD

GMD and XGMD Troubleshooting

GEM

GEMs

Spectrometers

HXRSSS

DAQ


LCLS-1 DAQ TIer-1 

AMI (LCLS1 and LCLS2)

...