If something is not working, try troubleshooting steps here first, then call Divya at (929) 247-0432 if urgent, or make a JIRA ticket if not urgent. If Divya doesn't pick up, follow the LCLS Instrument Support Escalation Tree (ECS & DS).pdf
Confluence
Control Room Machines
- Types:
- cxi-daq connected to primary daq
- cxi-monitor: connected to secondary daq
- cxi-console and cxi-control for general use
- If screens are out of order, open a terminal and run ‘nvidia.sh'
- Workaround for when encountering F-12 and F-11(enables beam stop in XRT) Keyboard shortcuts not working :
- Restart the gnome shell by pressing, ALT + F2 , typing "r" in the Enter command window.
- Reapply the gnome settings by running "gnome-settings.sh"
- This seems to be an issue with the red hat and has been observed in other hutches as well. SLAC IT has an open ticket with red hat for the same. Moving this to resolved for now CXI Specific issue.
EDM Screens
To open CXI Home Screen: Type "cxihome" in terminal.
- Green: Nominal and working
- White: IOC is intentionally powered off or problematic , ioc not visible, check in iocmanager to see if the IOC is off.
- Pink/Purple: Communication is not achieved. Verify digi/moxa cable is correct and connected, click on reinitialize, if that doesn't work try rebooting the IOC
Applications
Viewing Cameras: camViewer
Camera settings :
Check gain, exposure time, acquisition time, trigger mode, and event code.
Some issues can be fixed by clicking the Stop then Start buttons.
More information on CXI specific Cameras can be found here: CXI Cameras
IOC Manager (command : iocmanager / iocmanager cxi)
Main Use is to turn on/off an ioc
Description of fields:
- Current IOC: IOC ID of the currently selected IOC
- Heartbeat: Heartbeat of the currently selected IOC, if its incrementing, then it means its alive.
- Location: Location of the server it is running on. Image shows B999 CXI H5 R51 E33, which means FEH CXI Hutch 5, Rack 51 Elevation 33
- Boot Time: Date and Time when the IOC was last rebooted
- Description: Description of the server it is running on
- Columns ( Refer the image above)
- Alias: Nicknames for ioc
- IOC ID : Actual names of1446px the ioc
- State: Whether it is on/off (Dev and Prod mean on)
- Status: solid status, RUNNING means the IOC is on.
- Host: server the ioc is running on
- OS: to denote if it is a rhel5 or rhel7 IOC.
- Port: port on the server the ioc is accessible at (must be unique from other iocs on the same server)
- IOC Control drop down
- Soft IOC reboot
- Choose this first!
- Writes to a PV, telling it to restart
- Hard IOC reboot
- Second option
- Tells server to kill the ioc and restart it
- Reboot Server
- Last resort since it will restart all iocs running on that server
- At this point, call/contact Divya, before continuing
- Takes more time and could kill other IOCs that should not be killed
- How to find the IOC ID (if you only know the PV name)
- Enter the PV name in the "Find PV" bar at the bottom left. For example: CXI:SC2:MMS:01 and hit enter. It will open a window with the IOC ID and also take the cursor to the IOC ID it belongs.
More detailed guide on IOC manager can be found here : IOC Manager for Users
striptool
striptool is a tool that can be used for plotting PVs quickly and simulateously for online monitoring purposes.
Usage: Type "striptool" in the terminal window. It will open up a pop up window. Type/Drag the PV name in "Plot New Signal" bar, hit enter or press "Connect".
This will open up a new window called "StripTool Graph Window". Here you can see the signal being plotted in real time from when you pressed enter.
Motor Expert Screen
- Example: PIM /YAG 1
- FIND on CXI home screen in array of green buttons or
- Command:
motor-expert-screen CXI:DG1:MMS:08
- If motor position doesn’t seem right, try to re-initialize first.
- Set parameters(dial vs user):
- Offset: set dial to be zero , maybe at limit switch, and user set at interaction point
- Neg/Postive: since in y-direction, going positive actually is negative, so it is switched with these buttoms
- Do not mess with, UNLESS comfortable: run current/percentage
- Can move with velocity if experiment requires it
- Velocity vs speed:
- V: actual speed in ml/s
- Speed: rev/s
- Acceleration: requires calculations
- If something is not moving: CHECK the comments given in the ‘log messages’
- Power cycled error
- Button will show to power-up
- If not, go into Aux setting: Command response, type ‘pu 0’
- Error 92:
- Closed loop postioning error
- Check the size of deadband
- Go into aux settings, Command response,type ‘pr db’ or if specific position, ‘pr db 10’
Shell Commands
Run command -h for helptext
serverStat
This command line tool can be used to check the status of servers and reboot or power cycle them. This tool uses the ipmi interface to check if the server is alive and to perform the power cycle.
If the ipmi interface corresponding to the server is unavailable for some reason (you can see that with the status command), physically power cycling the server maybe necessary in some cases. (Contact nearest available pcds person for doing this)
Example: serverStat daq-cxi-master cycle
netconfig
This tool is capable of doing a multitude of actions, but users should only be primarily using "netconfig search <hostname>". Rest all the actions should only be left to pcds engineers.
- This command line tool can be used to find information about the server. Location could be useful if you need to power cycle it in person. There are additional useful information that can accessed with this such as subnet, IP address, Description of the server etc. See figure below, in this figure the location of the ioc-cxi-rec04 server can be found in the Location field : B999 H5(CXI) R52A E30 E31, which means the server is located in rack 52A, occupying elevation 30 and 31.
motor-expert-screen
- Open motor expert screen for a certain motor given its PV. Useful if you can't find it on the EDM screen.
- e.g.
motor-expert-screen CXI:PI1:MMS:01
Hutch Webcams
Hutch webcams can be accessed from CXI Home screen, from "Web Cameras" on the left hand side column or from bottom-embedded-screens tab. Detectors → Web Cameras (opens drop down with all cameras)
Grafana
- In firefox>pswww website>log in> controls Grafana> Dashboards, manage> CXI
- In Grafana> create>empty panel
- Used to check any PVs of interest
- Can look into their history
- Change timeframe to whatever is of interest
- To copy and paste: use middle click
- Can also click on stream: for current state, without the need to refresh
Jungfrau
- Controls found on CXI home > Detectors tab
- Documentation and Troubleshooting:
- takepeds & makepeds:
- Configure DAQ
- takepeds
- makepeds -u <username> -r <run #> -q ffbh3q
- (If missing geometry) makepeds -u <username> -r <run #> -g cxic00120
DAQ
- Use 'restartdaq' to open the daq (there's no 'startdaq' command)
- Event Codes / Timing
- Change on the DAQ Control Window: Edit>EVR>Event Codes
- Set one code to 'Readout' to define when the DAQ records data
- Set codes to 'Command' to have their states for each shot recorded in the data
- Event codes:
- To record only when the sequencer fires, set the DAQ to readout on 187 and put 187 in the sequencer
AMI
- Will automatically open with the DAQ
- If you need to restart it or open a second viewer, use 'startami'
Hutch Python
To start hutch python in CXI, type this in terminal : "cxi3". Information on hutch python seminar and documentation can be found in:
Hutch Python Seminar