Table of Contents

Starting the IOC Panels

Can also see these commands with "caget -tS RIX:VLS:CAM:01:LAUNCH_EDM" and "camViewer -c VLS -m".  "camViewer -l will give you a list of cameras"

• TMO: /cds/group/pcds/epics/ioc/common/andorCam/R0R1.61.01/children/build/iocBoot/ioc-tmo-andor1/edm-ioc-tmo-andor1.cmd
• RIX: /cds/group/pcds/epics/ioc/common/andorCam/R0R1.61.01/children/build/iocBoot/ioc-rix-dir-andor/edm-ioc-rix-dir-andor.cmd
                /cds/group/pcds/epics/ioc/common/andorCam/R0R1.61.01/children/build/iocBoot/ioc-rix-vls-andor/edm-ioc-rix-vls-andor.cmd
         /cds/group/pcds/epics/ioc/common/andorCam/R1.1.1/children/build/iocBoot/ioc-rix-norm-andor/edm-ioc-rix-norm-andor.cmd

NOTE: in the past we saw significant USB issues with the rix andor with serial number 22649.  On Oct. 27, 2022 this is the "dir" andor according to Dan.

NOTE: one can look at the EVR screen by clicking on the EVR button on the top right on main andor window

LCLS2 Timing Panel

Image Added

Mike Browne writes:  The "Rate Mode" can be Fixed, AC, or Seq.  Fixed gives you one choice of rates, AC another.  Seq gives you two fields to program... and Matt better knows what values to use here than I.  Destination for now should be "Don't Care".

Slow Full-Frame Tuning vs. 120Hz Full Vertical Binning with DAQ

Many experiments tune the position of the beam on the camera with a slow full-frame image then, after tuning is complete, run fast 120Hz data acquisition in "full vertical binning" (FVB) mode.  To do this, we recommend that users use the IOC's "python viewer" for the full-frame viewing/tuning (without running the DAQ) then switch to FVB for 120Hz DAQ operation.

...

• start the appropriate IOC panels (see below for command) on tmo-daq or rix-daq
• end the DAQ run if one is in progress
• click "Stop" in the IOC window to stop the Andor acquiring.  This is done in the "Image Collection" area in the middle-right of the main panel
• switch to full-frame mode by setting "Readout Mode" to "Image" in the middle of the "detail" sub panel
• click start in the IOC window to start the Andor in full-frame mode.  ensure "Cam Trig Event Code" (middle right of main panel) and "Exposure Time" (top right of main panel) are set appropriately (typically event code 120 when beam is going to the soft X-ray line).  Setting exposure time too long will interfere with 120Hz operation.
• open the "python viewer" at the top of the main panel
• tune beam position on camera
• when done tuning click "Stop" in the "Image Collection" area in the middle-right of the main panel
• in the middle of the "detail" subpanel set "Readout Mode" to "FVB" (full-vertical binning)
• click "Start" in the main panel middle-right to start the Andor acquiring in FVB mode
• Verify that "Capture Rate" in the "Image Collection" section of the main panel is what you expect
• start a DAQ run
• check for damage from the Andor in the DAQ-monitoring grafana window: psdaq#DAQMonitoring

Running in 90x90 "Crop" Mode

From Dan Damiani on 06/23/21.

NOTE: when changing running modes, always disable acquisition by hitting the "Stop" button in the "Image Collection" section of the main IOC window, then "Start" after all changes have been made.

The You can turn on crop from the expert screen setting "crop mode" to enable.  The underlying PV for turning this mode on/off is:  RIXRIX:DIR:CAM:01:AndorIsolatedCropMode

...

rixx43518 run 327 is a short run taken with the Andor running at 120Hz in 90x90 crop mode.  Here is an image from AMI:

Rate Measurements in Crop Mode

(from Dan Damiani)

FVB works at 120 Hz with both the 3Mhz and 1Mhz readout mode. This isn’t new. We have used the 1Mhz readout mode with full vertical binning in SXR in previous experiments, so it isn’t surprising that this still works.

...

Combining Crop Mode and Partial Vertical Binning

May 8, 2023.  Following a request from Kristjan Kunnus, Dan and Kristjan tried running the andor with a combination of crop mode and partial vertical binning with these settings.  It appeared to work and run at 10Hz, but it needs to be tested with x-rays on the detector to verify that it works.

Image Added

Miscellaneous Thoughts on running the Newton Camera

(courtesy of Dan Damiani)

...

A picture of the IOC screens when the camera is working in full-frame mode:

Single-Photon Mode Thoughts

From an email thread with RIX scientists on July 9, 2021:

...

We don't think the baselineClamp setting matters, but the IOC does have a different default setting of 0.

Power cycle procedure for camera/USB fiber extender

NOTE: This may not be necessary if one uses Dan's ~ddamiani/reset_andor.sh script described below.

Occasionally after power cycling the camera or everything if there was a power outage the USB fiber extender will come back in a bad state where it won't communicate with the camera. One time we saw these errors in /var/log/messages when the VLS camera broke:

May 27 14:03:38 daq-rix-andor-01 kernel: usb 1-2.4.1: usbfs: interface 0 claimed by usbfs while 'andor' sets config #1
May 27 14:03:39 daq-rix-andor-01 kernel: usb 1-2.4.1: usbfs: interface 0 claimed by usbfs while 'andor' sets config #1

This seems to happen to the direct beam path andor an order of magnitude more often than the one on the VLS. Too recover from this when it happens do the following procedure:

1. Power off the camera (NOTE: to do this I believe you have unplug the power supply, since the power supply just has a rocker switch which I think selects between standard-cooling/deep-cooling?).  The only LED that gives any indication of power is a green one inside the power supply, visible only if you put your eye somewhere near the deep-cooling ("||") side of the rocker switch at a specific angle.
2. Power off the camera side of the extender
3. Unplug the server-side extender USB cable from daq-rix-andor-01. The DIR andor extender has the black USB cable (top USB port) and the VLS andor extender is the silver cable (bottom USB port).
4. Re-plug the server-side extender USB cable into daq-rix-andor-01.
5. Power on the camera side of the extender. You should see the link led turn on on the extender, and the host led blink slowly.
6. Power on the camera. The host led on the camera side extender show turn on solid now, and the activity led should blink intermittently.
7. Restart the IOC (the existing IOC process my be locked up and may need to be kill -9 to die) or alternatively just reboot daq-rix-andor-01.

Remote Power Cycling of Cameras

As of Nov. 17, 2023 rix now has a control panel to remotely power cycle andor cameras:

Image Added

Debugging

You can see log files for the IOC here:  /cds/data/iocData/ioc-rix-dir-andor/iocInfo/.

For camera communication errors ("Unknown" in EDM "Model Number" field) can look for "USB disconnect" error messages in /var/log/messages.  Can see device numbers with "lsusb | grep Andor" and compare to device numbers in /var/log/messages.  Here is an example with two cameras hooked up to one node:

daq-rix-andor-01:~$ lsusb | grep Andor
Bus 001 Device 071: ID 136e:0005 Andor Technology Ltd. 
Bus 001 Device 063: ID 136e:0005 Andor Technology Ltd. 
daq-rix-andor-01:~$ 

Other debugging techniques:

• telnet to the ioc and restart
  • when restarting, ioc connects to all the cameras to see if it's the right one
• Sometimes have to reset all usb hubs/bridges like this:

bash-4.2# ~ddamiani/reset_andor.sh 3-2
bash-4.2# ~ddamiani/reset_andor.sh 3-2.4
bash-4.2# ~ddamiani/reset_andor.sh 3-2.4.4
bash-4.2# ~ddamiani/reset_andor.sh 3-2.4.4.1
bash-4.2# 

Workaround for RIX ANDOR-DIR hang

NOTE: We have seen cases where this does not work and we are forced to power cycle daq-rix-andor-01 to recover.

This is for when the camera is visible with "lsusb | grep Andor" (note uppercase "A") but the camera isn't communicating:

• Execute the following command on daq-rix-andor-01 as superuser: ~ddamiani/reset_andor.sh 1-2
• Restart one IOC (e.g. DIR), wait for it to finish
• Restart the second IOC, wait for it to finish

At the moment there is no way to know which camera is which, except for unplugging them one at a time.

The argument to the script ("1-2") can change, in principle but should do so rarely.  The “1-2” is coming from the USB bus and port enumerations from the operating system:
 
ddamiani@daq-rix-andor-01:~$ egrep 136e /sys/bus/usb/devices/*/idVendor
/sys/bus/usb/devices/1-2.3.1/idVendor:136e
/sys/bus/usb/devices/1-2.4.1/idVendor:136e
 
The first part of the for X-Y.blah bit which is 1-2 currently.

[root@daq-rix-andor-01 cpo]# more ~ddamiani/reset_andor.sh
#!/bin/bash

if [ $# -ne 1 ]; then
    echo "Usage $0 <device>"
    exit 1
fi

echo "${1}" > /sys/bus/usb/drivers/usb/unbind
echo "${1}" > /sys/bus/usb/drivers/usb/bind
[root@daq-rix-andor-01 cpo]# 

EVR Issues

On two occasions we have seen the EVR go into a bad state (click on the "EVR" button in upper right of the andor ioc window) cause some camera to not trigger at all and one camera to glitch between a "locked" and "unlocked" state.  The fiducial-rate turns red and fluctuates away from 360Hz:

Image Added

We tried recovering from this by rebooting the IOC and it worked for some minutes, but in the end in both cases we had to power cycle the node.

USB Extender

http://www.icron.com/products/icron-brand/usb-extenders/fiber/usb-2-0-ranger-2324/

Icron Ranger 2324 USB 2.0 extender.  Correct operation shows blue power LED, solid green "link" LED, solid green "host" LED, blinking yellow "activity" LED.  Manual is here:

Changing Cameras

When there are multiple cameras connected to a single node the IOC uses the camera serial number to decide which one to talk to, so the serial number must be changed in the IOC startup script when a camera is changed. The serial number is set in lines like this:

ddamiani@daq-rix-andor-01:/cds/home/c/cpo$ grep CAM_ID /cds/group/pcds/epics/ioc/common/andorCam/R0.6.0/children/build/iocBoot/ioc-rix-dir-andor/st.cmd
epicsEnvSet( "CAM_ID",       "26601" )

The serial number for a camera can be determined like this (I think this program connects to cameras that are not already connected to by other software like IOCs):

daq-rix-andor-01:~$ /reg/g/pcds/dist/pds/current/build/pdsapp/bin/x86_64-rhel7-opt/andorStandAlone -m
SDKVersion: 2.100.30026.0
Found 2 Andor Cameras
Waiting for hardware to finish initializing...
DeviceDriverVersion: 0.0.0.0
Software Version: eprom 0 coffile 0 vxdrev 0 vxdver 0 dllrev 100 dllver 2
Hardware Version: PCB 0 Decode 0 FirewareVer 20 FirewareBuild 24
Camera serial number: 26601
Camera Head Model: DU940P_BN
etc.