Starting the IOC Panels

Can also see these commands with "caget -tS RIX:VLS:CAM:01:LAUNCH_EDM" and "camViewer -c VLS -m".  "camViewer -l will give you a list of cameras"

  • TMO: /cds/group/pcds/epics/ioc/common/andorCam/R1.1.1/children/build/iocBoot/ioc-tmo-andor1/edm-ioc-tmo-andor1.cmd
  • RIX: /cds/group/pcds/epics/ioc/common/andorCam/R1.1.1/children/build/iocBoot/ioc-rix-dir-andor/edm-ioc-rix-dir-andor.cmd
           /cds/group/pcds/epics/ioc/common/andorCam/R1.1.1/children/build/iocBoot/ioc-rix-vls-andor/edm-ioc-rix-vls-andor.cmd
           /cds/group/pcds/epics/ioc/common/andorCam/R1.1.1/children/build/iocBoot/ioc-rix-norm-andor/edm-ioc-rix-norm-andor.cmd

NOTE: in the past we saw significant USB issues with the rix andor with serial number 22649.  On Oct. 27, 2022 this is the "dir" andor according to Dan.

NOTE: one can look at the EVR screen by clicking on the EVR button on the top right on main andor window

LCLS2 Timing Panel

Mike Browne writes:  The "Rate Mode" can be Fixed, AC, or Seq.  Fixed gives you one choice of rates, AC another.  Seq gives you two fields to program... and Matt better knows what values to use here than I.  Destination for now should be "Don't Care".

Slow Full-Frame Tuning vs. 120Hz Full Vertical Binning with DAQ

Many experiments tune the position of the beam on the camera with a slow full-frame image then, after tuning is complete, run fast 120Hz data acquisition in "full vertical binning" (FVB) mode.  To do this, we recommend that users use the IOC's "python viewer" for the full-frame viewing/tuning (without running the DAQ) then switch to FVB for 120Hz DAQ operation.

The steps below refer to two IOC panels.  Here are two screen shots of a working Andor.  The first image is the main panel, and the second image is a "detail" subpanel.  It is accessed from the lower box labelled "NewtonD0940P" on the middle-left of the main panel:

  

Slow Full-Frame tuning vs. 120Hz Full Vertical Binning procedure (IMPORTANT: do not switch between full-frame/FVB modes without stopping the camera from acquiring, as detailed in these steps. that can cause the camera software to hang):

  • start the appropriate IOC panels (see below for command) on tmo-daq or rix-daq
  • end the DAQ run if one is in progress
  • click "Stop" in the IOC window to stop the Andor acquiring.  This is done in the "Image Collection" area in the middle-right of the main panel
  • switch to full-frame mode by setting "Readout Mode" to "Image" in the middle of the "detail" sub panel
  • click start in the IOC window to start the Andor in full-frame mode.  ensure "Cam Trig Event Code" (middle right of main panel) and "Exposure Time" (top right of main panel) are set appropriately (typically event code 120 when beam is going to the soft X-ray line).  Setting exposure time too long will interfere with 120Hz operation.
  • open the "python viewer" at the top of the main panel
  • tune beam position on camera
  • when done tuning click "Stop" in the "Image Collection" area in the middle-right of the main panel
  • in the middle of the "detail" subpanel set "Readout Mode" to "FVB" (full-vertical binning)
  • click "Start" in the main panel middle-right to start the Andor acquiring in FVB mode
  • Verify that "Capture Rate" in the "Image Collection" section of the main panel is what you expect
  • start a DAQ run
  • check for damage from the Andor in the DAQ-monitoring grafana window: psdaq#DAQMonitoring

Running in 90x90 "Crop" Mode

From Dan Damiani on 06/23/21.

NOTE: when changing running modes, always disable acquisition by hitting the "Stop" button in the "Image Collection" section of the main IOC window, then "Start" after all changes have been made.

You can turn on crop from the expert screen setting "crop mode" to enable.  The underlying PV for turning this mode on/off is: RIX:DIR:CAM:01:AndorIsolatedCropMode

  • Click "Stop" in the "Image Collection" section of the main IOC window
  • Make sure the "Readout Mode" (middle of the IOC sub panel) in the "Collect" section is set to "Image" and NOT "FVB" (full vertical binning)
  • caput RIX:DIR:CAM:01:AndorIsolatedCropMode Enable
  • Set the roi to 90x90 on the main screen in the "Acquisition Settings" section (other settings may work at lower rates. need to ask Dan).
    • NOTE: I don't understand why, but I appear to have to enter the "Y" region size before I enter the "X" region size.
  • Set the "Cam Trig Event Code" to the appropriate value in the "Timestamps" section of the main IOC window
  • Click "Start" in the "image Collection" section of the main IOC window
  • Verify that "Capture Rate" in the "Image Collection" section of the main IOC panel is what you expect

Reverse the above steps to go back to normal Andor running mode.

A screenshot of the camera running at 120Hz in this mode is here:

rixx43518 run 327 is a short run taken with the Andor running at 120Hz in 90x90 crop mode.  Here is an image from AMI:

Rate Measurements in Crop Mode

(from Dan Damiani)

FVB works at 120 Hz with both the 3Mhz and 1Mhz readout mode. This isn’t new. We have used the 1Mhz readout mode with full vertical binning in SXR in previous experiments, so it isn’t surprising that this still works.

For crop mode it is slightly more complicated.

Note there is a PV you can use to check if the settings you have chosen will work for a particular trigger rate:

RIX:DIR:CAM:01:AcquirePeriod_RBV

This is just a time in seconds. So for example if set the camera to 1Mhz readout, crop mode, and 50x50 roi I get this:

caget RIX:DIR:CAM:01:AcquirePeriod_RBV

RIX:DIR:CAM:01:AcquirePeriod_RBV 0.00831

That number needs to be less than 1/trigger rate in order for things to work, and this is slightly less than 1/120.

Working from this you get (rounding to the nearest multiple of 5 works):

Trigger Rate

3MHz

1MHz

120 Hz

90x90

50x50

60 Hz

155x155

85x85

30 Hz

255x255

140x140

10 Hz

470x470

275x275

Combining Crop Mode and Partial Vertical Binning

May 8, 2023.  Following a request from Kristjan Kunnus, Dan and Kristjan tried running the andor with a combination of crop mode and partial vertical binning with these settings.  It appeared to work and run at 10Hz, but it needs to be tested with x-rays on the detector to verify that it works.

Miscellaneous Thoughts on running the Newton Camera

(courtesy of Dan Damiani)

  • Camera "Software Development Kit" (SDK) manual can be found here: Andor CCD
  • Do not set the exposure time to 5 ms: 2ms is the max exposure time for 120 hz, it takes 5+ ms for the readout so you are dropping every other event (with autocorrect at 120Hz).  A too-large setting will be seen in the main IOC panel with the "sync status" oscillating between locked/unlocked.
  • Also they have the evr delay set to zero, but with a large enough window that might be okay
    • Looks like the last time they ran the VLS at SXR the evr delay was 0.00085 s and the width was 1e-5 seconds
  • The IOC shutter settings do not matter: they are only to control an external shutter (which we don't have in TMO)
  • you can also try to experiment with toggling TMO:VLS:CAM:01:AndorKeepClean to see what effect that has (use Stop-Acquire before changing this setting).  use the pvget/pvput commands to modify the value.
  • use the "python viewer" at the top of the IOC screen to view images.  the edm viewer is broken.
  • we recommend tuning the full-frame image without running the DAQ using the python-viewer, then putting the camera into Full Vertical Binning (FVB) mode using the IOC window for 120Hz line-out running
  • the event code set in the IOC window should match the event code used in the DAQ readout group (in the "timing" system configuration).
  • ioc "autosave" files can be found here: /cds/data/iocData/ioc-tmo-andor1/autosave/. (you can look at the various autoSettings.sav* files)

  • As far as I can tell there are only a few PVs that actually matter for the temp:
    TMO:VLS:CAM:01:Temperature_RBV # readback of the actual setpoint temp. I am not sure this one works
    TMO:VLS:CAM:01:Temperature # get/put the temperature setpoint (edited) 

    # turn on/off the cooling
    TMO:VLS:CAM:01:AndorCooler
    # readback the on/off state
    TMO:VLS:CAM:01:AndorCooler_RBV
    # string "status" message about the cooling state
    TMO:VLS:CAM:01:AndorTempStatus_RBV
  • TMO:VLS:CAM:01:TemperatureActual is supposedly the temperature readback

A picture of the IOC screens when the camera is working in full-frame mode:

Single-Photon Mode Thoughts

From an email thread with RIX scientists on July 9, 2021:

Executive summary: now that you’ve set the preamp gain to 4 (we think you did that today, following Matt’s advice) we think you’re running with the same settings we used for sxrx30416 run 83, as best we can tell.  Note that the manual describes an EM (electron-multiplying) gain setting that is useful for single photon counting but we believe the LCLS cameras are Newton CCDs, not Newton EMCCDs.  See manual https://andor.oxinst.com/downloads/view/newton-ccd-emccd-hardware-guide
Details: Dan and I looked at the settings from sxrx30416 run 83 vs. the current running.  These are the settings we had:

In [2]: andorconfig.highCapacity()
Out[2]: 0

This is not settable in the IOC, but Dan looked and it defaults to 0 now, which we think is right (highCapacity=1 corresponds to large dynamic range, which you don’t want for single-photon regime)

In [3]: andorconfig.readoutSpeedIndex()
Out[3]: 0

This corresponds to 3MHz which is what you are running now, and it is correct: this increases the readout speed so we can run at 120Hz, but also increases noise.

In [4]: andorconfig.gainIndex()
Out[4]: 2

This “gain enum index” corresponds to the factor 4 you have just set (today) for the highest preamp gain, as Matt Seaberg suggested.

Other LCLS1 settings:

In [5]: andorconfig.exposureTime()
Out[5]: 0.0010000000474974513

In [6]: andorconfig.cropMode()
Out[6]: psana.Andor.EnumCropMode.ENUM_CROP_ON

In [7]: andorconfig.baselineClamp()
Out[7]: 1

We don't think the baselineClamp setting matters, but the IOC does have a different default setting of 0.

Power cycle procedure for camera/USB fiber extender

NOTE: This may not be necessary if one uses Dan's ~ddamiani/reset_andor.sh script described below.

Occasionally after power cycling the camera or everything if there was a power outage the USB fiber extender will come back in a bad state where it won't communicate with the camera. One time we saw these errors in /var/log/messages when the VLS camera broke:

May 27 14:03:38 daq-rix-andor-01 kernel: usb 1-2.4.1: usbfs: interface 0 claimed by usbfs while 'andor' sets config #1
May 27 14:03:39 daq-rix-andor-01 kernel: usb 1-2.4.1: usbfs: interface 0 claimed by usbfs while 'andor' sets config #1

This seems to happen to the direct beam path andor an order of magnitude more often than the one on the VLS. Too recover from this when it happens do the following procedure:

  1. Power off the camera (NOTE: to do this I believe you have unplug the power supply, since the power supply just has a rocker switch which I think selects between standard-cooling/deep-cooling?).  The only LED that gives any indication of power is a green one inside the power supply, visible only if you put your eye somewhere near the deep-cooling ("||") side of the rocker switch at a specific angle.
  2. Power off the camera side of the extender
  3. Unplug the server-side extender USB cable from daq-rix-andor-01. The DIR andor extender has the black USB cable (top USB port) and the VLS andor extender is the silver cable (bottom USB port).
  4. Re-plug the server-side extender USB cable into daq-rix-andor-01.
  5. Power on the camera side of the extender. You should see the link led turn on on the extender, and the host led blink slowly.
  6. Power on the camera. The host led on the camera side extender show turn on solid now, and the activity led should blink intermittently.
  7. Restart the IOC (the existing IOC process my be locked up and may need to be kill -9 to die) or alternatively just reboot daq-rix-andor-01.

Remote Power Cycling of Cameras

As of Nov. 17, 2023 rix now has a control panel to remotely power cycle andor cameras:

Debugging

You can see log files for the IOC here:  /cds/data/iocData/ioc-rix-dir-andor/iocInfo/.

For camera communication errors ("Unknown" in EDM "Model Number" field) can look for "USB disconnect" error messages in /var/log/messages.  Can see device numbers with "lsusb | grep Andor" and compare to device numbers in /var/log/messages.  Here is an example with two cameras hooked up to one node:

daq-rix-andor-01:~$ lsusb | grep Andor
Bus 001 Device 071: ID 136e:0005 Andor Technology Ltd. 
Bus 001 Device 063: ID 136e:0005 Andor Technology Ltd. 
daq-rix-andor-01:~$

Other debugging techniques:

  • telnet to the ioc and restart
    • when restarting, ioc connects to all the cameras to see if it's the right one
  • Sometimes have to reset all usb hubs/bridges like this:
bash-4.2# ~ddamiani/reset_andor.sh 3-2
bash-4.2# ~ddamiani/reset_andor.sh 3-2.4
bash-4.2# ~ddamiani/reset_andor.sh 3-2.4.4
bash-4.2# ~ddamiani/reset_andor.sh 3-2.4.4.1
bash-4.2#


Workaround for RIX ANDOR-DIR hang

NOTE: We have seen cases where this does not work and we are forced to power cycle daq-rix-andor-01 to recover.

This is for when the camera is visible with "lsusb | grep Andor" (note uppercase "A") but the camera isn't communicating:

  • Execute the following command on daq-rix-andor-01 as superuser: ~ddamiani/reset_andor.sh 1-2
  • Restart one IOC (e.g. DIR), wait for it to finish
  • Restart the second IOC, wait for it to finish

At the moment there is no way to know which camera is which, except for unplugging them one at a time.

The argument to the script ("1-2") can change, in principle but should do so rarely.  The “1-2” is coming from the USB bus and port enumerations from the operating system:
 
ddamiani@daq-rix-andor-01:~$ egrep 136e /sys/bus/usb/devices/*/idVendor
/sys/bus/usb/devices/1-2.3.1/idVendor:136e
/sys/bus/usb/devices/1-2.4.1/idVendor:136e
 
The first part of the for X-Y.blah bit which is 1-2 currently.

[root@daq-rix-andor-01 cpo]# more ~ddamiani/reset_andor.sh
#!/bin/bash

if [ $# -ne 1 ]; then
    echo "Usage $0 <device>"
    exit 1
fi

echo "${1}" > /sys/bus/usb/drivers/usb/unbind
echo "${1}" > /sys/bus/usb/drivers/usb/bind
[root@daq-rix-andor-01 cpo]#

EVR Issues

On two occasions we have seen the EVR go into a bad state (click on the "EVR" button in upper right of the andor ioc window) cause some camera to not trigger at all and one camera to glitch between a "locked" and "unlocked" state.  The fiducial-rate turns red and fluctuates away from 360Hz:

We tried recovering from this by rebooting the IOC and it worked for some minutes, but in the end in both cases we had to power cycle the node.

USB Extender

http://www.icron.com/products/icron-brand/usb-extenders/fiber/usb-2-0-ranger-2324/

Icron Ranger 2324 USB 2.0 extender.  Correct operation shows blue power LED, solid green "link" LED, solid green "host" LED, blinking yellow "activity" LED.  Manual is here:

usb-2-0-ranger-2324-manual.pdf

Changing Cameras

When there are multiple cameras connected to a single node the IOC uses the camera serial number to decide which one to talk to, so the serial number must be changed in the IOC startup script when a camera is changed. The serial number is set in lines like this:

ddamiani@daq-rix-andor-01:/cds/home/c/cpo$ grep CAM_ID /cds/group/pcds/epics/ioc/common/andorCam/R0.6.0/children/build/iocBoot/ioc-rix-dir-andor/st.cmd
epicsEnvSet( "CAM_ID",       "26601" )

The serial number for a camera can be determined like this (I think this program connects to cameras that are not already connected to by other software like IOCs):

daq-rix-andor-01:~$ /reg/g/pcds/dist/pds/current/build/pdsapp/bin/x86_64-rhel7-opt/andorStandAlone -m
SDKVersion: 2.100.30026.0
Found 2 Andor Cameras
Waiting for hardware to finish initializing...
DeviceDriverVersion: 0.0.0.0
Software Version: eprom 0 coffile 0 vxdrev 0 vxdver 0 dllrev 100 dllver 2
Hardware Version: PCB 0 Decode 0 FirewareVer 20 FirewareBuild 24
Camera serial number: 26601
Camera Head Model: DU940P_BN
etc.
  • No labels