Discussion on May 27, 2021

https://www.surface-concept.com/detectors/

Yves Acremann and Matt Weaver and engineer from Surface Concept

surface concept DLD (a.k.a. k-microscope endstation)
dld (delay line detector) with yves acremann
timing interface board: matt's timing signal with LVDS output to DLD.
- like camlink converter but needs timing input interface
- triger TTL output <40ps jitter from "tprtrig"?
- triggering: 3.3V TTL

who is involved?
- slac: zx shen
- germany
- lcls: bob schoenlein (georgi) Jake Koralek
schedule?
- start in spring 2022 in NEH2.2
monitoring
- we'll try ami and/or psana-python from shared memory
language of monitoring software?
- low level c++
- higher level python

data aggregated before daq node with switch going to ONE 10Gbit ethernet
data path: ethernet into one daq node
need deadtime

dead time requires buffering back pressure to work from the DAQ into the FPGAs (in particular over the ack'd UDP protocol that surface concept uses for ethernet).  best guess at the moment is that this works.

interface to DAQ (between surface-concept driver receiving UDP (ack'd) and DAQ)
two callbacks:
- a block of each electron has x,y,t, and lower-bits of pulse id
- are electrons in pulse-id-order? if not resorting could be a problem @1MHz
- they don't expect many electrons per pulse (10?)
- bytes per electron: 64-bit, 2 bytes x,y, t is 3 bytes, and lower-bits of pulse-id (80MB/s at 1MHz). number of bits are fixed, but can be changed in hardware if necessary.
- full timestamp (for upper bits) once per ms

Total data volume: 10 electrons * 8 bytes/electron * 1MHz = 80MB/s.

could maybe ship a tdc plus pulse generator to simulate electrons earlier to slac.

LCLS-II DAQ integration diagram from Yves.  They will generate a "full" signal.  The segment level code is on the top right in the box labelled "DAQ".  There is an interface between the surface concepts driver and the segment level code.

A diagram from Yves showing their shmem analysis.  They will likely use psana/ami.

Discussion on June 23, 2022

With Yves/Matt/Chris

  • how do we time-in the detector
    • proposal: we run a low-rate beam, see a "prompt" signal (high energy, lower TOF electrons), move the timing to be close to that prompt signal
    • other possibility: use a diode with accelerator at the experiment and use the simulator box to "calibrate" extra delays in the system.  one cable goes to photodiode, other goes to the simulator (either start signal, or output signal)
  • do electron hits cross pulse id boundaries
    • yes, this can happen
    • analysis software can't tell if an electron really belongs to this pulse (not complicated algorithm to assign electrons to other pulse-ids)
    • try to configure voltages so this possibility is minimized
  • how do we know what the pulse id boundaries are (e.g. is there one udp packet per pulse)
    • happens naturally 
  • clarify interface between daq and surface-concepts ack'd-udp code
    • this scTDC code is provided by surface concepts
    • need the "simulator box" and the "TDC box" (maybe received at SLAC sometime in July?)
  • ami plots needed

TDC box needs a second ethernet interface for DHCP and configuring the delay (start pulse to the detector).  ethernet interface to AXI4 registers.  also registers for configuring the pause threshold to the DAQ.  Python software exists for this.

SC software is a special version with support for the timestamp.  Yves will provide.  Closed source.  Will receive binaries and headers.  We're currently using GCC11.   GUI to play around with it.  Dependencies? QT5, sqlite.

Dedicated network interface for the bulk data (e.g. static IP).

Additional software for a GUI for the simulator (over USB, from linux).  Python software exists for this.

Potential issue: we may need both timestamp and pulse-id.  Yves only sends surface concepts FPGA all bits of the pulse-id and they send full pulse-id once per ms and include lower bits with every electron.  But LCLS receives a "reconstructed" full pulse-id with every electron.  The smaller number of bits (17 bits) used for pulse-id internally by SC could perhaps break if we go "dead" for a long time.  We would get incorrect pulse-id's for a while, but would fix itself after about a millisecond.  We will need to get the timestamp in the DAQ. in another way, probably with a KCU1500 receiving the timing system.  Need to match the SC data with the KCU timestamp using the pulse-id.

They use ubuntu, need to see if we need to change compilers to be compatible with conda.

https://www.surface-concept.com/downloads-detectors/# (SDK examples)

www.surface-concept.com/downloads/software/tmp/DldGui2_Ubuntu2004_v0.2.3_scTDC_v1.3013.12.tar.gz (GUI software)

www.surface-concept.com/downloads/software/2022/scTDC1_devclass45_ubuntu2004_v0.3.8.tar.gz (shared libraries)

www.surface-concept.com/downloads/software/2022/scTDC1_ubuntu2004_v1.3014.11.tar.gz (more shared libraries)

Martin Ellguth of Surface Concept recompiled some libraries for us:


Hi Matt,

I prepared a Docker image based on a Ubuntu 20.04 LTS image, compiled gcc-12.1.0 from sources and used it to build the following archives:

www.surface-concept.com/downloads/software/2022/tmp/scTDC1_ubuntu2004_gcc12_v1.3017.5.tar.gz
www.surface-concept.com/downloads/software/2022/tmp/scTDC1_devclass45_ubuntu2004_gcc12_v0.3.82.tar.gz

The second link is the "device class" library needed for TDCs with an Ethernet port.

Mit freundlichen Grüßen / Best Regards

Dr. Martin Ellguth


Analysis Discussion

Meeting on July 21 2022 with Quynh, Yves, Matt, Jake, Jonathan Sobota, Markus Scholz, Georgi, cpo

Controls interface? who?  Markus Scholz in Germany.  Need SLAC controls contact.  iseg controller for HV is already at slac (jonathan's office).  hexapod also needs to be integrated.  smaract controller for linear motors for microscope (jonathan).  alex wallace group is working on it.

momentoscope python library dependencies: dask, xarray, pandas.  available in conda?  maybe they can do this (markus scholz)

pump probe time from timetool? laser-on and laser-off?  timetool high-rate line camera?  only need pump-probe time for analysis not real-time monitoring. step scan of time delay (programmed electronic delay stage).  use new line-camera timetool.

multiple projections of xyt:  xy, xt, yt, t-histogram, pump probe time, ROI, time-windowed

yves' "clean table" is like our h5 file: maybe we can't support variable number of electrons per event, but yves say max of 4 should work

yves works with chris and markus on ami

matt works with yves on daq integration

quynh works with silke/cpo on analysis

Hardware

Front side elex.  The simulator is shown on top.  The simulator is powered via a USB connector.  It receives a start pulse and then simulates the TDC output and transmits it via its rear HDMI connector.

Back side elex.  The TDC output from the simulator attaches to the HDMI TDC input on the front of the main chassis.  The fiber timing signal from an XPM attaches to the PulseID input of the main chassis.  Two ethernet connectors provide control and data.

Inside view.

Discussion On May 15, 2024

With Markus Scholz, Yves Acreman, Matt, Chris, Jake Koralek, Jon Sabota, 

Schedule:

KCU1500 receives a timing fiber for transitions and L1Accept timestamp, but the udp data is received over a standard network interface on the drp node (behaves like the BLD/PVA detectors in this respect).

May need to ask Martin Ellguth to compile for 13.1.0 (our latest conda compiler) but we'll see if the previous v12 still works for us.  If we had the udp-packet format then in principle we would not need the SC libraries (euXFEL reused the libraries)

Other things the SC library does:

  • provides parameters for calibration (sent over "dedicated ethernet" connection): these are applied on a per-event basis (maybe done on the fpga?).
  • stuff to avoid: projections (we will do it ourselves)
  • an API to receive the raw data
  • capacity of link doesn't support the full timestamping: the SC software adds the full timestamp

Configuration of detector:

  • via .ini file that is passed to SC software and sent over dedicated-ethernet link, which then configures the FPGA.
    • users don't touch .ini file, usually.  sometimes if things look weird then have to tweak (1) triggering "window" (when an electron is accepted) (2) timing "offset": a.k.a. start time (3) compensation for different cable lengths in the detector 
  • the timing system configuration is done via the service port (e.g. readout-group).  service port uses DHCP.

SC FPGA code is proprietary (rightmost in above diagram).  Leftmost FPGA is mostly open source (standard Xilinx): communication through service port is not open to the world.

80MB/s implies can all be done with 1 node.

Simulator can be configured over a USB port: e.g. maximum number of electrons per event (usually 0.1, but can turn it up).  With real data typically have 1 or 2 electrons per shot.

Start with the GUI first, then switch to API as a second step.

  • No labels