Pietro Confluence Information

Running ePixUHR-35kHz (v2) with GT Readout Platform

Notes

https://github.com/slaclab/epix-uhr-dev
should use v1.0.1
contains the FEB firmware (both with/without asic emulation)

qsfp transceiver:
- 3 lanes of data (0-2)
- 1 lane of timing (3)

devGui:
- --emuMode=1 for asic emulation firmware, --emuMode=0 for real asic

use generic firmware for kcu1500:

drp-srcf-cmp004:~$ cat /proc/datadev_0 
-------------- Axi Version ----------------
     Firmware Version : 0x2010000
           ScratchPad : 0x0
        Up Time Count : 3729983
            Device ID : 0x0
             Git Hash : 4bcd9306a2d41d8bd37afaf1d442f5722e2013c1
            DNA Value : 0x00000000400200010117ed6125408385
         Build String : XilinxKcu1500Pgp4_6Gbps: Vivado v2021.2, rdsrv317 (Ubuntu 20.04.4 LTS), Built Fri 26 Aug 2022 10:23:37 AM PDT by ruckman

new pattern mode:
can be used to under

epixhr:
- pixel order descrambling in firmware

epixuhr:
- future: pixel order descrambling
- a single asic would be in natural order
- a 2x2 "piece of silicon" will have 4 asics. 4MPx has 4x1MPx (with
  "many" asics ... 6x2?)

"viewer" and "ruckus" are potentially optional submodules to include
in the DAQ.

to do list for riccardo:
- get packages (and possibly submodules) into conda
- work with Pietro to learn how to run devGui
  o run it on your own with your own git checkout
- identify a node (either lab3 or fee-alcove)
- program the generic kcu1500 software
- plug in the module using multimode transceivers/fibers, use
  a breakout cable to plug in xpm timing
  o they are NOT using XpmMini for timing: a standalone board KCU1500:
    https://confluence.slac.stanford.edu/pages/viewpage.action?spaceKey=~weaver&title=Standalone+PCIe+XPM

data node: rdsrv318
timing node with kcu1500: pc94280

Install in Lab3
- received package epix-uhr-dev and submodules
- run by using the command line

python scripts/devGui.py --dev /dev/datadev_1 --emuMode 1 --linkRate=636

the linkRate value can change, check what value is requested at startup and if needed restart it
search in the output for:
###############################################################################
Serial Rate = 636 Mb/s detected

- node used drp-tst-dev008
- kcu1500 general firmware installed
- asics emulation firmware installed (EpixUhrEmulation-0x01000000-20230512090347-pking-a49ffa9.mcs)
- first 3 lanes are connected via 2 multi mode breakout cables
- 4th lane is connected to the distribution unit to acquire timing (no XPM)

Initial Discussion

epixuhr:
- can download software
- one set of bare boards in lab1 (have been running it)

https://github.com/slaclab/epix-uhr-dev
instructions for running devGui above
kcu1500 firmware from https://github.com/slaclab/pgp-pcie-apps/releases

goals:
- get events flowing in the daq
  - check that timing link from the xpm works
- charge injection calibration software can be written now, but
  won't provide real data until we have real asic

emulator needs 6V digital and 6V analog power supply
will provide banana plugs
detector group will provide power supplies

will go in lab3 or FEE alcove (currently in lab1)

major differences between HR/UHR:
- 17kHz frames (eventually 35kHz)
- shape is different
- saci bus is different (register writes)
  o all the registers going to the asic are on saci
  o all other registers are on axi
- definition of saci registers is in

start from curent HR or from devGui? Lots of changes to make clocking
synchronous so not clear.

talk to pietro about technical stuff
lower priority than epixM and epixHR

Fiber Count

Summary

From Matt on Aug 22, 2024: This is how the fiber count goes for the ePix detectors:  6 detector segments / MP.  Each segment needs one timing link, so a 4MP detector needs 24 timing links.  The timing links get merged with the data links in the detector interface box.  I think you should count those fibers as independent from the data links.  Data links are 3 pairs of fiber (LR4 scheme) per detector segment, so 72 pairs per 4MP.

Fiber Count Reduction

Matt and Larry have a good summary of fiber-count-reduction options here: https://docs.google.com/presentation/d/1Z8OS7qLrYs-SCkenFze7cyB6ErPXyOBW65gDNdNz5Q8/edit?usp=sharing

Data Fibers

slack discussion with dionisio and cpo on jan. 17, 2024 for 35kHz 16Mpx

So are you saying to get a 35kHz 16M that it’s 24*12*4=1152 fiber pairs?  That seems “challenging”, to put it politely.

I agree that it is challenging
let me check how many we are effectively using so we can be closer to reality that to the upper bound
This is the current thought on fibers
based on the bandwidth we could reduce from 12 to 8 fibers per transceiver
but if we look forward to the future 100kfps we need all fiber and compression…

Timing Fibers

From Matt on Feb. 13, 2024: Dionisio told me yesterday that there are 6 segments for every Mpixel, and one timing link per segment.  So 16Mpx * 6 segm/Mpx = 96 timing links.

Preparation for Nov. 2024 Beamtime

Notes from mtg on Sept 27, 2024 with Claus, King, Melchiorri, Weaver, Doering, Batyuk, cpo

Note that only 48x48 regions per asic have sensors.

Beamtime is Nov 8 and 11

cooling 50W
24V power supply and 3.3V supply
- pietro will provide a diagram
both detectors in lab1
c1100 or kcu1500?
- kcu1500 @10Gbps
up to 12 lanes? using 6.  leap transceiver (12 bidirectional)
- 1 timing
which parts do we take?
- detector
- conversion box (need to share, take the epixM conversion box after Tuesday)
- power cables
which parts do we provide?
- kcu1500
- chiller
- power supply from ASC or lab3 one? (may not reach 24V)
- transceivers
- fibers
remote power supply? get shut-off unit from TID
software status? need new detector packages
 - ruckus, surf, lcls2timingcore, axi*,

rogue5? pietro will move to 6.1.1
beb detector in kcu? no beb
which detector?
- one on pietro's desk  (move the other one to pietro's desk)
- work with dionisio to pick right one
epixm issues for epixuhr?
- DRP port is important
- fiber powers exist for both timing/data
environmental monitoring for IOC?
- temperature monitoring
- record it in epicsarch
- available through register but not in a stream, can't be done with IOC
- automatic data will be lost
- descrambling is different
- separate beb subframes
- 4 asics 4 lanes 162*198
- camera beb frame+timing
- headers/trailers?  data format and descrambling from pietro
- can lanes go out of lock like epixM? so far have not lost lock
- cmp003
- kcu pgp4 10Gbps same as epixM

timing/triggering similar to epixM?
- daq/run triggers? should be same as epixM. "synchronous" timing, less jitter
- going to run similar phase-space of timing during the beamtime?
  o run trigger 35kHz
  o daq trigger 120Hz, scale it up to pgp saturation (20kHz)
    20kHz=5GB/s on the edge for one node

Geometry: ASIC Orientations and Readout Order

from Pietro on Oct. 31, 2024

Picture showing asics 2,3 are rotated 180 degrees (note the "double line" for wire-bonding):

Per-asic test pattern which can be used to check that software has correct asic orientation:

Gain Information

From Pietro: ePixUHR 35kHz - Pixel configuration and gain settings

Unlike epixM, Pietro says that charge-injection should be "set it and forget it".

Raw Data Format

Phil says pixel size is 100umx100um

From Pietro: GT Readout Platform with UHR-35kHz%28v2%29 - Data format and readout