Table of Contents

Terminology

An e-group is a collection of up to 8 configurable-bandwidth e-links, with a total rate of 1.28 Gb/s. Each Rx (uplink) e-link can be configured at 80, 160, 320, 640, or 1280 Mb/s. For ITk pixel, there is 1 e-link @ 1.28 Gb/s per e-group.

Each GBCR or lpGBT takes 6 data e-groups (24 e-groups per optoboard). Each lpGBT merges 6 e-groups into a fast link running at 10.24 Gb/s. Note that 6 x 1.28 Gb/s = 7.68 Gb/s. The difference of 2.56 Gb/s arises from the FEC12 protocol. Fast links are converted from electrical to optical signal by the VTRX+.

References:

FELIX2 Server - Deployed in B33

Instructions for cold startup of readout for B33 system test:

Optosystem hardware setup

We are currently only using 1 optoboard at a time (primarily 4400069, which is v4). Ensure that the VTRX+ optical ferrule has its glossy notch opposite of the key on the MPO fiber labeled "FELIX2". If you invert the connection, you will not see light on fast link 0. The optoboards receive 2.5V distributed through the connector board and power boards in the optocrate. Each power board has 5 BPOL12V receiving 12V. See Optosystem Routing for the mapping. The v4 optoboard sits in slot 5, which receives power from BPOL12V C.

Make sure the Thermocube chiller is on at 15C, the temperature will stabilize quickly. With 12V, the current will reach 0.2A with 1 optoboard unconfigured and will climb to 0.3A after configuration.

Felix/optoboard setup

Setup for Felix is cumbersome, so pick a machine to do it on and leave it connected for the whole working session (use the command caffeinate if you are connecting to Felix from a MacBook). There are some actions that require interacting with the FLX-712 card which cannot be run at the same time (noted by standalone running in the following).

You will need to open 6 tabs on felix2: 1 for random work, 1 for optoboard configuration, 1 for YARR, and 3 for Felix-star. In each, first ssh to felix star 

ssh <username>@atlascr.slac.stanford.edu. ## if not a valid user contact Matthias

# from inside atlascr ssh to felix 
ssh slacrce@felix2

For all 6 tabs, do the following:

cd daq
source alias.sh
cd optoboard-system
source flx_opto_setup.sh # one tab should use flx_opto_setup_fresh.sh (standalone running)

# In optoboard tab only
cd optoboard_felix
source setup_packages.sh

# In YARR tab only
cd yarr/YARR-flxclient
caltrig # standalone running

In the 3 Felix-star tabs, run the aliases fstar-tohost, fstar-toflx, and fstar-register. NOTE: If you see "error: /tmp/bus/stats does not exist or is not FIFO", simply run mkfifo /tmp/bus/stats. We are now using the itk-ic-over-netio-next communication library (previously we used flpgbtconf) so we you must leave these running for all subsequent steps. Note that to run these commands, you may need to run make-fifo in case you see an error related to elinks not being found when running felix-star.

Optoboard setup. Power on the optoboard. Run flx-info podpower to confirm you see light (should be ~700 µW on channel 0 for master lpGBT). Run flx-info link to confirm that link alignment is achieved on channel 0. If it is not, power cycle the optoboard (this is necessary due to a bug in the lpGBT v1 design affecting startup).

Once you have link 0 aligned, run the alias initopto in the optoboard tab. It should take about 30 seconds to configure the optoboard. Then you can run workopto which will start an interactive python session. You can do the following things:

from routines import *
from configurations import *

set_nominal_configuration(opto, barrel_Zm_PP0_twinax_config['4400069']['J2-J3']) # J1 and J4 are also available
bertPRBS7(opto.lpgbt1) # PRBS7 pattern checker on lpGBT, needed to check uplink and downlink polarity

opto.lpgbt1.tx_polarity_swap(dl) # dl = 0-7, which correspond to the downlink lanes 0,2,4,...,14 in the connectivity spreadsheet and control downlink on all 4 lpgbt from lpgbt1 alone, since it is the master
opto.lpgbt1.rx_polarity_swap(egroup) # egroup=0-5

# To run parallel soft error scans
opto.parallelsoftErrorPhaseScan(FD_OL_LPGBT_EG__0 + FD_OL_LPGBT_EG__2, meastime=1, filename="phaseScans/softErrorPhaseScan_myTest", API=False)

opto.parallelsoftErrorScan(FD_OL_GBCR_EG__0 + FD_OL_GBCR_EG__2, meastime=1, filename="eqScans/softErrorScan_myTest", API=False, plot=True, plotOnly=False, HFmin=0, HFmax=15, MFmin=0, MFmax=15, jump=5)

# Some other helper functions
set_phase(opto.lpgbt1, egroup=0, phase=0)
set_phase_mode(opto.lpgbt1, egroup=0, mode=0)
get_current_phase(opto.lpgbt1, egroup=0)
set_eq(opto.lpgbt1, egroup=0, MF=8, HF=8)

# Need triplet to be configured in PRBS7 mode
opto.bertScan("lpgbt1", "gbcr1", 1, 6, bertmeastime=11, filename="eqScans/bertScan_TAG", plot=True, HFmin=0, HFmax=4, MFmin=0, MFmax=4, jump=2)

bertPRBS7(opto.lpgbt1) # All channels for 3 seconds

# Retiming
opto.gbcr1.set_retiming_mode(channel=0, phase=5) # Run per channel

These python references can be found in /home/slacrce/daq/optoboard-system/optoboard_felix/lls/inner_system/, and are things I set up myself for convenience to allow polarity/equalization settings to be fixed for a given setup.

Services configuration. For each set of connections through a PP0 bundle, the uplink/downlink polarity (and optionally the optimal GBCR equalization and lpGBT phase). This is saved for the L0 PP0 connections as shown above. To determine the correct Tx configuration, you need to do trial and error (i.e. checking if the module is configured). To check the Rx polarity, you need to configure the chip configs to send PRBS7 by setting SerSelOut0/1/2/3: 2 (1 is used for nominal 64b66b). Then you can use the bertPRBS7 function to check if you get near 100% (wrong polarity) or near 0% (correct polarity) BER. You can keep settings to be -1 if you want the code to skip configuring them when you set the nominal configuration. 

Running YARR

After optoboard is configured with correct downlink polarity, you can run for example the following:

bin/scanConsole -l configs/logging/default_and_trace_front_end.json -c /home/slacrce/daq/yarr/common-configs/connectivity/L0_Stave/20UPIMS2102138-20UPIMS2102141_cold.json -r /home/slacrce/daq/yarr/YARR-flxclient/configs/controller/felix_client_pixels.json -s /home/slacrce/daq/yarr/YARR/configs/scans/rd53b/std_digitalscan.json -p

The connectivity files for L0 are all saved in the L0_Stave folder. All the chip configs are already modified to work with FELIX readout. See the python script for updating the configs. 

In case of doubt about stable connection use falign to check. 

That's it! If you have questions, ask Brendon. Good luck! slightly smiling face 

FELIX Server - B84 Data Transmission Testing

The felix server is used for DAQ for the services chain testing in B84. It is based on direct installation of software, rather than containerized software builds. 

FELIX versions: 5.1.3 (software) + itk-ic-over-netio-next, 5.1 (firmware), 4.19 (driver)

optoboard_FELIX: 1.0.52

YARR: 1.5.4

Notes about itk-ic-over-netio-next

For commissioning, needed to run the following commands to get fast link alignment with first lpGBT:

fgpolarity -c 0 -r set
flx-config -d 0 set LPGBT_FEC=0xFFFFFFFFFFFF

 To run itk-ic-over-netio-next, which significantly improves QOL by increasing the speed of optoboard configuration and allows Felix-star to be run continuously, requires installing it in the FELIX software build and setting the following environmental variables (which are set up in the flx_opto_setup*.sh scripts)

export INTERFACE=192.168.1.201 # Needs to be set according to the IP of eno1 from output of ifconfig command
export FELIX_BUS=/tmp/bus/
export TX_TAG=17
export RX_TAG=25


FELIX3 Server - Work In Progress

The felix3  server is used for DAQ for LLS pre-production system tests. It is intended to be used based on dockge containers for Felix software, microservices, and YARR. 

Setup

One can gain an initial familiarity with the system by reviewing the tutorial by Benedikt Vormwald (GitLab, slides). All of the relevant containers are given in the itk-demo-sw GitLab project.

  • Dockage password/username can be reset using docker compose exec dockge pnpm run reset-password

The firmware/software version requirements were established based on the data transmission chain test setup in B84 using felix2 (see Optoboard / Optobox / Optopanel). 

FELIX

Felix firmware/software/etc versions and documentation can be found at atlas-project-felix.web.cern.ch.

  • FELIX firmware. The 24 channel Pixel firmware is required. The latest release as of March 2025 is 5.1 (built July 2024). See information related to x8x8 PCIe bifurcation. Flashing the firmware with fflashprog (see FELIX Readout for more information). This determines which firmware is loaded from flash memory on power up. Even easier is to start the felix container and so source /scripts/flashFW.sh, which uses the firmware .mcs file in the microservices (which is currently 24CH 5.1). Open the script to be sure you're flashing the right device. In case the FLX-712 is programmed for the first time, or otherwise has non-matching HW (4) Regmap, felix software (including the docker container) cannot be run. To fix it, the board needs to be manually programmed with Xilinx programmer/vivado.
    • First program the FLX cards using vivado. Then do a restart (not shutdown), using sudo shutdown -r now. Then program using fflashprog (in the felix container is preferred, but can do standalone).
  • FELIX hardware drivers. For the above firmware, the driver version 4.19 or above is needed (4.20 is currently installed). See felix drivers for instructions.
  • FELIX software. To use the 24CH release, version 5.0.4 or greater is needed. See FELIX-Software-Updates for details on revision history. See microservices section below.
  • Tips and tricks
    • Vivado lab 2024.1 is installed under /tools/Xilinx/Vivado_Lab/2024.1, and can be setup with source VivadoLab_2024.1_settings64.sh from the home directory.
    • After firmware installation using Xilinx programmer, check flx-info -c0 or cat /proc/flx to see if the new firmware is loaded. If you see the error ### ERROR open: Error from ioctl(GET_TLP) then you are not using the latest firmware driver.
    • When running Init FELIX in the GUI or flx-info on the command line, the message ### ERROR open: Failed to open /dev/flx0 likely indicates that x8x8 PCIe bifurcation is not correctly set up. 
    • PCIe slots 2 and 5 have x8x8 bifurcation already enabled. This can be set in the BIOS. Reboot, repeatedly press delete. Navigate to Advanced >> Chipset Configuration >> North Bridge >> IIO Configuration >> CPU1 Configuration. The FLX-712 cards are therefore installed on slots 2 and 5.

Microservices

We have a branch of Benedikt's tutorial GitLab repository checked out under the following path: /home/atlas/LLS-system-test/microservices

  • This was set up following the README (running the docker compose command in the command line will replace any previously composed containers in another location)
  • The branch is SLAC, modifying the branch provided by Juliette Martin (mm_stable) to mount the LLS-system-test folder (which can be accessed when using the bash interface on Dockge).

Hardware

The FELIX cards take 24CH MTP #2 (on the right), which fans out to 12 Rx lanes (CH1-12) and 12 Tx lanes (CH13-24). See pages 16 and 17 for diagrams.

Running

From Firefox, go to localhost:5001/compose and do the following:

  1. Start the felix container. 

If you want to do work in the terminal (and the dockge bash application annoys you), you enter the relevant container folder (e.g. felix) in a terminal and run docker compose exec felix bash.

  • No labels