...
Currently specific for SLAC IS demonstrator:
4x 2x Optoboard V2.1 are housed in one Optobox. We have 1 v4 optoboard. This Optobox is housed in a Test-Optopanel.
We have 2 FELIX servers in B84 EPP lab, felix (running Centos7) and felix2 (running Alma Linux). The felix server has older versions of the optoboard software, whereas felix2 is brought up with the latest software and firmware releases. felix2 has to be cold rebooted, warm reboot will hang. The issue may be due to ethernet port being shared between the host and IPMI board.
Interfaces:
Please refer to our the Bern Interface Document here. All the QC connectors of the Optoboard System is explained there.
...
Optoboard system support:
We collect Bern collects support requests and general questions on our Mattermost channel Bern-Optoboard (invite link). Your contacts as of Aug. 2022 are Aaron O'Neill and Daniele del Santo.
Optoboard software for FELIX:
...
FELIX, optoboard software, and YARR setup:
FELIX firmware (see atlas-project-felix.web.cern.ch for more information):
- felix: firmware version 2221, recommended by Sasha on July 26
...
- ,
...
- 2024 (CERNbox link
...
- )
- felix2: there are a few versions of the firmware that can be used:
- Release from Ricardo Luz' March ITk week presentation
- A nightly after ae8a2460of master, pipeline ran on April 24 (build link)
- The testing bitfile directly from Ricardo
FELIX software:
- felix:
- felix2: software git commit 73d2d72a of master (compatible with the latest firmware). Note that for local compilation you need to use
source cmake_tdaq/bin/setup.sh x86_64-el9-gcc13-opt.
Microservices software for optoboard, developed by Bern: https://gitlab.cern.ch/bat/optoboard_felix. A lot is documented in the readme and wiki of the repository.
- felix: optoboard_FELIX
- felix2: same as above, tag 1.0.37
YARR (see Angira Rastogi's March ITk week presentation and LBL documentation):
- felix: v1.5.0, used primarily felixcore
- felix2: v1.5.1, using felix-star
Instructions for felix server:
In a first terminal felixcore:
The repository is located in the directory:
/home/itkpix/optoboard-system
To configure an Optoboard V2.1 with the default configurations found simply run these scripts:
In a first terminal felixcore:
| Code Block |
|---|
cd /home/itkpix/felix-sw/felix-distribution # Jan/2023 ITkPixV1. Previously was felix-sw/flx-sw-2022-08-11
source setup.sh
x86_64-centos7-gcc11-opt/felixcore/felixcore -d 0 --data-interface lo --elinks 0,4,8,12 # starts felixcore, check your links! |
Note that the elinks might change according your connectivity scheme. More information from Ismet here.
In a second terminal:
...
| Code Block |
|---|
cd /home/itkpix/felix-sw/optoboardfelix-system source flx_opto_setup.sh # warm start |
- In case the optoboard was just powered or got stuck, you need to do a cold start setup (this is incompatible with felixcore so that you need to terminate felixcore in the other window before doing this):
| Code Block |
|---|
cd /home/itkpix/optoboard-system
source flx_opto_setup_fresh.sh # cold start
|
At this point the master link between FELIX and optoboard should be established which you can check using the command
| Code Block |
|---|
flx-info link
|
which should typically show something like:
distribution # Jan/2023 ITkPixV1. Previously was felix-sw/flx-sw-2022-08-11
source setup.sh
x86_64-centos7-gcc11-opt/felixcore/felixcore -d 0 --data-interface lo --elinks 0,4,8,12 # starts felixcore, check your links! |
Note that the elinks might change according your connectivity scheme. More information from Ismet here.
In a second terminal:
- In case the optoboard setup was done previously and remained powered on, you can proceed with just a simple environmental setup:
| Code Block |
|---|
cd /home/itkpix/optoboard-system
source flx_opto_setup.sh # warm start
export PYTHONPATH="/home/itkpix/optoboard-system/optoboard_felix_2023Nov/src/:${PYTHONPATH}" # Needed in newer versions of optoboard software |
- In case the optoboard was just powered or got stuck, you need to do a cold start setup (this is incompatible with felixcore so that you need to terminate felixcore in the other window before doing this):
| Code Block |
|---|
cd /home/itkpix/optoboard-system
source flx_opto_setup_fresh.sh # cold start
|
At this point the master link between FELIX and optoboard should be established which you can check using the command
| Code Block |
|---|
flx-info link
|
which should typically show something like:
| Code Block |
|---|
Link alignment status
------------------------
Channel | 0 1
|
| Code Block |
Link alignment status ------------------------ Channel | 0 1 ---------- Aligned | YES NO Channel | 2 3 ---------- Aligned | YES NO Channel NO | 2 3 ---------- Aligned | NO NO |
which indicates the master lpGBT link (Channel 0) is established which indicates the master lpGBT link (Channel 0) is established so that further configuration can proceed. The optoboards only have the master lpGBT enabled at powerup while the other lpGBTs will be brought up at the optoboard configuration stage (this requires felixcore up and running):
...
| slot: | OB1 | OB2 | OB3 | OB4 | OB5 | OB6 | OB7 | OB8 | V4 |
|---|---|---|---|---|---|---|---|---|---|
| serial: | 2400006 | - | 2400003 | - | 2400007 | - | 2400011 | - | 4400069 |
| powering from bPOL: | A | A | B | B | C | C | D | D | |
| idle current* [mA] | 182 | - | 165 | - | 168 | - | 174 | - | |
| FELIX link: | |||||||||
| Current Location | SLAC | ANL | OSU | SLAC | SLAC |
*after power-up and inside the test-Optopanel with 9.0 V supplied. If powered outside the test-Optopanel directly on the connector with 2.5 V expect a current ~415 mA.
| History | Date | Comments |
|---|---|---|
| 2400011 | Oct/22-Feb/23 | Resistor R108 removed Oct/2022 to debug FELIX FEC12 mode problem. R108 back on Feb/2023. |
Connectors:
Check the above interfaces document for more detailled description.
| 2400006,2400011 | May/2/2023 | bpol2V5 mezzanine R10 330KOhm→316KOhm to raise V1.2 to V1.24 to stablelize lpGBT3,4. Overall current 0.92->1.03A. |
| 2400006/4400069 | Nov/20/2023 | Brought in V4 board 4400069 in exchange of 2400006 to go back to Bern. |
Optobox/Board service connections:
Optobox LV/DCS services:
Check the above interfaces document for more detailled description.
Banana Banana plugs are labelled according to powered bPOL12V.
Cooling and dry air inlet/outlet: FESTO 8 mm
...
Currently no MOPS is mounted on the Powerboard inside the test-Optopanel (due to availability). We will ship one as soon as we got our hands on them.
SLAC Single Optoboard Test Setup
The initial single optoboard setup used for the early SLAC tests with single chip cards is summarized in the diagram below:
...
OptoBoard LV/DCS:
For optoboard standalone test setups, a mini LV/DCS service cable bundle is needed to supply 2.5V to optoboard and lead out its NTC monitoring out through the Samtec SFM-104-01-L-D connector .
Standalone optoboard power adaptor cable spec
Optoboard-FELIX Fiber Connectivity:
Each FELIX PCIe card has two MPO24 ports, serving 12 pairs of TX+RX each. Our lab test setups typically use fiber splitter bundles to breakout the MPO24 into individual LCs and use LC patch panels to map connectivity to the optoboard VTRX+ fiber pigtail. How the FELIX MPO24 port is structured is described in the FELIX readout and Direct FELIX confluence pages. There are two ways to breakout the FELIX MPO24:
- Most setups elsewhere and our easier setups all used MPO24→2xMPO12→24xLC splitters are described in the FELIX readout and Direct FELIX confluence pages.
- Our latest setups used direct MPO24→24xLC splitter in one step. The actual component inventory is in the "Fiber + patch" tab of the of test readout component Stanford Drive GoogleSheet.
How the VTRX+ fiber pigtail lines are structured is described in the Optosystem Interface document AT2-IP-GM-0010 (Table 4) with 5 out of 12 fibers active per optoboard which can be followed from the MPO12-LC splitter fiber color coding.
SLAC Single Optoboard Test Setup
The initial single optoboard setup used for the early SLAC tests with single chip cards is summarized in the diagram below:
| Bonn miniDP-SMA | Bern ERF-SMA | lpGBT |
|---|---|---|
| AUX | DOWNLINK 01 | 1 |
| LN3 | L1 UPLINK 01 | 1 |
| LN2 | L1 UPLINK 02 | 1 |
| LN1 | L1 UPLINK 03 | 1 |
| LN0 | L1 UPLINK 04 | 1 |
Each fiber in the pigtail corresponds to data coming from a separate lpGBT (verification needed for RX 3 and 4):
| lpGBT | VTRX+ fiber | FELIX fiber |
|---|---|---|
| 1 | red (7) | TX 1 |
| 1 | white (6) | RX 1 |
| 2 | slate (5) | RX 2 |
3  | brown (4) | RX 3 |
| 4 | green (3) | RX 4 |
Note that the Optoboard VTRX+ fiber pigtail
Each fiber in the pigtail corresponds to data coming from a separate lpGBT (verification needed for RX 3 and 4):
...
Note that the Optoboard VTRX+ fiber pigtail MT ferrules connection to MPO12-LC fiber splitter is a simple slip on which requires careful manual recognition of right polarity (see ITkPixV1 readout page) which can easily slip off but prevents accidental large force breaking the fiber. It is advisable to always check the link light level with flx-info podpower to verify the fiber link is healthy (>700uW). How the VTRX+ fiber pigtail lines are structured is described in the Optosystem Interface document AT2-IP-GM-0010 (Table 4) with 5 out of 12 fibers active per optoboard. How the FELIX MPO24 port is structured and fanout with MPO24->2xMPO12→LC splitters are described in the FELIX readout and Direct FELIX confluence pages.
Current firmware allows only 2 lpGBTs to be readout, such that the link alignment string is fixed to Current firmware allows only 2 lpGBTs to be readout, such that the link alignment string is fixed to x0000000300000000 for channels 00 and 01 in unaligned state. Alignment can be checked using data from module RX 0 (L0 on miniDP-SMA) after module configuration, giving the following link alignment strings independently of which e-links are subscribed on felixcore startup (for optoboard 2400006):
...
| Register | Value | Comments |
|---|---|---|
| dllEnable | True | Enable DLL clock for for retiming |
| dllCapReset | False | |
| dllForceDown | False | DLL can have harmonic lock problem which would need dllForceDown=T to release it |
| dllChargePumpCurrent | 0x0 | Set charge injection current to min value. Higher current enables faster locking but may cause larger jitter. We care less about locking time but care more about jitter. | dllClockDelay | scan | Scan this delay for the active channel for good BER window bathtub.
| it | ||
| dllChargePumpCurrent | 0x0 | Set charge injection current to min value. Higher current enables faster locking but may cause larger jitter. We care less about locking time but care more about jitter. |
| dllClockDelay | scan | Scan this delay for the active channel for good BER window bathtub. |
Vakhtang's readout software
Vakhtang Tsiskaridze (VT) wrote a standalone python package to perform digital scans outside of YARR. Use the following steps to run on felix.
For all windows (4 in total are needed):
| Code Block |
|---|
# Login and setup
ssh daq@felix # password: slacrce
cd ~/daq
source complete-setup.sh
cd ~/daq/itkdaq |
Window #1:
| Code Block |
|---|
# Monitoring for Tx sent to FELIX
./run-felix-toflx.sh |
Window #2:
| Code Block |
|---|
# Monitoring for Rx returned from FELIX
./run-felix-tohost.sh |
Window #3:
| Code Block |
|---|
# Runs an instance of felix-star
python run_readout.py |
Window #4:
| Code Block |
|---|
# Starts interactive session where configuration and digital scans are steered
python -i run_sendcmd.py>>> repower_init() # Quick power cycles the module and initializes with nominal configurations
>>> full_scan() # Must be done 2+ times, see below for example of output
>>> fec2.write_field("SLDO_VTRIM", 0xaa) # Hacky solution that fixes readout issues caused by chip 2
>>> scan() |
The output of a digital scan should appear in data, and can be plotted using the plotting script plot_occmap_quad.py.
| Code Block |
|---|
# Output after first full_scan()
[2023-12-04 16:40:44.526] [rd53b.decoder] [info] scan time: 3.77 sec, rate: 157.0 Mbps
[2023-12-04 16:40:44.526] [rd53b.decoder] [info] tval: 100, Error in pixels: 29730. vmin: 95, vmax: 24743
[2023-12-04 16:40:44.526] [rd53b.decoder] [info] tag_stats: 25600 25600 25600 25600 25600 25600 25600 25600 25543 25425 25421 25398 25378 25349 25326 25297
[2023-12-04 16:40:44.526] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 30099829 1 0 0 0 0 0 0
[2023-12-04 16:40:44.526] [rd53b.dataproc.quad] [info] [2] SCAN STOP
[2023-12-04 16:40:44.549] [rd53b.decoder] [info] scan time: 3.80 sec, rate: 155.6 Mbps
[2023-12-04 16:40:44.549] [rd53b.decoder] [info] tval: 100, Error in pixels: 32604. vmin: 93, vmax: 24666
[2023-12-04 16:40:44.549] [rd53b.decoder] [info] tag_stats: 25600 25600 25601 25600 25600 25600 25600 25600 25466 25336 25307 25287 25264 25236 25216 25185
[2023-12-04 16:40:44.549] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 30012842 0 0 0 0 0 0 9
[2023-12-04 16:40:44.549] [rd53b.dataproc.quad] [info] [1] SCAN STOP
[2023-12-04 16:40:44.572] [rd53b.decoder] [info] scan time: 3.82 sec, rate: 154.7 Mbps
[2023-12-04 16:40:44.572] [rd53b.decoder] [info] tval: 99, Error in pixels: 125302. vmin: 91, vmax: 24659
[2023-12-04 16:40:44.572] [rd53b.decoder] [info] tag_stats: 25592 25592 25592 25594 25592 25594 25592 25593 25458 25224 25314 25293 25271 25232 25208 25179
[2023-12-04 16:40:44.572] [rd53b.decoder] [info] tag_hits: 0 0 0 1 2 1056 0 6 29930769 0 0 0 0 0 0 0
[2023-12-04 16:40:44.572] [rd53b.dataproc.quad] [info] [0] SCAN STOP
[2023-12-04 16:40:44.594] [rd53b.decoder] [info] scan time: 3.84 sec, rate: 153.9 Mbps
[2023-12-04 16:40:44.594] [rd53b.decoder] [info] tval: 100, Error in pixels: 31999. vmin: 93, vmax: 24668
[2023-12-04 16:40:44.594] [rd53b.decoder] [info] tag_stats: 25600 25600 25600 25600 25600 25600 25600 25600 25468 25341 25306 25289 25264 25230 25215 25184
[2023-12-04 16:40:44.594] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 30016538 0 0 0 0 0 0 0
# Output after setting VLDO_TRIM on fec2
[2023-12-04 16:53:59.413] [rd53b.decoder] [info] scan time: 3.74 sec, rate: 92.1 Mbps
[2023-12-04 16:53:59.413] [rd53b.decoder] [info] tval: 100, All pixels are good.
[2023-12-04 16:53:59.413] [rd53b.decoder] [info] tag_stats: 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600
[2023-12-04 16:53:59.413] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 15360000 0 0 0 0 0 0 0
[2023-12-04 16:53:59.413] [rd53b.dataproc.quad] [info] [1] SCAN STOP
[2023-12-04 16:53:59.436] [rd53b.decoder] [info] scan time: 3.76 sec, rate: 91.6 Mbps
[2023-12-04 16:53:59.436] [rd53b.decoder] [info] tval: 100, All pixels are good.
[2023-12-04 16:53:59.436] [rd53b.decoder] [info] tag_stats: 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600
[2023-12-04 16:53:59.436] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 15360000 0 0 0 0 0 0 0
[2023-12-04 16:53:59.436] [rd53b.dataproc.quad] [info] [2] SCAN STOP
[2023-12-04 16:53:59.458] [rd53b.decoder] [info] scan time: 3.78 sec, rate: 91.0 Mbps
[2023-12-04 16:53:59.458] [rd53b.decoder] [info] tval: 100, All pixels are good.
[2023-12-04 16:53:59.458] [rd53b.decoder] [info] tag_stats: 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600
[2023-12-04 16:53:59.458] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 15360000 0 0 0 0 0 0 0
[2023-12-04 16:53:59.458] [rd53b.dataproc.quad] [info] [0] SCAN STOP
[2023-12-04 16:53:59.481] [rd53b.decoder] [info] scan time: 3.80 sec, rate: 90.5 Mbps
[2023-12-04 16:53:59.481] [rd53b.decoder] [info] tval: 100, All pixels are good.
[2023-12-04 16:53:59.481] [rd53b.decoder] [info] tag_stats: 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600
[2023-12-04 16:53:59.481] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 15360000 0 0 0 0 0 0 0
|
Additional notes:
- 17 different triggers, on 9th trigger getting all hits
- Scan time (~4 seconds), rate is per chip
- Number of tag hits should be 400 * 384 * N_triggers
- If you run 100 triggers, scan time is dominated by mask stepping
Some Useful Links
- Optoboard System Documentation
- FELIX JIRA (Oct/22) on optoboard + ITkPix readout setup
- CERN mattermost Bern-Optoboard channel
- Talk (Dec/9/2022) by Angira Rastogi on Optoboard-FELIX setup at LBNL
...