...
Currently specific for SLAC IS demonstrator:
4x 2x Optoboard V2.1 are housed in one Optobox. We have 1 v4 optoboard. This Optobox is housed in a Test-Optopanel.
We have 2 FELIX servers in B84 EPP lab, felix (running Centos7) and felix2 (running Alma Linux). The felix server has older versions of the optoboard software, whereas felix2 is brought up with the latest software and firmware releases. felix2 has to be cold rebooted, warm reboot will hang. The issue may be due to ethernet port being shared between the host and IPMI board.
Interfaces:
Please refer to our the Bern Interface Document here. All the QC connectors of the Optoboard System is explained there.
...
Optoboard system support:
We collect Bern collects support requests and general questions on our Mattermost channel Bern-Optoboard (invite link). Your contacts as of Aug. 2022 are Aaron O'Neill and Daniele del Santo.
Optoboard software for FELIX:
...
FELIX, optoboard software, and YARR setup:
FELIX firmware (see atlas-project-felix.web.cern.ch for more information):
- felix: firmware version 2221, recommended by Sasha on July 26
...
- ,
...
- 2024 (CERNbox link
...
- )
- felix2: there are a few versions of the firmware that can be used:
- Release from Ricardo Luz' March ITk week presentation
- A nightly after ae8a2460of master, pipeline ran on April 24 (build link)
- The testing bitfile directly from Ricardo
FELIX software:
- felix:
- felix2: software git commit 73d2d72a of master (compatible with the latest firmware). Note that for local compilation you need to use
source cmake_tdaq/bin/setup.sh x86_64-el9-gcc13-opt
.
Microservices software for optoboard, developed by Bern: https://gitlab.cern.ch/bat/optoboard_felix. A lot is documented in the readme and wiki of the repository.
- felix: optoboard_FELIX
- felix2: same as above, tag 1.0.37
YARR (see Angira Rastogi's March ITk week presentation and LBL documentation):
- felix: v1.5.0, used primarily felixcore
- felix2: v1.5.1, using felix-star
Instructions for felix server:
In a first terminal felixcore:
The repository is located in the directory:
/home/itkpix/optoboard-system
To configure an Optoboard V2.1 with the default configurations found simply run these scripts:
In a first terminal felixcore:
Code Block |
---|
cd /home/itkpix/felix-sw/felix-distribution # Jan/2023 ITkPixV1. Previously was felix-sw/flx-sw-2022-08-11
source setup.sh
x86_64-centos7-gcc11-opt/felixcore/felixcore -d 0 --data-interface lo --elinks 0,4,8,12 # starts felixcore, check your links! |
Note that the elinks might change according your connectivity scheme. More information from Ismet here.
In a second terminal:
...
Code Block |
---|
cd /home/itkpix/felix-sw/optoboardfelix-system source flx_opto_setup.sh # warm start |
- In case the optoboard was just powered or got stuck, you need to do a cold start setup (this is incompatible with felixcore so that you need to terminate felixcore in the other window before doing this):
Code Block |
---|
cd /home/itkpix/optoboard-system
source flx_opto_setup_fresh.sh # cold start
|
At this point the master link between FELIX and optoboard should be established which you can check using the command
Code Block |
---|
flx-info link
|
which should typically show something like:
Code Block |
---|
Link alignment status
------------------------
Channel | 0 1
----------
Aligned | YES NO
Channel | 2 3
----------
Aligned | NO NO |
...
distribution # Jan/2023 ITkPixV1. Previously was felix-sw/flx-sw-2022-08-11
source setup.sh
x86_64-centos7-gcc11-opt/felixcore/felixcore -d 0 --data-interface lo --elinks 0,4,8,12 # starts felixcore, check your links! |
Note that the elinks might change according your connectivity scheme. More information from Ismet here.
In a second terminal:
- In case the optoboard setup was done previously and remained powered on, you can proceed with just a simple environmental setup:
Code Block |
---|
cd /home/itkpix/optoboard-system
source flx_opto_setup.sh # warm start
export PYTHONPATH="/home/itkpix/optoboard-system/optoboard_felix_2023Nov/src/:${PYTHONPATH}" # Needed in newer versions of optoboard software |
- In case the optoboard was just powered or got stuck, you need to do a cold start setup (this is incompatible with felixcore so that you need to terminate felixcore in the other window before doing this):
Code Block |
---|
cd /home/itkpix/optoboard-system/optoboard_felix python InitOpto.py -optoboard_serial 00000000 -vtrx_v 1.3 -flx_G 0 -flx_d 0 -configure 1source flx_opto_setup_fresh.sh # cold start |
At this point the master link between FELIX and optoboard should be established which you can check using the command
Code Block |
---|
flx-info link
|
which should typically show something like:
Code Block |
---|
Link alignment status
------------------------
Channel | 0 1
----------
Aligned | YES NO
Channel | 2 3
----------
Aligned | NO NO |
which indicates the master lpGBT link (Channel 0) is established so that further configuration can proceed. The optoboards only have the master lpGBT enabled at powerup while the other lpGBTs will be brought up at the optoboard configuration stage (this requires felixcore up and running):
Code Block |
---|
cd /home/itkpix/optoboard-system/optoboard_felix
python InitOpto.py -optoboard_serial 00000000 -vtrx_v 1.3 -flx_G 0 -flx_d 0 -configure 1 |
The -optoboard_serial
specifies the serial number used (00000000 is the default one for Optoboard V2.1), -flx_G
is the link group number of FELIX and -flx_d
the FELIX device. The option -configure
set to 1 will configure all lpGBTs, if set to 0 will configure only the master. If -optoboard_serial
is set to 00000000 and -
The -optoboard_serial
specifies the serial number used (00000000 is the default one for Optoboard V2.1), -flx_G
is the link group number of FELIX and -flx_d
the FELIX device. The option -configure
set to 1 will configure all lpGBTs, if set to 0 will configure only the master. If -optoboard_serial
is set to 00000000 and -vtrx_v 1.3
, optoboard_felix sends the parameters in the config ~/optoboard_felix/configs/optoboard2_lpgbtv1_gbcr2_vtrxv1_3_default.json
to the lpGBTs, GBCR and VTRx+. Check the FELIX documentation or the optoboard_felix readme for more information on this. If this succeeded, the single optoboard LV current will go up to ~900mA.
...
A collection of photographs from the test-Optopanel and its sister at CERN SR1 can be found here. See also this Twiki here of the SR1 test-Optopanel which has similiar information (but some very different configurations).
Optoboards:
slot: | OB1 | OB2 | OB3 | OB4 | OB5 | OB6 | OB7 | OB8 | V4 | |||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
serial: | 2400006 | - | 2400003 | - | 2400007 | - | 2400011 | - | 4400069 | |||||||||||||||||||||||||||||
powering from bPOL: | A | OB1A | OB2B | OB3B | OB4C | OB5C | OB6D | OB7 | OB8 | V4 | D | |||||||||||||||||||||||||||
idle current* [mA] | 182 | - | 165 | serial: | 2400006 | - | 2400003168 | - | 2400007174 | - | 2400011 | - | 4400069 | powering from bPOL: | A | A | B | B | C | C | D | D | idle current* [mA] | 182 | - | 165 | - | 168 | - | 174 | - | FELIX link: | Current Location | SLAC | ANL | OSU | SLAC | SLAC |
*after power-up and inside the test-Optopanel with 9.0 V supplied. If powered outside the test-Optopanel directly on the connector with 2.5 V expect a current ~415 mA.
...
FELIX link: | |||||||||
---|---|---|---|---|---|---|---|---|---|
Current Location | SLAC | ANL | OSU | SLAC | SLAC |
*after power-up and inside the test-Optopanel with 9.0 V supplied. If powered outside the test-Optopanel directly on the connector with 2.5 V expect a current ~415 mA.
History | Date | Comments |
---|---|---|
2400011 | Oct/22-Feb/23 | Resistor R108 removed Oct/2022 to debug FELIX FEC12 mode problem. R108 back on Feb/2023. |
2400006,2400011 | May/2/2023 | bpol2V5 mezzanine R10 330KOhm→316KOhm to raise V1.2 to V1.24 to stablelize lpGBT3,4. Overall current 0.92->1.03A. |
2400006/4400069 | Nov/20/2023 | Brought in V4 board 4400069 in exchange of 2400006 to go back to Bern. |
Optobox/Board service connections:
Optobox LV/DCS services:
Check the above interfaces document for more detailled description.
Banana plugs are labelled according to powered bPOL12V.
Cooling and dry air inlet/outlet: FESTO 8 mm
Currently no MOPS is mounted on the Powerboard inside the test-Optopanel (due to availability). We will ship one as soon as we got our hands on them.
OptoBoard LV/DCS:
For optoboard standalone test setups, a mini LV/DCS service cable bundle is needed to supply 2.5V to optoboard and lead out its NTC monitoring out through the Samtec SFM-104-01-L-D connector .
Standalone optoboard power adaptor cable spec
Optoboard-FELIX Fiber Connectivity:
Each FELIX PCIe card has two MPO24 ports, serving 12 pairs of TX+RX each. Our lab test setups typically use fiber splitter bundles to breakout the MPO24 into individual LCs and use LC patch panels to map connectivity to the optoboard VTRX+ fiber pigtail. How the FELIX MPO24 port is structured is described in the FELIX readout and Direct FELIX confluence pages. There are two ways to breakout the FELIX MPO24:
- Most setups elsewhere and our easier setups all used MPO24→2xMPO12→24xLC splitters are described in the FELIX readout and Direct FELIX confluence pages.
- Our latest setups used direct MPO24→24xLC splitter in one step. The actual component inventory is in the "Fiber + patch" tab of the of test readout component Stanford Drive GoogleSheet.
How the VTRX+ fiber pigtail lines are structured is described in the Optosystem Interface document AT2-IP-GM-0010 (Table 4) with 5 out of 12 fibers active per optoboard which can be followed from the MPO12-LC splitter fiber color coding.
Connectors:
Check the above interfaces document for more detailled description.
Banana plugs are labelled according to powered bPOL12V.
Cooling and dry air inlet/outlet: FESTO 8 mm
DCS:
Currently no MOPS is mounted on the Powerboard inside the test-Optopanel (due to availability). We will ship one as soon as we got our hands on them.
SLAC Single Optoboard Test Setup
...
Note that the Optoboard VTRX+ fiber pigtail MT ferrules connection to MPO12-LC fiber splitter is a simple slip on which requires careful manual recognition of right polarity (see ITkPixV1 readout page) which can easily slip off but prevents accidental large force breaking the fiber. It is advisable to always check the link light level with flx-info podpower
to verify the fiber link is healthy (>700uW). How the VTRX+ fiber pigtail lines are structured is described in the Optosystem Interface document AT2-IP-GM-0010 (Table 4) with 5 out of 12 fibers active per optoboard. How the FELIX MPO24 port is structured and fanout with MPO24->2xMPO12→LC splitters are described in the FELIX readout and Direct FELIX confluence pagesaccidental large force breaking the fiber. It is advisable to always check the link light level with flx-info podpower
to verify the fiber link is healthy (>700uW).
Current firmware allows only 2 lpGBTs to be readout, such that the link alignment string is fixed to x0000000300000000 for channels 00 and 01 in unaligned state. Alignment can be checked using data from module RX 0 (L0 on miniDP-SMA) after module configuration, giving the following link alignment strings independently of which e-links are subscribed on felixcore startup (for optoboard 2400006):
...
Register | Value | Comments |
---|---|---|
dllEnable | True | Enable DLL clock for for retiming |
dllCapReset | False | |
dllForceDown | False | DLL can have harmonic lock problem which would need dllForceDown=T to release it |
dllChargePumpCurrent | 0x0 | Set charge injection current to min value. Higher current enables faster locking but may cause larger jitter. We care less about locking time but care more about jitter. |
dllClockDelay | scan | Scan this delay for the active channel for good BER window bathtub. |
Vakhtang's readout software
Vakhtang Tsiskaridze (VT) wrote a standalone python package to perform digital scans outside of YARR. Use the following steps to run on felix.
...
Code Block |
---|
# Starts interactive session where configuration and digital scans are steered python -i run_sendcmd.py>>> repower_init() # Quick power cycles the module and initializes with nominal configurations >>> full_scan() # Must be done 2+ times, see below for example of output >>> (1) below for example of outputfec2.write_field("SLDO_VTRIM", 0xaa) # Hacky solution that fixes readout issues caused by chip 2 >>> scan() |
The output of a digital scan should appear in data, and can be plotted using the plotting script plot_occmap_quad.py.
Code Block |
---|
# Output after first full_scan() [2023-12-04 16:40:44.526] [rd53b.decoder] [info] scan time: 3.77 sec, rate: 157.0 Mbps [2023-12-04 16:40:44.526] [rd53b.decoder] [info] tval: 100, Error in pixels: 29730. vmin: 95, vmax: 24743 [2023-12-04 16:40:44.526] [rd53b.decoder] [info] tag_stats: 25600 25600 25600 25600 25600 25600 25600 25600 25543 25425 25421 25398 25378 25349 25326 25297 [2023-12-04 16:40:44.526] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 30099829 1 0 0 0 0 0 0 [2023-12-04 16:40:44.526] [rd53b.dataproc.quad] [info] [2] SCAN STOP [2023-12-04 16:40:44.549] [rd53b.decoder] [info] scan time: 3.80 sec, rate: 155.6 Mbps [2023-12-04 16:40:44.549] [rd53b.decoder] [info] tval: 100, Error in pixels: 32604. vmin: 93, vmax: 24666 [2023-12-04 16:40:44.549] [rd53b.decoder] [info] tag_stats: 25600 25600 25601 25600 25600 25600 25600 25600 25466 25336 25307 25287 25264 25236 25216 25185 [2023-12-04 16:40:44.549] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 30012842 0 0 0 0 0 0 9 [2023-12-04 16:40:44.549] [rd53b.dataproc.quad] [info] [1] SCAN STOP [2023-12-04 16:40:44.572] [rd53b.decoder] [info] scan time: 3.82 sec, rate: 154.7 Mbps [2023-12-04 16:40:44.572] [rd53b.decoder] [info] tval: 99, Error in pixels: 125302. vmin: 91, vmax: 24659 [2023-12-04 16:40:44.572] [rd53b.decoder] [info] tag_stats: 25592 25592 25592 25594 25592 25594 25592 25593 25458 25224 25314 25293 25271 25232 25208 25179 [2023-12-04 16:40:44.572] [rd53b.decoder] [info] tag_hits: 0 0 0 1 2 1056 0 6 29930769 0 0 0 0 0 0 0 [2023-12-04 16:40:44.572] [rd53b.dataproc.quad] [info] [0] SCAN STOP [2023-12-04 16:40:44.594] [rd53b.decoder] [info] scan time: 3.84 sec, rate: 153.9 Mbps [2023-12-04 16:40:44.594] [rd53b.decoder] [info] tval: 100, Error in pixels: 31999. vmin: 93, vmax: 24668 [2023-12-04 16:40:44.594] [rd53b.decoder] [info] tag_stats: 25600 25600 25600 25600 25600 25600 25600 25600 25468 25341 25306 25289 25264 25230 25215 25184 [2023-12-04 16:40:44.594] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 30016538 0 0 0 0 0 0 0 # Output after setting VLDO_TRIM on fec2 [2023-12-04 16:53:59.413] [rd53b.decoder] [info] scan time: 3.74 sec, rate: 92.1 Mbps [2023-12-04 16:53:59.413] [rd53b.decoder] [info] tval: 100, All pixels are good. [2023-12-04 16:53:59.413] [rd53b.decoder] [info] tag_stats: 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 [2023-12-04 16:53:59.413] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 15360000 0 0 0 0 0 0 0 [2023-12-04 16:53:59.413] [rd53b.dataproc.quad] [info] [1] SCAN STOP [2023-12-04 16:53:59.436] [rd53b.decoder] [info] scan time: 3.76 sec, rate: 91.6 Mbps [2023-12-04 16:53:59.436] [rd53b.decoder] [info] tval: 100, All pixels are good. [2023-12-04 16:53:59.436] [rd53b.decoder] [info] tag_stats: 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 [2023-12-04 16:53:59.436] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 15360000 0 0 0 0 0 0 0 [2023-12-04 16:53:59.436] [rd53b.dataproc.quad] [info] [2] SCAN STOP [2023-12-04 16:53:59.458] [rd53b.decoder] [info] scan time: 3.78 sec, rate: 91.0 Mbps [2023-12-04 16:53:59.458] [rd53b.decoder] [info] tval: 100, All pixels are good. [2023-12-04 16:53:59.458] [rd53b.decoder] [info] tag_stats: 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 [2023-12-04 16:53:59.458] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 15360000 0 0 0 0 0 0 0 [2023-12-04 16:53:59.458] [rd53b.dataproc.quad] [info] [0] SCAN STOP [2023-12-04 16:53:59.481] [rd53b.decoder] [info] scan time: 3.80 sec, rate: 90.5 Mbps [2023-12-04 16:53:59.481] [rd53b.decoder] [info] tval: 100, All pixels are good. [2023-12-04 16:53:59.481] [rd53b.decoder] [info] tag_stats: 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 25600 [2023-12-04 16:53:59.481] [rd53b.decoder] [info] tag_hits: 0 0 0 0 0 0 0 0 15360000 0 0 0 0 0 0 0 |
Additional notes:
- 17 different triggers, on 9th trigger getting all hits
- Scan time (~4 seconds), rate is per chip
- Number of tag hits should be 400 * 384 * N_triggers
- If you run 100 triggers, scan time is dominated by mask stepping
Some Sme Useful Links
- Optoboard System Documentation
- FELIX JIRA (Oct/22) on optoboard + ITkPix readout setup
- CERN mattermost Bern-Optoboard channel
- Talk (Dec/9/2022) by Angira Rastogi on Optoboard-FELIX setup at LBNL
...