Observed: some of the XPM links does not come up or get a huge amount of errors.
Theory: high speed repeater, located on the AMC card, which repeat (clean) the link from/to SFP/FPGA, have bad settings.
Proposal: Scan of the equalizer settings in order to find the best configuration (data received from detector)
Script: https://github.com/slac-lcls/lcls2/blob/master/psdaq/psdaq/pyxpm/pyxpm_hsrepeater.py
Summary of Results
From Julian on Feb 28, 2024:
From all of the measurement we did, the following values never showed any errors: [0x00, 0x02, 0x03, 0x07, 0x0b, 0x0f]. As we saw error once on 0x01, I would suggest to avoid 0x00 and 0x02 which are close from it. Additionally, as upper the value is, more we got errors, I would suggest avoiding 0x0b and 0x0f. Finally, 0x15 showed errors as well, I would go with 0x03
Tests result
In order to validate the theory, the script was tested on XPM10, XPM6 and XPM5. First goal was to fix non working links on XPM6 (AMC1 port 3, 4 and 5)
Scan equalizer settings for link 11 / XPM 6
Link[11] status (eq=00): Ready (Rec: 242973 - Err: 38)
Link[11] status (eq=01): Ready (Rec: 20635028 - Err: 0)
Link[11] status (eq=02): Ready (Rec: 20635028 - Err: 0)
Link[11] status (eq=03): Ready (Rec: 20635028 - Err: 0)
Link[11] status (eq=07): Ready (Rec: 20635029 - Err: 0)
Link[11] status (eq=15): Not ready (Rec: 0 - Err: 4294966931)
Link[11] status (eq=0B): Ready (Rec: 20635029 - Err: 0)
Link[11] status (eq=0F): Ready (Rec: 20635028 - Err: 0)
Link[11] status (eq=55): Not ready (Rec: 0 - Err: 17872)
Link[11] status (eq=1F): Not ready (Rec: 0 - Err: 7686)
Link[11] status (eq=2F): Not ready (Rec: 0 - Err: 4294964401)
Link[11] status (eq=3F): Not ready (Rec: 0 - Err: 4294958545)
Link[11] status (eq=AA): Not ready (Rec: 0 - Err: 13246)
Link[11] status (eq=7F): Not ready (Rec: 0 - Err: 4294940893)
Link[11] status (eq=BF): Not ready (Rec: 0 - Err: 4294950259)
Link[11] status (eq=FF): Not ready (Rec: 0 - Err: 4294922138)
[Configured] Set eq = 0x01
The scan reports errors for equalizer value of 0x2F (default value), which is a good sign. By moving to other values, we started getting data and link ready up. Therefore, additional tests, on different links/XPM have been made to find a common working point:
| Equalizer | XPM6 link 12 | XPM6 link 11 | XPM6 link 10 | XPM5 link 13 | XPM5 link 12 | XPM6 link 13 | XPM10 link 7 |
|---|---|---|---|---|---|---|---|
| 0x00 | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| 0x01 | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | |
| 0x02 | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| 0x03 | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| 0x07 | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| 0x15 | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | |
| 0x0B | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| 0x0F | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| 0x55 | ✔️ | ||||||
| 0x1F | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ||
| 0x2F | ✔️ | ✔️ | ✔️ | ✔️ | |||
| 0x3F | ✔️ | ✔️ | ✔️ | ||||
| 0xAA | ✔️ | ||||||
| 0x7F | |||||||
| 0xBF | |||||||
| 0xFF |
Low values sound to be the best settings, even if it does not provide a 100% guarantee. Setting an initial value of 0x03 with a way to execute a scan in case of issue sounds to be a good deal. Additional tests were carried out to ensure that the configuration is consistent over scans and hard reboot. The scans was run 3 times in a row for XPM 13, link 6:
Run 1 errors:
- Link[13] status (eq=55): Not ready (Rec: 4294966397 - Err: 18172)
- Link[13] status (eq=AA): Ready (Rec: 4294945582 - Err: 45710) - Link[13] status (eq=7F): Not ready (Rec: 4294967293 - Err: 9120) - Link[13] status (eq=BF): Not ready (Rec: 0 - Err: 4294966592) - Link[13] status (eq=FF): Not ready (Rec: 0 - Err: 25655)
Run 2 errors:
- Link[13] status (eq=55): Ready (Rec: 4294966484 - Err: 13768)
- Link[13] status (eq=AA): Ready (Rec: 4294966334 - Err: 4294947967) - Link[13] status (eq=7F): Ready (Rec: 4294963255 - Err: 8614) - Link[13] status (eq=BF): Not ready (Rec: 0 - Err: 4294963111) - Link[13] status (eq=FF): Not ready (Rec: 0 - Err: 24541)
Run 3 errors:
- Link[13] status (eq=55): Ready (Rec: 1756 - Err: 16360)
- Link[13] status (eq=AA): Not ready (Rec: 4294966637 - Err: 46671) - Link[13] status (eq=7F): Ready (Rec: 2806 - Err: 10299) - Link[13] status (eq=BF): Not ready (Rec: 0 - Err: 2857) - Link[13] status (eq=FF): Not ready (Rec: 0 - Err: 23747)
Results from this test shows a consistent behavior over scans. Final test consisted in looking at XPM10 / link 7 before and after a hard reset:
Before:
- Link[7] status (eq=01): Ready (Rec: 4294371331 - Err: 65) - Link[7] status (eq=7F): Ready (Rec: 4294964175 - Err: 6412) - Link[7] status (eq=BF): Not ready (Rec: 0 - Err: 25808) - Link[7] status (eq=FF): Not ready (Rec: 0 - Err: 53525)
After:
- Link[7] status (eq=7F): Not ready (Rec: 27406 - Err: 4294944911) - Link[7] status (eq=BF): Not ready (Rec: 4294967292 - Err: 4469) - Link[7] status (eq=FF): Not ready (Rec: 0 - Err: 14706)
Configuration showed a small amount of errors with setting 0x01 before being rebooted but disappeared after. However, according to the really small number, this difference is negligible and can be ignored.
Summary
Setting good valued to the high speed repeater equalizer fix the issue. Nevertheless, it is difficult to guarantee a unique value that will guarantee a working link. However, the default one (0x2F) sounds to be one of the worst we could set according to the statistics we got. Therefore, it is recommended to initialize the repeater with 0x03 and run the scan script to optimize it in case of link issue.
ToDo list
Additional tests that could help getting even better idea of link quality issue:
- Try plugging in a couple of different detectors (and different fiber lengths) into a few of the xpm6 ports to get a feeling for how much variation there is in the equalizer settings. This would address my question of whether a small piece of dust in the wave8 fiber path could be affecting the settings.
Test different detectors and loop-back cable
Use the same xpm port (XPM 6 port 13) to test:
- TXI-wave8 (XPM 6 AMC 13)
- TMO-Piranha-cmp013 (XPM 6 AMC 8)
- TDetSim-cmp001 (XPM 6 AMC 1)
- Loop-back
TXI-wave8
Scan equalizer settings for link 13 / XPM 6
Link[13] status (eq=00): Ready (Rec: 20635023 - Err: 0)
Link[13] status (eq=01): Ready (Rec: 20635022 - Err: 0)
Link[13] status (eq=02): Ready (Rec: 20635022 - Err: 0)
Link[13] status (eq=03): Ready (Rec: 20635022 - Err: 0)
Link[13] status (eq=07): Ready (Rec: 20635022 - Err: 0)
Link[13] status (eq=15): Ready (Rec: 20635023 - Err: 0)
Link[13] status (eq=0B): Ready (Rec: 20635022 - Err: 0)
Link[13] status (eq=0F): Ready (Rec: 20635022 - Err: 0)
Link[13] status (eq=55): Ready (Rec: 4294963140 - Err: 59621)
Link[13] status (eq=1F): Ready (Rec: 4294955167 - Err: 4834)
Link[13] status (eq=2F): Ready (Rec: 20635022 - Err: 0)
Link[13] status (eq=3F): Ready (Rec: 20635022 - Err: 0)
Link[13] status (eq=AA): Ready (Rec: 20635022 - Err: 0)
Link[13] status (eq=7F): Ready (Rec: 7039 - Err: 656)
Link[13] status (eq=BF): Not ready (Rec: 0 - Err: 5874)
Link[13] status (eq=FF): Not ready (Rec: 0 - Err: 4294936130)
[Configured] Set eq = 0x00
Piranha4
Scan equalizer settings for link 13 / XPM 6
Link[13] status (eq=00): Ready (Rec: 20635023 - Err: 0)
Link[13] status (eq=01): Ready (Rec: 20635023 - Err: 0)
Link[13] status (eq=02): Ready (Rec: 20635023 - Err: 0)
Link[13] status (eq=03): Ready (Rec: 20635023 - Err: 0)
Link[13] status (eq=07): Ready (Rec: 20635024 - Err: 0)
Link[13] status (eq=15): Ready (Rec: 20635023 - Err: 0)
Link[13] status (eq=0B): Ready (Rec: 20635023 - Err: 0)
Link[13] status (eq=0F): Ready (Rec: 20635023 - Err: 0)
Link[13] status (eq=55): Ready (Rec: 4036 - Err: 7263)
Link[13] status (eq=1F): Ready (Rec: 20635023 - Err: 0)
Link[13] status (eq=2F): Ready (Rec: 20635023 - Err: 0)
Link[13] status (eq=3F): Ready (Rec: 4294855814 - Err: 815)
Link[13] status (eq=AA): Not ready (Rec: 4294967126 - Err: 4294947187)
Link[13] status (eq=7F): Not ready (Rec: 0 - Err: 54563)
Link[13] status (eq=BF): Not ready (Rec: 0 - Err: 4294948881)
Link[13] status (eq=FF): Not ready (Rec: 0 - Err: 25433)
[Configured] Set eq = 0x00
Timing cmp001
Scan equalizer settings for link 13 / XPM 6
Link[13] status (eq=00): Ready (Rec: 20635057 - Err: 0)
Link[13] status (eq=01): Ready (Rec: 20635057 - Err: 0)
Link[13] status (eq=02): Ready (Rec: 20635057 - Err: 0)
Link[13] status (eq=03): Ready (Rec: 20635056 - Err: 0)
Link[13] status (eq=07): Ready (Rec: 20635057 - Err: 0)
Link[13] status (eq=15): Ready (Rec: 20635057 - Err: 0)
Link[13] status (eq=0B): Ready (Rec: 20635057 - Err: 0)
Link[13] status (eq=0F): Ready (Rec: 20635057 - Err: 0)
Link[13] status (eq=55): Ready (Rec: 4294937885 - Err: 377)
Link[13] status (eq=1F): Ready (Rec: 20635057 - Err: 0)
Link[13] status (eq=2F): Ready (Rec: 20635057 - Err: 0)
Link[13] status (eq=3F): Ready (Rec: 20635057 - Err: 0)
Link[13] status (eq=AA): Ready (Rec: 835 - Err: 3163)
Link[13] status (eq=7F): Ready (Rec: 231 - Err: 4038)
Link[13] status (eq=BF): Not ready (Rec: 0 - Err: 3353)
Link[13] status (eq=FF): Not ready (Rec: 0 - Err: 5149)
[Configured] Set eq = 0x00
Loopback
Scan equalizer settings for link 13 / XPM 6
Link[13] status (eq=00): Ready (Rec: 928583 - Err: 0)
Link[13] status (eq=01): Ready (Rec: 928583 - Err: 0)
Link[13] status (eq=02): Ready (Rec: 928584 - Err: 0)
Link[13] status (eq=03): Ready (Rec: 928583 - Err: 0)
Link[13] status (eq=07): Ready (Rec: 928583 - Err: 0)
Link[13] status (eq=15): Ready (Rec: 928583 - Err: 0)
Link[13] status (eq=0B): Ready (Rec: 928583 - Err: 0)
Link[13] status (eq=0F): Ready (Rec: 928583 - Err: 0)
Link[13] status (eq=55): Ready (Rec: 77 - Err: 24357)
Link[13] status (eq=1F): Ready (Rec: 928583 - Err: 0)
Link[13] status (eq=2F): Ready (Rec: 928583 - Err: 0)
Link[13] status (eq=3F): Ready (Rec: 928584 - Err: 0)
Link[13] status (eq=AA): Ready (Rec: 4294966695 - Err: 972)
Link[13] status (eq=7F): Ready (Rec: 4294967207 - Err: 4294918729)
Link[13] status (eq=BF): Not ready (Rec: 0 - Err: 12824)
Link[13] status (eq=FF): Not ready (Rec: 0 - Err: 4294966149)
[Configured] Set eq = 0x00
Overview
Content Tools