...

• eye-scans for all transceivers
• work on high-speed-digitizer timing robustness using teststand
  • occasional need to restart hsdioc process
  • kcu1500 can lose link and hsd loses/regains power, and can only be recovered by power cycling cmp node
• check wave8 timing robustness
• (almost done) program hsd firmware over pcie?
• manufacture new xpm boards (4 for txi)
  • do we need another xpm/crate order for mfx? (separate from LCLS-II-HE?).  go from mfx hutch back to 208 or the mezzanine?
• reproduce/fix timing nodes assigning wrong timestamp to configure transition by 1 or 2 buckets
  • matt thinks this is on the receiver side: some fifos that carry daq data separate from timing data.  matt thinks perhaps we have to connect the resets to those fifos.
  • have seen this is hsd/wave8.  see both being problematic after a power outage here: /cds/home/opr/tmoopr/2024/03/04_17:11:56_drp-srcf-cmp030:teb0.log (and Riccardo saw it in his tests, below)
• (perhaps done by fixing reset logic?) reproduce/fix link-lock failure on timing system KCUs
• make pyxpm processes robust to timing outages?
• (done) ensure that Matt's latest xpm firmware fixes the xpm link-glitch storms
• (perhaps done by fixing reset logic ?) reproduce/fix TxLinkReset workaround
• (perhaps done by fixing reset logic?) reproduce/fix xpmmini-to-lcls2timing workaround
• (done, fixed with equalizer 0x3 setting) check/fix loopback fiber problem in production xpms in room 208 
• this file shows a failure mode of a tdet kcu1500 on drp-srcf-cmp010 where its timestamps were off by one clock-tick: teb log file showing the cmp010 problem: /cds/home/opr/rixopr/scripts/logfiles/2024/04/08_11:58:28_drp-srcf-cmp013:teb0.log.  Powercycling "fixed" the problem.  Split event output from that log (two Andor's on cmp010):

rix-teb[2111]: <W> Fixup Configure, 008a4a15bf8ff2, size 0, source 0 (andor_norm_0)
rix-teb[2111]: <W> Fixup Configure, 008a4a15bf8ff2, size 0, source 1 (andor_dir_0)
rix-teb[2111]: <W> Fixup Configure, 008a4a15bf8ff3, size 0, source 2 (manta_0)
rix-teb[2111]: <W> Fixup Configure, 008a4a15bf8ff3, size 0, source 3 (mono_encoder_0)

Summary Of Testing

These are the results of the tests that have been conducted in the FEE alcove to determine if the XPM glitch can be reproduced.
Every test is run from a starting behavior where the DAQ can allocate, configure, run, and disable.
Whenever the DAQ does not follow the starting behavior remedies are applied to recover it.

...