Page History
...
These are the results of the tests that have been conducted in the FEE alcove to determine if the XPM glitch can be reproduced.
Every test is run from a starting behavior where the DAQ can allocate, configure, run, and disable.
Whenever the DAQ does not follow the starting behavior remedies are applied to recover it.
XPM firmware 3.6.0
Opal config does not have xpm mini -timing2 hack
cnf file uses -D fakecam for additional timing nodes
xpm10 and 11 connections
XPM schematics
in 2023/11/14
...
remove fiber from xpm10 to xpm11 fiber 10 times
for 5 seconds (amc0 port1)
...
Removing fiber from xpm10 to timing 1 fiber 10 times for 5 seconds (Amc1 port0)
...
Removing fiber from xpm11 to opal fiber 10 times for 5 seconds (Amc1 port1)
...
in 2023/10/30
...
stop pyxpm 10 and 11
fru-deactivate and activate xpm 10
restart pyxpm 10 and 11
start DAQ
...
stop pyxpm 10 and 11
fru-deactivate and activate xpm 10
restart pyxpm 10 and 11
start DAQ
...
reboot timing nodes
...
stop pyxpm 10 and 11
fru-deactivate and activate xpm 10
restart pyxpm 10 and 11
start DAQ
...
ctrl-x in the terminal successfully restart them
example of the timing shift in the timing nodes (before -D fakecam).
XPM firmware 3.5.4
Opal_config.py still has xpm mini – timing2 hack
In 2023/10/26:
...
stop pyxpm 10 and 11
fru-deactivate and activate xpm 11
restart pyxpm 10 and 11
start DAQ
...
stop pyxpm 10 and 11
fru-deactivate and activate xpm 10
restart pyxpm 10 and 11
start DAQ
...
...
stop pyxpm 10 and 11
fru deactivate activate 10
fru deactivate activate 11 (in order)
devGui switch xpm mini/timing2
if needed restart opal from terminal
Observation Of Front-Panel XPM Link Glitch With Version 3.5.4
Perhaps fixed by Matt in later firmware version?
XPM11 glitches between 4pm and after 6pm and also around 10:10 am the next day
2023/10/27: updating firmware
xpm11 to xpm_noRTM-0x030601000-20231011111938-weaver-645bee8.mcs
xpm10 to xpm-0x030601000-20231011111954-weaver-645bee8.mcs
Testing Details
In 2023/10/24 :
| action | result | remedy | result |
|---|---|---|---|
| Remove XPM10 fiber timing in the back while DAQ running | *** XpmDetector: timing link ID is ffffffff = 4294967295^M | TxlinkReset of cmp015 in XPM11 | DAQ recovers |
| Repeat XPM10 fiber timing removal removal | DAQ cannot disable | --- | DAQ recovers by itself at restart |
| Repeat XPM10 fiber timing removal removal | --- | --- | no issue |
| Repeat XPM10 fiber timing removal removal | DAQ cannot disable | --- | DAQ recovers by itself at restart |
| Remove XPM10 fiber timing in the back while DAQ stopped | --- | --- | DAQ starts with no issue |
| Repeat XPM10 fiber timing removal removal while DAQ stopped | --- | --- | DAQ starts with no issue |
| Remove transceiver from XPM10 in the back (DAQ stopped) | --- | --- | DAQ starts with no issue |
| Remove transceiver from XPM10 in the back (DAQ started) | --- | --- | DAQ starts with no issue |
timing 1 shutsdown by itself | TXlinkReset on XPM10 for XPM11 | DAQ recovers | |
| Remove fiber on XPM10 to XPM11 | --- | --- | DAQ starts with no issue |
| Remove transceiver on XPM10 to XPM11 | --- | --- | DAQ starts with no issue |
| Remove fiber on XPM11 AMC0 port 0 | --- | --- | DAQ starts with no issue |
| Remove transceiver on XPM11 AMC 0 port0 | --- | --- | DAQ starts with no issue |
opal disappears from the list f detectors | restart DAQ | DAQ starts with no issue | |
| power cycle xpm10 via switch only AMC0 | XPM 11 looses timing node | Restart pyxpm 10 and 11 restart pyxpm 11 | DAQ restarts but opal shutsdown |
opal still shutdown | devGui xpmmini timing v2 | no avail | |
Stop pyxpm 10 and 11 | DAQ starts with no issue |
In 2023/10/26:
| action | issue found | error stat | remedy |
|---|---|---|---|
stop pyxpm 10 and 11 | no issue has been detected | 0/10 | --- |
stop pyxpm 10 and 11 | at first xpmpva DAQ:NEH:XPM:11 does not come up Then Opal shutsdown | 3/20 | stop pyxpm 10 and 11 |
Observation Of Front-Panel XPM Link Glitch With Version 3.5.4
Perhaps fixed by Matt in later firmware version?
XPM11 glitches between 4pm and after 6pm and also around 10:10 am the next day
2023/10/27: updating firmware
xpm11 to xpm_noRTM-0x030601000-20231011111938-weaver-645bee8.mcs
xpm10 to xpm-0x030601000-20231011111954-weaver-645bee8.mcs
XPM firmware 3.6.0
Opal config does not have xpm mini -timing2 hack
cnf file uses -D fakecam for additional timing nodes
in 2023/10/30
| action | issue found | stat | remedy |
|---|---|---|---|
stop pyxpm 10 and 11 | 9/20 | --- | |
stop pyxpm 10 and 11 | Opal fails in configuration | 5/20 | reboot timing nodes |
stop pyxpm 10 and 11 | groupca and xpmpva are shutdown at startup | 4/20 | ctrl-x in the terminal successfully restart them |
example of the timing shift in the timing nodes (before -D fakecam).
XPM firmware 3.5.4
Opal_config.py still has xpm mini – timing2 hack
in 2023/11/14
| action | issue found | stat | remedy |
|---|---|---|---|
| stop pyxpm 10 and 11 fru-deactivate and activate xpm 11 restart pyxpm 10 and 11 start DAQ | bucket issue | 1/10 | rebooting timing node cmp001 |
| rebooting timing node cmp001 | no issue | 0/5 | |
remove fiber from xpm10 to xpm11 fiber 10 times | no issue | 0/10 | |
Removing fiber from xpm10 to timing 1 fiber 10 times for 5 seconds (Amc1 port0) | no issue | 0/10 | |
Removing fiber from xpm11 to opal fiber 10 times for 5 seconds (Amc1 port1) | no issue | 0/10 |
in 2023/11/17
New opal_config.py: remove sleep introduce while
test power cycle she-fee-daq01/2 10 times
1 bucket issue - timing1
try txlinkreset (timing AND opal) — no avail
Try reboot cmp001 — NO AVAIL
Try reboot cmp005 (opal) — NO AVAIL
Power cycle xpm10 solved the issue
2 bucket issue - timing1 and 2
Try power cycle xmp10 solved the issue
3 no issue
4 no issue
5 no issue
6 no issue
7 bucket issue timing1 and timing 2
power cycle XPM no avail
power cycle XPM solved the issue
8 no issue
9 no issue
40 minutes RTM disconnected in XPM10
No issue
5 minutes
no issue
10 minutes
No issue
2 hours
Every timing shutdown
Opal too
permission denied issue
Try restart pypxmps naval
Try txlinkreset no avail
try power cycle xpm10 no avail
try power cycle xpm11
recompile daq
bucket issue
power cycle xpm10
worked!
Conclusion
It appears that yanking the timing fiber can cause disturbances in the system, but they are not repeatable 100% of the time.
XPMs Power spikes can set the DAQ in a behavior similar to the XPM glitch, but only if pyxpms are running. To be repeated.
Upgrading XPM firmware seems to have mitigated all the issues (to 3.6.0 from 3.5.4). The bucket issue becomes more prominent, probably because other issues are not happening. This issue appears when power cycling the xpm11.
...
Overview
Content Tools