Page History

...

Note that the datadev driver can handle multiple buffers (O(1000)) per interrupt, so the IRQ rate tends to fall well below the rate set by the IRQ HOLD / cfgIrqHold value at high trigger rates.  At 1 MHz trigger rate, the IRQ rate is a few hundred Hz.

Setting interrupt affinity to avoid 'BUG: Soft lockup'

...

notifications

If there is significant deadtime coming from a node hosting multiple DRPs at high trigger rate and kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0] messages appear in dmesg, etc., it may be that the interrupts are being handled for both datadev devices by one core, usually CPU0.  To avoid this, the interrupt handling can be moved to two different cores, e.g., CPU4 and 5 (to avoid the Weka cores in SRCF). 

First, disable the irqbalance service and then set the IRQ affinity values (here datadev IRQs 369 and 370, as found from dmesg):

sudo sh -c "echo 4 > /proc/irq/369/smp_affinity_list"
sudo sh -c "echo 5 > /proc/irq/370/smp_affinity_list"

determine the IRQ(s) that have been allocated for the KCU datadev device(s), e.g., with:

claus@drp-srcf-cmp024:~$ dmesg -T | grep datadev | grep IRQ
[Fri Oct 21 08:32:59 2022] datadev 0000:41:00.0: Init: IRQ 369
[Fri Oct 21 08:32:59 2022] datadev 0000:42:00.0: Init: IRQ 370

When there is just one datadev device, we see:

claus@drp-srcf-cmp007:~$ dmesg -T | grep datadev | grep IRQ
[Wed Nov 16 14:38:30 2022] datadev 0000:41:00.0: Init: IRQ 368

Next, For a more permanent solution, it may be better to modify /etc/sysconfig/irqbalance to avoid the datadev's IRQs:

IRQBALANCE_ARGS=--banirq=368 --banirq=369 --banirq=370

All 3 IRQs we typically see are listed so that the irqbalance file can be used on all machines without modification. 

Finally, adding Add the following to the kcu or tdetsim service file , e.g.sets the interrupt affinity for IRQs 369 and 370 to cores 4 and 5, respectively:

ExecStartPost=/usr/bin/sh -c "/usr/bin/echo 4 > /proc/irq/369/smp_affinity_list"
ExecStartPost=/usr/bin/sh -c "/usr/bin/echo 5 > /proc/irq/370/smp_affinity_list"Note that some KCU firmwares provide only one datadev device.  When this is the case, the associated IRQ is 368.

It's unclear to me how the IRQ numbers are assigned, so we may need to consider the possibility that they can change from system (or driver?) restart to restart or be different from node to node.  However, I've seen no evidence that this is the case.

...