...

• user "systemctl list-unit-files" to see if tdetsim.service or kcu.service is enabled
• associated .service files are in /usr/lib/systemd/system/
• to see if events are flowing from the hardware to the software: "cat /proc/datadev_0" and watch "Tot Buffer Use" counter under "Read Buffers"
• if you see the error "rogue.GeneralError: AxiStreamDma::AxiStreamDma: General Error: Failed to map dma buffers. Increase vm map limit: sysctl -w vm.max_map_count=262144".  This can be caused by that parameter being too low, but it can also be caused by loading the driver with CfgMode=1 in tdetsim.service (we use CfgMode=2).  This CfgMode parameter has to do with the way memory gets cached.
  • make sure the tdetsim.service (or kcu.service) is the same as another working node
  • make sure that the appropriate service is enabled (see first point)
  • check that the driver in /usr/local/sbin/datadev.ko is the same as a working node
  • check that /etc/sysctl.conf has the correct value of vm.max_map_count
  • It seems that vm.max_map_count must be at least 4K (the page size = the MMU granularity, perhaps?) larger than the datadev service's cfgRxCount value, and in some cases much larger (doubled or more) than the cfgRxCount value if Rogue is also used in the executeable
    
  • we have also seen this error when the datadev.ko buffer sizes and counts were configured incorrectly in tdetsim.service.
  • have also seen this error running opal updateFebFpga.py because it loops over 4 lanes which increases the usage of vm.max_map_count by 4.  devGui and DAQ worked fine because they only used one lane.  worked around it by hacking the number of lanes in the script to 1.  Ryan Herbst is thinking about a better fix.
• For high rate running, many DMA buffers (cfgRxCount parameter) are needed to avoid deadtime. The number of DMA buffers is also used to size the DRPs' pebble, so slow but large DRPs like PvaDetector and EpicsArch need a comparatively small cfgRxCount value.  Neglecting to take this into account can result in attempting to allocate more memory than the machine has, resulting in the DRP throwing std::bad_alloc.
  
  • The number of buffers returned by dmaMapDma() is the sum of the cfgTxCount and the cfgRxCount values.  The DRP code rounds this up to the next power of 2 (if it's not already a power of 2) and uses the result to allocate the number of pebble buffers.  To avoid wasting up to half the allocation, set the sum of cfgTxCount and cfgRxCount to a power of 2.
• DMA buffers can be small (cfgSize parameter) for most (all?) tdetsim applications.  16 KB is usually sufficient.

...

Roughly, if a DRP chain is stalled for some reason, the DMA buffers will be consummed at the trigger rate.  So in the above example, the HSDs will start back pressuring into firmware after roughly 1 second given a trigger rate of 1 MHz.  For a given trigger rate, there is no clear benefit to having one DRP have more or less DMA buffers than another.  The first one to run out of buffers will cause backpressure and ultimately inhibit triggers, leaving additional buffers on other DRPs inaccessible.  Thus, I suggest making the number of DMA buffers (cfgRxCount + 4cfgTxCount) the same for each DRP in a given readout group and to roughly keep the cfgRxCounts in the same ratio as the trigger rates of the groups (while still following the 2**N - 4 cfgTxCount rule of above).

taskset

Ric does "taskset -c 4-63" for daq executables to avoid the cores where weka processes are running

...

If there is significant deadtime coming from a node hosting multiple DRPs at high trigger rate and kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0] messages appear in dmesg, etc., it may be that the interrupts are being handled for both datadev devices by one core, usually CPU0.  To avoid this, the interrupt handling can be moved to two different cores, e.g., CPU4 and 5 (to avoid the Weka cores in SRCF). 

First, determine the IRQ(s) that have been allocated for the KCU datadev device(s), e.g., with:

claus@drp-srcf-cmp024:~$ dmesg -T | grep datadev | grep IRQ
[Fri Oct 21 08:32:59 2022] datadev 0000:41:00.0: Init: IRQ 369
[Fri Oct 21 08:32:59 2022] datadev 0000:42:00.0: Init: IRQ 370

When there is just one datadev device, we see:

claus@drp-srcf-cmp007:~$ dmesg -T | grep datadev | grep IRQ
[Wed Nov 16 14:38:30 2022] datadev 0000:41:00.0: Init: IRQ 368

This can be done automatically in the tdetsim/kcu.service files with the lines:

ExecStartPost=-/usr/bin/sh -c "/usr/bin/echo 4 > /proc/irq/`grep datadev_0 /proc/interrupts | /usr/bin/cut -d : -f 1 | /usr/bin/tr -cd [:digit:]`/smp_affinity_list"
ExecStartPost=-/usr/bin/sh -c "/usr/bin/echo 5 > /proc/irq/`grep datadev_1 /proc/interrupts | /usr/bin/cut -d : -f 1 | /usr/bin/tr -cd [:digit:]`/smp_affinity_list"

The irqbalance service will override the above unless the service is disabled or it is told to avoid the datadev devices' IRQs.  Unfortunately I haven't found a way to do that automatically, so for now we Next, modify /etc/sysconfig/irqbalance to avoid the datadev's IRQs:

IRQBALANCE_ARGS=--banirq=368 --banirq=369 --banirq=370

All 3 IRQs we typically see in SRCF are listed so that the irqbalance file can be used on all SRCF machines without modification. 

Finally, adding the following to the kcu or tdetsim service file sets the interrupt affinity for IRQs 369 and 370 to cores 4 and 5, respectively:

ExecStartPost=/usr/bin/sh -c "/usr/bin/echo 4 > /proc/irq/369/smp_affinity_list"
ExecStartPost=/usr/bin/sh -c "/usr/bin/echo 5 > /proc/irq/370/smp_affinity_list"

It's unclear to me how the IRQ numbers are assigned, so we may need to consider the possibility that they can change from system (or driver?) restart to restart or be different from node to node.  However, I've seen no evidence that this is the caseNote that if another device is added to the PCI bus, the IRQ values may be different.

HSD

General Debugging

• look at configured parameters using (for example) "hsdpva DAQ:LAB2:HSD:DEV06_3D:A"
• for kcu firmware that is built to use both QSFP links, the naming of the qsfp's is swapped.  i.e. the qsfp that is normally called /dev/datadev_0 is now called /dev/datadev_1
• HSD is not configured to do anything (Check the HSD config tab for no channels enabled)
• if hsd timing frames are not being received at 929kHz (status here), click TxLink Reset in XPM window.  Typically when this is an issue the receiving rate is ~20kHz.
• The HSD readoutGroup number does not match platform number in .cnf file (Check the HSD "Config" tab)
• also check that HEADERCNTL0 is incrementing in "Timing" tab of HSD cfg window.
• in hsd Timing tab timpausecnt is number of clocks we are dead (156.25MHz clock ticks).  dead-time fraction is timpausecnt/156.25e6
• in hsd expert window "full threshold(events)" sets threshold for hsd deadtime
• in hsd Buffer tab "fex free events" and "raw free events" are the current free events.
• in hsd status window "write fifo count" is number of timing headers waiting for HSD data to associate.
• "readcntsum" on hsd timing tab goes up when we send a transition OR L1Accepts. "trigcntsum" counts L1Accepts only.
• check kcuStatus for "locPause" non-zero (a low level pgp FIFO being full).  If this happens then:  configure hsd, clear readout, reboot drp node with KCU
• if links aren't locking in hsdpva use "kcuStatus" to check that the tx/rx clock frequencies are 156MHz.  If not (we have seen lower rates like 135MHz) a node power cycle (to reload the KCU FPGA) can fix this.  Matt writes: "kcuStatus should have an option to reset the clock to its factory input before attempting to program it to the standard input."
• If the drp doesn't complete rollcall and the log file shows messages about PADDR_U being zero, restarting the corresponding hsdioc process may help.

...