...

• Jeremy and CPO are on vacation
• Chris and I have set up daq-tst-dev06 in Lab 3 for testing with a GPU & KCU
  • The KCU appears as /dev/datagpu_1
  • After some debugging to get the software to recognize /dev/datagpu_1 (as opposed to _0) the interCardGui comes up and shows sensible values
  • test_dma also runs, but the AxiVersion.ScratchPad register does not go from 0x0 to 0xdeadbeef
    • Neither does test_dma see any DMAs
• Larry urges to move axi-streampcie-drivers devel forward to v4.1.0 rather than working with v4.0.0
  • Mudit has created v4.2.0 (CPU/GPU DMA switch?) but it has not been tested with hardware yet
  • It is unclear whether Jeremy has gotten a baseline working system yet
• Ryan agreed that my hacks of test_dma.cu and _Root.py to target/dev/datagpu_1 should be sufficient
• Ryan would like us all to work on one machine and get confidence in it before we branch out to different machines
• I suggested that we could pull out the current /dev/datagpu_0 from dev06 so that the KCU of interest becomes _0
  • Ryan & Larry suggested pullling the other ones out as well
• Ryan suggests setting iommu=no in the BIOS as well as on the command line
  • I later found that there is no iommu parameter in dev06's BIOS
• There's no experience with multiple KCU cards and the datagpu driver - it is untested
  • Larry is aware of some issue with the usual datadev driver and multiple KCUs
• Work on rdsrv419
  • A GPU is installed
  • CUDA is installed
  • A KCU is installed and has PCIe slot 03:00

July 15, 2024

• CPO and Ryan on vacation
• Let's wait with contacting NVIDIA without GPUDirect issues until Chris gets back

• Ric to look into the BIOS settings again to see if we can glean something

• Jeremy is finding that when he executes KCU register reads with various methods, he always gets back the Version value from register offset 0

  • So far, he's baffled