...

Diagram of system from conversation with Quincey Koziol and Rebanta Mitra on March 20th, 2024

NVLink

Update about putting SLAC’s FPGA’s on the NVLink bus, from someone at NVIDIA who’s close to the NVLink work:  

- It's not possible today
- Or better said, it would be very hard today. :-)
- It could be possible to connect them to the Arm C2C link, which speaks the standard Arm CHI protocol.
- NVLink is a multiplanar network. You would need to connect all of the FPGAs to all 18 planes of the network because the GPU does an address based spray across the planes.

    In that direction, here’s info about NVLink-C2C (which is what I believe that he was referring to):   https://www.nvidia.com/en-us/data-center/nvlink-c2c/ and I think this quote from that page is relevant:

"Supports Arm’s AMBA CHI (Coherent Hub Interface) or Compute Express Link (CXL) industry standard protocols for interoperability between devices.”

GPU Direct Storage

a.k.a GDS.  Supported by weka: https://docs.nvidia.com/gpudirect-storage/troubleshooting-guide/index.html#gds-config-file-changes

datadev Driver Meeting With TID

On May 1, 2024

Slides from Ryan and Jeremy: https://docs.google.com/presentation/d/1yJ-WIs73lon9LzBYxIhKKyNvoYWAq8OUNDiN_TOD2Pw/edit?usp=sharing

Useful docs and links

• CUDA documentation root
• CUDA C Programming Guide
• CUDA Samples
• CUDA Runtime API
• CUDA Driver API
• GPUDirect RDMA
• GPUs installed in drp-srcf-gpu001-3: NVIDIA RTX A5000 (Ampere architecture, compute capability 8.6)
• An Open Source General Purpose DMA Engine For DAQ Systems (Ryan Herbst)
• axi-pcie-devel repo
• axi-pcie-CORE repo