To Do List

With Stefano and Ric (Gabriel was absent) to try to set up a roadmap of future projects.

Priorities:

• get first-pass TDet firmware from Mudit
• drp-executable infrastructure?
  • makefile support (Gabriel?).  Ric feels this will be fairly straightforward
    • switch to spack (Gabriel?).  our make system may depend on conda.  Gabriel thinks switch to gpu/spack may be straightforward.
  • gpu-direct to weka (gabriel?)
  • dgram creation/handling not optimal for gpu? use cpu? (ric)
  • reverse BEB-packet iteration not optimal for gpu? (low priority)
  • slow transitions (e.g. configure) go to cpu, fast transitions (l1accept, slowupdate) go to gpu (ric)
  • have to make the teb/meb work with "trigger primitives" (ric)
    • has to be on the CPU to communicate with IB via libfabric
    • do need the meb buffer-number stuff to work "soon-ish"
    • full data needs to go from GPU to CPU for meb purposes on a fraction of the L1s

algorithms? (stefano)

• managing calibration constants on the GPU
  • cpu has access to these (for the portion of the detector that the drp node sees) in psana via drp-python (as a python dictionary in det.calibconst)
  • need to "copy the python dictionary" with cudaMemCopy to the GPU: pedestal/gain/bad-pixels.  have to figure out how to do this elegantly.  constants can change at every beginrun transition.
  • the dictionary is detector-specific.
  • could be a std::map?  Ric says that std::map may not be implemented on GPU.
• LC, cuSZ (work on cpu decompression in psana)
• ROI (trivial)
• others like radial average with masks (standalone and pyfai)
• cube
• peakfinders (GPU peakfinder8 is in pyFAI)

LC performance

Update May 29: Upon further analysis (correcting for an error in the data saving format), for LC-GPU using A5000 compression speeds for segments of (352x384) pixels is:

May 14 and 20, 2024

• Stefano showed 68GB/s LC compression speed with 7.5x compression (raw uint16 vs. compressed float32) for 1 segment.  88GB/s for 16 segments.
  • Stefano will get the raw kernel launches to Gabriel so Gabriel can incorporate in his scripts
  • Stefano will post his results to confluence and we will share with Martin
  • Stefano will also continue to work on cuSZ
  • Stefano has verified that different data compresses at the same 68GB/s for 1 segment
  • cupy in spack supports up to cuda 12.1 but not 12.3, Robert is working on it
• Gabriel results with ped/gain/common-mode using 2-byte "fake" raw data:
  • 1 segment (352x384) - 11-12 us -> 22.755 GB/s
  • 6 segments (6x352x384) - 40 us -> 40.109 GB/s
  • 12 segments (12x352x384) - 72 us -> 44.598 GB/s
  • maybe we don't need common-mode in the drp?  do it offline?  Gabriel points out that this is different than what is currently being done.  cpo has a gut feeling it's still OK.
    • have asked Silke for her opinion about leaving out common mode in drp

    • (From Valerio) In my opinion it could work, and pedestal/gain is not going to improve with time (while common mode can!) so I think what you suggest, Chris, makes perfect sense. However, Silke has a more practical perspective, and I am curious what she thinks

    • (from Silke) I agree with Valerio. For many detectors, the gain from the common mode is not as crucial and for a given type of physics/detector setup it may not even be necessary. While there have been times where gain/pedestal did improve 'later', those are the cases we will try to monitor and avoid. "common mode only when in fixed gain": is that a LCLS2 behavior? I thought for LCLS1 we did calculate common-mode even when switching/mixed gain, but ignore the lower-gain pixels.

    • another advantage of no-common-mode in drp: calibrations become more easily "reversible" (can be redone with better constants later)
• Gabriel will investigate
  • multiple streams
    • cpo thinks multiple-events maps best to multiple-streams
    • multiple-segments should be done in one stream?
    • the writing of data with GPU-direct-storage might be in a different stream?
  • combining LC with the calibration (should this be one kernel? or kernels-launching-kernels (dynamic parallelism)? or (less likely) split up across streams?)

June 3, 2024

• Gabriel looked at launching calibrate/compress in one kernel or two.  Two kernels feels better: more modular (#blocks, #threads, use of shared-memory).  Continuing to look at streaming.
• cpo should ask TID if they have "TDet" firmware that can receive timing packets.  Maybe we don't need this and can just reuse existing TDet firmware?
• have asked TID if the gpu/cpu switchyard exists.  Ric would like see if we can run the daq entirely on the cpu with the datagpu driver.
• need a timing fiber
• lower priority: ideally need mellanox IB fixed
• could also ideally test a second node by adding a kcu1500

Aug 5, 2024

• Stefano is achieving 140GB/s on with his standalone angular integration (includes masking) for a 16Mpixel image.
• Stefano will work on pyFAI
• Stefano and cpo talked with Jiannan and Robert:
  • suggested using graphs for launching LC kernels (Stefano will look at this)
  • were worried about "outliers" getting clipped in LC.  But talked with Martin and he said there is no clipping.
  • priorities for Robert/Jiannan: make sure cupy bug is fixed so Stefano benchmark cuSZ, streams with cuSZ, support cpu-decompression
• Ric is working with Mudit
  • having difficulty enabling pcie bifurcation on dev06
    • we think we need to have two gpus to take advantage of bifurcation
    • notional idea of kcu/gpu structure: have multiple kcu's (~6 existing kcu's to get 50GB/s into GPU, only if pcie bus can do it).  pcie bandwidths are here: https://en.wikipedia.org/wiki/PCI_Express
  • supermicro update utility seems to report non-bifurcated when we think it is?
  • suggested mudit write a pgpread-style program for gpu: mudit's stuff doesn't work with gpu yet
  • ric working on drp-gpu code (targeting cpu side first)
  • tried running mudit firmware on gpu001 (non-bifurcated). saw events coming into cpu with pgpread. turning on bifurcation worked on gpu001.  dev06 bifurcation-issue seems to be specific to that machine (matt claims it used to work though)
  • Jeremy has example of cuda graphs if that ends up being useful

Aug 12, 2024

• Stefano working on cuSZ (Robert has provided cupy fix), pyFAI, and pi-slices azimuthal integration with standalone kernel
• Ric getting to the point where he needs Mudit's stuff to work.
  • Makefile system is kludgy
  • Can get through phase1 of configure
  • Any interaction with firmware messes up dma's to CPU.  register map assumed to be same as interlard test?  only know the mode-select address for certain.

Sept 9, 2024

Gabriel will put his ethernet weka performance numbers on confluence.  Sees only 2GB/s single-writer where Ric/cpo recall seeing 5GB/s previously.

Ric's code to flip transitions to GPU only for dma lane 0.  Questions:

• Why so many bits?
  • Axi stream demux (from Ben): lower 0x0000 configures a route-mask and a destination here: https://github.com/slaclab/axi-pcie-core/blob/main/python/axipcie/_AxiGpuAsyncCore.py#L163-L178.  upper 0xffff0000.  One stream in, two streams out (cpu and gpu).  Don't overthink it: just two values (otherwise get into the swamp of the general demux written by Ben that is being reused here)
• How do we support multiple GPUs?
• How do we support multiple pgp cards per GPU?
• Would like power-on default to be to vector the data to the CPU
  • Mudit/Ryan say that software initialization has to be done already (e.g. setting up clocks) so not a big deal to do this in software
• FYI for TID: Aiming to have the same software work for multiple detectors: e.g. TDet, epixM, epixUHR.  "GpuDetector" is analog of "PgpDetector".
• Aiming to have the same firmware for GPU/CPU e.g. for TDet.
  • Mudit confirms that we will be able to use same firmware for both CPU/GPU deployments
  • Should be true for other detectors as well, like epixUHR
• Question for Mudit: most TDet registers are in the 0x00a00000 space, but a few (e.g. the switch between cpu/gpu directions) are in the 0x00d00000 space.  What's the difference between the two spaces and should the "switch" be moved to 0x00a? (along with everything else, so nothing would be in 0x00d)
  • 0xa00000 space has the axi-lite registers for TDetsemi module, following Matt's original code
  • 0xd00000 space has the axi-lite registers for the dmux/asyncgpu modules