...

python software/scripts/updatePcieFpga.py --path ~weaver/mcs --type SPIx8 --dev /dev/datadev_1

As of Jan. 20, 2021, Matt has firmware here:

• ~weaver/mcs/DrpTDet-20200612*.mcs (tdetsim)
• /afs/slac/u/ec/weaver/projects/cameralink-gateway/firmware/targets/ClinkKcu1500Pgp2b/images/ClinkKcu1500Pgp2b-0x04090000-20201125220148-weaver-dirty*.mcs (timetool opal, but compatible with normal opal ... he will merge into Larry's repo).
  

Riccardo's notes:

one can find the KCU firmware in https://github.com/slaclab/pgp-pcie-apps/releases 
in this case provide with --path the folder where the firmware is stored

datadev Driver

• user "systemctl list-unit-files" to see if tdetsim.service or kcu.service is enabled
• associated .service files are in /usr/lib/systemd/system/
• to see if events are flowing from the hardware to the software: "cat /proc/datadev_0" and watch "Tot Buffer Use" counter under "Read Buffers"
• if you see the error "rogue.GeneralError: AxiStreamDma::AxiStreamDma: General Error: Failed to map dma buffers. Increase vm map limit: sysctl -w vm.max_map_count=262144".  This can be caused by that parameter being too low, but it can also be caused by loading the driver with CfgMode=1 in tdetsim.service (we use CfgMode=2).  This CfgMode parameter has to do with the way memory gets cached.
  • make sure the tdetsim.service (or kcu.service) is the same as another working node
  • make sure that the appropriate service is enabled (see first point)
  • check that the driver in /usr/local/sbin/datadev.ko is the same as a working node
  • check that /etc/sysctl.conf has the correct value of vm.max_map_count
  • It seems that vm.max_map_count must be at least 4K (the page size = the MMU granularity, perhaps?) larger than the datadev service's cfgRxCount value, and in some cases much larger (doubled or more) than the cfgRxCount value if Rogue is also used in the executeable
    
  • we have also seen this error when the datadev.ko buffer sizes and counts were configured incorrectly in tdetsim.service.
  • have also seen this error running opal updateFebFpga.py because it loops over 4 lanes which increases the usage of vm.max_map_count by 4.  devGui and DAQ worked fine because they only used one lane.  worked around it by hacking the number of lanes in the script to 1.  Ryan Herbst is thinking about a better fix.
• For high rate running, many DMA buffers (cfgRxCount parameter) are needed to avoid deadtime. The number of DMA buffers is also used to size the DRPs' pebble, so slow but large DRPs like PvaDetector and EpicsArch need a comparatively small cfgRxCount value.  Neglecting to take this into account can result in attempting to allocate more memory than the machine has, resulting in the DRP throwing std::bad_alloc.
  
  • The number of buffers returned by dmaMapDma() is the sum of the cfgTxCount and the cfgRxCount values.  The DRP code rounds this up to the next power of 2 (if it's not already a power of 2) and uses the result to allocate the number of pebble buffers.  To avoid wasting up to half the allocation, set the sum of cfgTxCount and cfgRxCount to a power of 2.
• DMA buffers can be small (cfgSize parameter) for most (all?) tdetsim applications.  16 KB is usually sufficient.

...