Teledyne Digitizer Evaluation

    CBXFEL Digitizer Test Plan
    Chris Ford <caf@slac.stanford.edu>
    Nov 1, 2022

------- Phase 1 ---------------------------------------------------

Receive Teledyne documentation and software

Receive charge code

Work with Thuy to install digitizer and timing hardware in 2U crate
 - Building 034
 - Remote access
 - root or sudo access

------- Phase 2 ---------------------------------------------------

Verify basic compatibility with SLAC's Linux environment
 - Teledyne kernel module (insmod)
 - Teledyne library (build and run example C program)

Verify reading a DC voltage

------- Phase 3 ---------------------------------------------------

Connect SLAC trigger
Verify synchronous operation at 1 Hz
Verify synchronous operation at 120 Hz

    CBXFEL Digitizer Test Plan
    Chris Ford <caf@slac.stanford.edu>
    Nov 29, 2022

------- Phase 1 --(complete)---------------------------------------

Receive Teledyne documentation and software

Receive charge code

Work with Thuy to install digitizer and timing hardware in 2U crate
 - Building 034
 - Remote access
 - root or sudo access

Verify that card is recognized by Linux using lspci
(Note: we did this with 1U cpu and CentOS 7)

------- Phase 2 --(in progress)------------------------------------

Verify basic compatibility with SLAC's Linux environment
 - obtain GPL'd kernel module  (complete)
 - build kernel module in SLAC LinuxRT environment (in progress)
 - Teledyne kernel module: insmod
 - Teledyne library: build and run ADQAPI_simple_example.c

Verify reading a DC voltage

------- Phase 3 ---------------------------------------------------

Connect trigger to SLAC EVR (requires 2U cpu)

Verify synchronous operation at 1 Hz

Verify synchronous operation at 120 Hz

------- Open Issues -----------------------------------------------

1. Missing compatible power cable for 2U 8201
2. Teledyne library depends on libusb even though ADQ14 is pci

INSTALLATION
============
To install the SDK packages, enter the "packages" directory and find the linux
distribution and processor architecture you are using.  Specific instructions
for each distribution follows below.
After installing all necessary packages, reboot the system so that udev reads
the updated configuration and the driver loads.
**Note** In earlier versions there was a package called 'adqupdater' which is
now replaced with the 'adqtools' package.
Ubuntu and Debian
-----------------
Install delivered versions of our packages using:
    dpkg -i packagename
Use the following order:
 - spd-adq-pci-dkms (not needed for USB devices)
 - libadq0
 - adqtools
OpenSUSE and SUSE Linux Enterprise
----------------------------------
Install these packages using:
    zypper install packagename
Use the following order:
 - make
 - kernel-devel
 - kernel-source
 - gcc
The version of kernel-devel and kernel-source must match your current
kernel. Install delivered versions of our packages using:
    rpm -U packagename
Use the following order:
 - dkms
 - spd-adq-pci-dkms (not needed for USB devices)
 - libadq0
 - adqtools
OpenSUSE Leap 15.0 or later
---------------------------
Install these packages using:
    zypper install packagename
Use the following order:
 - make
 - kernel-devel
 - kernel-source
 - gcc
 - dkms
The version of kernel-devel and kernel-source must match your current
kernel. Install delivered versions of our packages using:
    rpm -U packagename
Use the following order:
 - spd-adq-pci-dkms (not needed for USB devices)
 - libadq0
 - adqtools
Fedora 19, 20 and 21
--------------------
Install the dkms package using:
    yum install dkms
Install delivered versions of our packages using:
    rpm -U packagename
Use the following order:
 - spd-adq-pci-dkms (not needed for USB devices)
 - libadq0
 - adqtools
Fedora 22 and higher
--------------------
Install the dkms package
    dnf install packagename
Install delivered versions of our packages using:
    dnf install packagename
Use the following order:
 - spd-adq-pci-dkms (not needed for USB devices)
 - libadq0
 - adqtools
CentOS / Red Hat Enterprise Linux / Scientific Linux
----------------------------------------------------
**Note** For RHEL6, use CentOS6 packages.
Install these packages using:
    yum install packagename
Use the following order
 - make
 - kernel-devel
 - gcc
The version of kernel-devel must match your current kernel.
Install delivered versions of our packages using:
    rpm -U packagename
Use the following order:
 - dkms
 - spd-adq-pci-dkms (not needed for USB devices)
 - libadq0
 - adqtools
Device access permissions
=========================
ADQ devices show up as `/dev/adq_pcie_` and `/dev/adq_usb_`, the default udev
setting is to add read/write access to the user group "adq".  The libadq0
package will create a user group called "adq" if that group doesn't already
exist in the system.  To be able to access the devices, in order to add your
user to the "adq" group use the command:
    usermod -a -G adq username
The user will have to logout and login again for the changes to take effect.
Kernel module version compatibility
===================================
The PCIe kernel module supports kernel versions from 2.6.32 and forward, however
kernel version 3.8.0 or newer is recommended.
The PCIe kernel module is not signed and thus will not load if your kernel uses
"secure boot".  If "secure boot" is enabled, it will need to be disabled before
the kernel module can be loaded. Refer to your distribution documentation on
how to do this.
API user guide
==============
The API user guide is included as a .pdf document in the "doc" directory. All
API functions are described there.
Example code
============
Example code for different languages can be found in the "examples" directory.
C/C++ examples
--------------
C/C++ example code is compiled using Make by going to the directory with the
Makefile and running "make". To setup include paths and defines for the
compiler "pkg-config" is used in the C/C++ examples. If not already installed,
pkg-config can be installed using the appropriate distribution package tools.
Python examples
---------------
The Python examples are written for Python 3. They use the "ctypes" module to
call ADQAPI functions and convert Python data structures to the appropriate
format.

This is the PCIe driver for Signal Processing Devices digitizers.

If your Linux distribution supports DKMS, the easiest way to install
the driver is by using the appropriate .deb or .rpm package.

To install the driver manually, you need to first compile it and then
install it. As a prerequisite, the kernel headers corresponding to
your installed Linux kernel needs to be installed. See your distributions
documentation on how to install the kernel headers, the package is
usually called 'linux-kernel-headers', 'kernel-devel', 'kernel-headers',
'linux-headers' or something similar.

After installing the prerequisites, compile the PCIe driver by running
this command in the same folder as this README:

     make

Then install the driver by running this command as root (using su or sudo):

     make install

After installation, the module will normally be loaded automatically at
boot when a card is attached. A manual module load can be performed by
running the following command as root:

     modprobe spd_adq_pci

By default only root has access to the driver device files.
When installing the ADQAPI software package, udev rules will be
automatically set up so that the device files can be accessed by users.

Note that if you upgrade your kernel, you will need to re-run 'make' and
'make install' in order to compile the driver for your new kernel.
When using DKMS, this is handled automatically.

root@cpu-b34-sp02]$ ls
adq.ko
[ root@cpu-b34-sp02]$ ls /dev > dev_before_insmod.txt
[ root@cpu-b34-sp02]$ dmesg   > dmesg_before_insmod.txt
[ root@cpu-b34-sp02]$ insmod adq.ko 
[102834.973910] adq: loading out-of-tree module taints kernel.
[102834.979952] adq: SPD ADQ PCI Device driver v1.21 loaded.
[102834.985819] spd_adq_pci 0000:da:00.0: Found Signal Processing Devices ADQ14, device PCIe/PXIe/uTCA bus 218 slot 0 function 0
[ root@cpu-b34-sp02]$ ls /dev > dev_after_insmod.txt
[ root@cpu-b34-sp02]$ dmesg   > dmesg_after_insmod.txt
[ root@cpu-b34-sp02]$ 
[ root@cpu-b34-sp02]$ ls -l
total 229
-rw-------    1 laci     lcls         32048 Nov 30 16:42 adq.ko
-rw-r--r--    1 root     root          3857 Nov 30 16:45 dev_after_insmod.txt
-rw-r--r--    1 root     root          3840 Nov 30 16:44 dev_before_insmod.txt
-rw-r--r--    1 root     root         95069 Nov 30 16:46 dmesg_after_insmod.txt
-rw-r--r--    1 root     root         94816 Nov 30 16:45 dmesg_before_insmod.txt
[ root@cpu-b34-sp02]$ diff dev_before_insmod.txt dev_after_insmod.txt 
--- dev_before_insmod.txt
+++ dev_after_insmod.txt
@@ -1,3 +1,4 @@
+adq_pcie_218_0_0
 bus
 console
 cpu
[ root@cpu-b34-sp02]$ diff dmesg_before_insmod.txt dmesg_after_insmod.txt 
--- dmesg_before_insmod.txt
+++ dmesg_after_insmod.txt
@@ -1347,3 +1347,6 @@
 [   23.257900] random: dropbear: uninitialized urandom read (32 bytes read)
 [   24.553478] ipmi device interface
 [   24.768942] IPv6: ADDRCONF(NETDEV_UP): eth5: link is not ready
+[102834.973910] adq: loading out-of-tree module taints kernel.
+[102834.979952] adq: SPD ADQ PCI Device driver v1.21 loaded.
+[102834.985819] spd_adq_pci 0000:da:00.0: Found Signal Processing Devices ADQ14, device PCIe/PXIe/uTCA bus 218 slot 0 function 0
[ root@cpu-b34-sp02]$ 

lcls-dev3$ pwd
/u/ey/caf/digitizer/sandbox2/linuxKernel_Modules/adqDriver/R1.29.0/build/buildroot-2019.08-x86_64 lcls-dev3$ make clean
rm -rf *.o *~ core .depend .*.cmd *.ko *.mod.c .tmp_versions *.symvers  *.order *.c *.h module_* original.* lcls-dev3$ ls
Makefile lcls-dev3$ make
### copy the sources to make a directory for building
cp -r ../../src/* .
make compile
make[1]: Entering directory `/afs/slac.stanford.edu/g/controls/development/users/caf/digitizer/sandbox2/linuxKernel_Modules/adqDriver/R1.29.0/build/buildroot-2019.08-x86_64'
make ARCH=x86_64 CROSS_COMPILE=/afs/slac/package/linuxRT/buildroot-2019.08/host/linux-x86_64/x86_64/usr/bin/x86_64-buildroot-linux-gnu- -C /afs/slac/package/linuxRT/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/linux-4.14.139 M=/u/ey/caf/digitizer/sandbox2/linuxKernel_Modules/adqDriver/R1.29.0/build/buildroot-2019.08-x86_64 modules
make[2]: Entering directory `/afs/slac.stanford.edu/package/linuxRT/vol7/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/linux-4.14.139'
  CC [M]  /u/ey/caf/digitizer/sandbox2/linuxKernel_Modules/adqDriver/R1.29.0/build/buildroot-2019.08-x86_64/spd_adq_pci.o
  LD [M]  /u/ey/caf/digitizer/sandbox2/linuxKernel_Modules/adqDriver/R1.29.0/build/buildroot-2019.08-x86_64/adq.o
  Building modules, stage 2.
  MODPOST 1 modules
  CC      /u/ey/caf/digitizer/sandbox2/linuxKernel_Modules/adqDriver/R1.29.0/build/buildroot-2019.08-x86_64/adq.mod.o
  LD [M]  /u/ey/caf/digitizer/sandbox2/linuxKernel_Modules/adqDriver/R1.29.0/build/buildroot-2019.08-x86_64/adq.ko
make[2]: Leaving directory `/afs/slac.stanford.edu/package/linuxRT/vol7/buildroot-2019.08/buildroot-2019.08-x86_64/output/build/linux-4.14.139'
make[1]: Leaving directory `/afs/slac.stanford.edu/g/controls/development/users/caf/digitizer/sandbox2/linuxKernel_Modules/adqDriver/R1.29.0/build/buildroot-2019.08-x86_64' lcls-dev3$ make install
mkdir -p ../../buildroot-2019.08-x86_64/include/
cp  spd_adq_pci_ioctl.h ../../buildroot-2019.08-x86_64/include/.
cp  *.ko ../../buildroot-2019.08-x86_64/. lcls-dev3$  lcls-dev3$ ls -l ../../buildroot-2019.08-x86_64/
total 34
-rw-rw-r-- 1 caf ey 32048 Dec  2 13:26 adq.ko
drwxrwxr-x 2 caf ey  2048 Dec  2 13:26 include lcls-dev3$  lcls-dev3$ ls -l ../../buildroot-2019.08-x86_64/include 
total 5
-rw-rw-r-- 1 caf ey 4466 Dec  2 13:26 spd_adq_pci_ioctl.h lcls-dev3$