Overview
Details of US Digital USB Encoder support in LCLS2.
Verifying device connection
listencoders output (daq-tst-dev03)
$ sudo listencoders Initializing device Found 1 device! Device 0: Model : 68 Version : 1 Serial Number: 100ec Date : 9/12/2012 Shutting down device $
"Operation not permitted" when opening /dev/tpra
Another process may have /dev/tpra open
$ sudo lsof /dev/tpra COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME pvcam 26650 nobody mem CHR 242,14 10503 /dev/tpra pvcam 26650 nobody 10u CHR 242,14 0t0 10503 /dev/tpra
Updating the bld receiver
Matt writes
I added a program (usdusb) to read the encoder and send BLD. It's now in psdaq/psdaq/app/usdusb.cc.
Can you give it a try and then add code to psdaq/drp/bld.cc to receive it?
Look for sections with strings like "ebeam" and add similar code for "encoder" (you choose the name).
I think tmo.cnf has an example of starting the bld drp receiver.
The program is started on daq-tst-dev03 like so:
(ps-3.1.16) bash-4.2$ usdusb -h
Usage: usdusb [OPTIONS]
-a <addr> : MC address [default: 0xefff1904]
-i <addr/name> : MC interface [no default]
-t : disable timestamp check
-z : zero encoder
-T <trigger device file,output,delay_ticks,width_ticks>
: [default: /dev/tpra0,0,0,1]
-1 <event code>: LCLS1 event select [default: false]
-2 <rate> : LCLS2 event select [default: 5 (10Hz)]
-v : verbose mode
usdusb -a 239.255.25.4 -i enp1s0f1 -1 40 -v -T /dev/tpra4,0,0,10
This reads the encoder on eventcode 40, triggering on TTL output 0 (delay=0, width=10 ticks).
It sends out multicasts to address 239.255.25.4 on interface enp1s0f1.
Testing
daq-tst-dev03 (1 Hz)
$ usdusb -a 239.255.25.4 -i enp1s0f1 -1 45 -v -T /dev/tpra4,0,0,10 Using interface 239.255.25.4 (239.255.25.4) Using interface enp1s0f1 (10.0.0.3) Initializing device Found 1 devices Configuration Done Read count 94643899334656 Using /dev/tpra (4) Version PulseID TimeStamp Markers BeamReq BsaInit BsaActiv BsaAvgD BsaDone TprBase::channel[4] evtsel 4000102c control 4 TprBase::channel[4] evtsel 4000102d control 7 ReadFIFO 1 records ReadFIFO 1 records ReadFIFO 1 records ReadFIFO 1 records ReadFIFO 1 records ReadFIFO 1 records ReadFIFO 1 records ^Cclose_usb 2 Unlinking semaphore Usb4-0000
drp-tst-dev009
$ drp_bld -l 0x1 -k "interface=eno1" -P txt -p1 -D usdusb -d /dev/datadev_1 -u usdusb_0 -v txt-drp_bld[363434]: <I> logging configured txt-drp_bld[363434]: <D> tcp://:29991 txt-drp_bld[363434]: <I> dmaCount 65540 dmaSize 131072 txt-drp_bld[363434]: <I> nbuffers 131072 pebble buffer size 131072 txt-drp_bld[363434]: <I> nTrBuffers 32 transition buffer size 262144 txt-drp_bld[363434]: <W> Unable to update Prometheus configuration: directory not provided txt-drp_bld[363434]: <I> Providing run-time monitoring data on port 9200 txt-drp_bld[363434]: <I> Timing RefClk 185.920640 MHz txt-drp_bld[363434]: <I> output dir: n/a txt-drp_bld[363434]: <I> Ready for transitions
Lab 3 Connections
daq-tst-dev03
-bash-4.2$ hostname daq-tst-dev03.pcdsn -bash-4.2$ ls -l /dev/datadev_? crw-rw-rw- 1 root root 240, 0 Oct 19 10:16 /dev/datadev_0 -bash-4.2$ ifconfig enp1s0f0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.21.148.110 netmask 255.255.252.0 broadcast 172.21.151.255 inet6 fe80::ec4:7aff:fe3a:2ac4 prefixlen 64 scopeid 0x20<link> ether 0c:c4:7a:3a:2a:c4 txqueuelen 1000 (Ethernet) RX packets 19957585 bytes 8237210479 (7.6 GiB) RX errors 0 dropped 1 overruns 0 frame 0 TX packets 889936 bytes 595082502 (567.5 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 enp1s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9000 inet 10.0.0.3 netmask 255.255.252.0 broadcast 10.0.3.255 inet6 fe80::ec4:7aff:fe3a:2ac5 prefixlen 64 scopeid 0x20<link> ether 0c:c4:7a:3a:2a:c5 txqueuelen 1000 (Ethernet) RX packets 743488 bytes 77981121 (74.3 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 210197 bytes 98169259 (93.6 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
drp-tst-dev004
-bash-4.2$ hostname drp-tst-dev004 -bash-4.2$ ls -l /dev/datadev_? crw-rw-rw- 1 root root 235, 0 Sep 30 11:04 /dev/datadev_0 crw-rw-rw- 1 root root 234, 0 Sep 30 11:04 /dev/datadev_1 -bash-4.2$ ifconfig eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.21.148.247 netmask 255.255.252.0 broadcast 172.21.151.255 inet6 fe80::ae1f:6bff:fe57:312a prefixlen 64 scopeid 0x20<link> ether ac:1f:6b:57:31:2a txqueuelen 1000 (Ethernet) RX packets 477966578 bytes 483440990448 (450.2 GiB) RX errors 4 dropped 13 overruns 0 frame 4 TX packets 378138875 bytes 385326968039 (358.8 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eno2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether ac:1f:6b:57:31:2b txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044 inet 172.21.52.147 netmask 255.255.252.0 broadcast 172.21.55.255 inet6 fe80::526b:4b03:42:e984 prefixlen 64 scopeid 0x20<link> Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8). infiniband 80:00:00:86:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 4096 (InfiniBand) RX packets 623 bytes 74953 (73.1 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 965 bytes 102121 (99.7 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
drp-tst-dev009
-bash-4.2$ hostname drp-tst-dev009 -bash-4.2$ ls -l /dev/datadev_? crw-rw-rw- 1 root root 235, 0 Oct 20 12:17 /dev/datadev_0 crw-rw-rw- 1 root root 234, 0 Oct 20 12:17 /dev/datadev_1 -bash-4.2$ ifconfig eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.21.148.63 netmask 255.255.252.0 broadcast 172.21.151.255 inet6 fe80::ae1f:6bff:fe57:3122 prefixlen 64 scopeid 0x20<link> ether ac:1f:6b:57:31:22 txqueuelen 1000 (Ethernet) RX packets 13336630 bytes 6017437841 (5.6 GiB) RX errors 0 dropped 10 overruns 0 frame 0 TX packets 2562762 bytes 601922653 (574.0 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eno2: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether ac:1f:6b:57:31:23 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ib0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044 inet 172.21.52.152 netmask 255.255.252.0 broadcast 172.21.55.255 inet6 fe80::526b:4b03:42:f1f8 prefixlen 64 scopeid 0x20<link> Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8). infiniband 00:00:04:D1:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 4096 (InfiniBand) RX packets 39 bytes 6903 (6.7 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 309 bytes 37513 (36.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Configure Static IP Address
10.0.3.255 Network
-bash-4.2$ hostname drp-tst-dev008 -bash-4.2$ pwd /etc/sysconfig/network-scripts -bash-4.2$ diff -C 10 ifcfg-eno2~ ifcfg-eno2 *** ifcfg-eno2~ 2020-10-27 16:20:38.588365666 -0700 --- ifcfg-eno2 2020-10-27 17:45:39.571303813 -0700 *************** *** 1,15 **** TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no ! BOOTPROTO=dhcp ! DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=eno2 UUID=3d07bc9d-65b7-42e3-bd4a-587fde3ef026 DEVICE=eno2 ! ONBOOT=no --- 1,18 ---- TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no ! BOOTPROTO=none ! DEFROUTE=no IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=eno2 UUID=3d07bc9d-65b7-42e3-bd4a-587fde3ef026 DEVICE=eno2 ! ONBOOT=yes ! IPADDR=10.0.1.8 ! NETMASK=255.255.252.0 ! MTU=9000 -bash-4.2$
xcasttest
Matt writes
You won't see a particular multicast group with tcpdump unless you have a client running which has subscribed to that multicast group. So, without a receiver, I see the same thing as you. If I setup a receiver like so xcasttest -i eno1 -a 239.255.25.0 -p 10148 -r (to receive ebeam bld) then I get the following... 11:15:08.750133 IP ioc-sys0-bd02-fnet.slac.stanford.edu.31111 > 239.255.25.0.10148: UDP, length 224 11:15:08.767132 IP ioc-sys0-bd02-fnet.slac.stanford.edu.31111 > 239.255.25.0.10148: UDP, length 224
Multicast Routing
Effect of ip route add
-bash-4.2$ hostname drp-tst-dev008 -bash-4.2$ ping -c 2 224.0.0.1 PING 224.0.0.1 (224.0.0.1) 56(84) bytes of data. 64 bytes from 172.21.148.57: icmp_seq=1 ttl=64 time=0.014 ms 64 bytes from 172.21.148.247: icmp_seq=1 ttl=64 time=0.164 ms (DUP!) 64 bytes from 172.21.148.63: icmp_seq=1 ttl=64 time=0.165 ms (DUP!) 64 bytes from 172.21.148.57: icmp_seq=2 ttl=64 time=0.014 ms --- 224.0.0.1 ping statistics --- 2 packets transmitted, 2 received, +2 duplicates, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.014/0.089/0.165/0.075 ms -bash-4.2$ -bash-4.2$ route Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface default gateway 0.0.0.0 UG 0 0 0 eno1 10.0.0.0 0.0.0.0 255.255.252.0 U 0 0 0 eno2 link-local 0.0.0.0 255.255.0.0 U 1002 0 0 eno1 link-local 0.0.0.0 255.255.0.0 U 1003 0 0 eno2 link-local 0.0.0.0 255.255.0.0 U 1004 0 0 ib0 172.21.52.0 0.0.0.0 255.255.252.0 U 0 0 0 ib0 172.21.148.0 0.0.0.0 255.255.252.0 U 0 0 0 eno1 -bash-4.2$ -bash-4.2$ sudo ip route add 224.0.0.0/4 dev eno2 [sudo] password for caf: -bash-4.2$ -bash-4.2$ ping -c 2 224.0.0.1 PING 224.0.0.1 (224.0.0.1) 56(84) bytes of data. 64 bytes from 10.0.1.8: icmp_seq=1 ttl=64 time=0.016 ms 64 bytes from 10.0.0.3: icmp_seq=1 ttl=64 time=0.101 ms (DUP!) 64 bytes from 10.0.1.4: icmp_seq=1 ttl=64 time=0.153 ms (DUP!) 64 bytes from 10.0.1.9: icmp_seq=1 ttl=64 time=0.307 ms (DUP!) 64 bytes from 10.0.1.8: icmp_seq=2 ttl=64 time=0.032 ms --- 224.0.0.1 ping statistics --- 2 packets transmitted, 2 received, +3 duplicates, 0% packet loss, time 999ms rtt min/avg/max/mdev = 0.016/0.121/0.307/0.105 ms -bash-4.2$