Pictures
In the detector lab of a 0.5M module
Larger detectors are made up of several of these smaller modules.
Setup
on psdev: /reg/common/tools/bin/netconfig search *jungfrau* --brief
det-daq:~$ ping det-jungfrau-31
PING det-jungfrau-31.pcdsn (172.21.80.239) 56(84) bytes of data.
^C
--- det-jungfrau-31.pcdsn ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5000ms
det-daq:~$
det-daq:~$ /reg/common/tools/bin/netconfig view det-jungfrau-31
name: det-jungfrau-31
subnet: cds-xcs.pcdsn
Ethernet Address: 00:50:c2:46:d8:b3
IP: 172.21.80.239
PC#: 00000
Location: ASC Room 1034
Contact: uid=ddamiani,ou=People,dc=reg,o=slac
Description: Jungfrau module bchip031 control interface (a.k.a the 512k)
Puppet Classes:
det-daq:~$
det-daq:~$ /reg/common/tools/bin/netconfig edit det-jungfrau-31 --subnet cds-det.pcdsn
Checking parameters against LDAP database ...
Please confirm the following operation:
Modify det-jungfrau-31 properties:
Subnet: cds-det.pcdsn
IP address: 172.21.58.73
Do you really want to apply those changes (y/N) ? y
Updating database ...
newsuperior: dc=cds-det.pcdsn,ou=Subnets,dc=reg,o=slac
newrdn: cn=det-jungfrau-31
dn; cn=det-jungfrau-31,dc=cds-xcs.pcdsn,ou=Subnets,dc=reg,o=slac
Edited node cn=det-jungfrau-31,dc=cds-det.pcdsn,ou=Subnets,dc=reg,o=slac in LDAP directory.
Notify network services that the configuration has changed:
Re-running the command on relay psldapsrv as root, you may
be asked to type in your password:
Warning: Permanently added 'psldapsrv' (ED25519) to the list of known hosts.
[sudo] password for cpo:
Creating new DNS zone file ...
Executing command '/cds/sw/tools/src/LDAP_Helpers/ldap2zone.py /var/named '134.79 172.21.9' cds-det.pcdsn'
Creating new DHCP config file ...
Executing command '/cds/sw/tools/src/LDAP_Helpers/ldap2dhcp.py --ldapsrv psldapsrv --basedn ou=Subnets,dc=reg,o=slac --file /etc/dhcp/dhcpd.conf'
Opening /etc/dhcp/dhcpd.conf for writing ... done
Generating DHCP configuration from ou=Subnets,dc=reg,o=slac ... done
Restarting services ... done.
Network services are now in sync with the LDAP directory.
Connection to psldapsrv closed.
det-daq:~$
det-daq:~$ ping det-jungfrau-31
PING det-jungfrau-31.pcdsn (172.21.58.73) 56(84) bytes of data.
64 bytes from 172.21.58.73 (172.21.58.73): icmp_seq=1 ttl=64 time=0.449 ms
64 bytes from 172.21.58.73 (172.21.58.73): icmp_seq=2 ttl=64 time=0.315 ms
^C
--- det-jungfrau-31.pcdsn ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.315/0.382/0.449/0.067 ms
det-daq:~$
Notes
- each jungfrau runs linux (blackfin?)
- ethernet goes to CDS subnet in DAQ lab (a couple of small switches lying around have this subnet)
- has both software and firmware
- ethernet goes to CPU
- fiber goes to FPGA
- data streams out of FPGA via UDP (we need jumbo frames, but not pause frames, might need other ethtools settings, we think intel/myricom nic settings are the good at the defaults)
- the FEZ interface that receives the EVR multicasts (and transmits the bulk data to the DSS nodes) needs interrupt coalescing (may depend on the nic). maybe 75us for myricom and 50us for the intel nic? the coalescing matters on nodes reading out large detectors, or large numbers of small detectors
- jungfrau segment level
- event builds multiple modules
- stitches together 8 asics from each 0.5M in a natural order
- sometimes need to update software/firmware
- can update firmware via CPU, but if that fails have to jtag, but it's altera. altera is installed on machine daq-det-standalone (had to do this more with the early jungfrau's)
- each 0.5M silicon has a serial number that can't be read out via the DAQ
- these serial numbers are used by mikhail to lookup calibration constants, but must be managed administratively
- use the fiber farthest away from the ethernet
- most module fibers are "green" when locked, but one of them is red unfortunately
- in daq lab fiber goes to daq-det-jungfrau (also used for epix's)
- second fiber is used for high-rate bonding which isn't supported yet
- on det-daq-jungfrau machine fez is 172.21.59.53
- the detector fiber is hooked up to one of the other interfaces
- use ethtool to look for link locked (ethtool enp5s0)
- standalone executable to read out jungfrau's (same code as segment level)
- see also Running DAQ devices standalone
- want to run as detopr because executable creates temporary files then there can be permissions problems cleaning them up to allow a future instance to run
- executables live in /reg/g/pcds/dist/pds/current/build/pdsapp/bin (no env setup necessary)
- /reg/g/pcds/dist/pds/current/build/pdsapp/bin/x86_64-rhel7-opt/jungfrauStandAlone -P 32410 -H 10.1.1.105 -m 00:60:dd:45:66:df -d 10.1.1.55 -s det-jungfrau-31 (use -h for help)
- -P is port number on the detector side
- -H is the host IP address of the fiber interface
- -m is the mac of the host fiber interface
- -d the ip address of the detector
- -s hostname of the detector control interface (what's in netconfig)
- look at cnf file to get params: "grep jungfrau-31 /cds/group/pcds/dist/pds/det/scripts/det.cnf"
Generate one frame in standalone:
daq-det-jungfrau:~> /reg/g/pcds/dist/pds/current/build/pdsapp/bin/x86_64-rhel7-opt/jungfrauStandAlone -P 32410 -H 10.1.1.105 -m 00:60:dd:45:66:df -d 10.1.1.55 -s det-jungfrau-31 Shared memory created /slsDetectorPackage_multi_0 Shared memory created /slsDetectorPackage_multi_0_sls_0 detector udp_rx interface appears to be unset setting up detector udp_rx interface cmd_put rx_udpport: 32410 cmd_put rx_udpip: 10.1.1.105 cmd_put rx_udpmac: 00:60:dd:45:66:df cmd_put detectorip: 10.1.1.55 cmd_put detectormac: 00:aa:bb:cc:dd:ee cmd_put configuremac: 0 detector udp_rx interface is up Configuring 1 modules checking status of module 0 reg_gett 0x5e: 0x0 module chips need to be powered on configuring dacs of module 0 Setting Dacs: setting vb_ds to 1000 cmd_put dac:5: 1000 setting vb_comp to 1220 cmd_put dac:0: 1220 setting vb_pixbuf to 750 cmd_put dac:4: 750 setting vref_ds to 480 cmd_put dac:6: 480 setting vref_comp to 420 cmd_put dac:7: 420 setting vref_prech to 1450 cmd_put dac:3: 1450 setting vin_com to 1053 cmd_put dac:2: 1053 setting vdd_prot to 3000 cmd_put dac:1: 3000 configuring adc of module 0 powering on the chip Detector returned error: Writing to register 0x5e failed: wrote 0x1 but read 0x3 Write to register failed reg_put 0x5e - 0x1: 0x0 resetting the adc adc_put 0x8 - 0x3: 0xffffffff adc_put 0x8 - 0: 0xffffffff adc_put 0x14 - 0x40: 0xffffffff adc_put 0x4 - 0xf: 0xffffffff adc_put 0x5 - 0x3f: 0xffffffff adc_put 0x18 - 0x2: 0xffffffff reg_put 0x43 - 0x453b2a9c: 0x453b2a9c configuring clock speed of module 0 setting detector to half speed cmd_put clkdivider: 1 configuring acquistion settings of module 0 reseting run control ... done setting trigger delay to 0.000238 configuring for free run reg_put 0x4e - 0: 0x0 cmd_put cycles: 1 cmd_put frames: 1 cmd_put period: 0.200000000 setting exposure time to 0.000010 seconds cmd_put exptime: 0.000010000 configuring gain and bias of module 0 setting bias voltage to 200 volts cmd_put vhighvoltage: 200 setting gain mode 0 clearbit 0x5d - 0: 0xf00 clearbit 0x5d - 1: 0xf00 clearbit 0x5d - 2: 0xf00 clearbit 0x5d - 12: 0xf00 clearbit 0x5d - 13: 0xf00 starting detector: idle got frame: 1 stopping detector: idle daq-det-jungfrau:~>
4M Idiosyncracies
- read out on two nodes because too much data for one