SLAC Teststand Setup

Basic info:

ATCA shelf and COB connected to: rdusr219
Shelf name, IP: lbne1_0102,192.168.1.254
RCE SDK path: /afs/slac.stanford.edu/g/cci/volumes/vol1/sdimages/sdk/current (To get env: source /afs/slac.stanford.edu/g/cci/volumes/vol1/sdimages/sdk/V3.3.0/i86-linux-64/tools/envs-sdk.sh)
RCE Soft/Firmware path: /u1/hps/heavy-photon-daq
- https://github.com/slaclab/heavy-photon-daq/releases/tag/release-4.1-run2016
The HPS COB is in ATCA slot #2; slot #1 is used by other groups.
DHPC server config: /u1/dhcp/dnsmasq.local.conf (restart server with: sudo service restart dnsmasq).
Another way to restart the dns server is to >sudo /etc/init.d/dnsmasq restart
Useful infos for ATCA and RCE commands:
Software FAQ

Check the ti crate

tcpClient slac1 tiStatus

Installing Rogue

The instructions about how to install rogue with conda can be found here:

https://slaclab.github.io/rogue/installing/anaconda.html

Rogue is installed on the centos7 machines on the sdf location:

/sdf/group/hps/software

To set it up:
source /sdf/group/hps/software/anaconda3/etc/profile.d/conda.sh

Then to activate:
conda activate rogue_5.7.0

The conda installation also features cmake v3.14 and root 6.24

Installing rogue_lite

For HPS data taking is necessary to compile the C++ only version of rogue outside conda environment. This is because rogue library gets dynamically loaded by coda and the latter is not exposed to the anaconda environment.
It's important that the version of the standalone rogue library matches the version loaded in the conda environment. As of June 2021, we are using rogue v5.8.0 that can be obtained from:
https://github.com/slaclab/rogue
This external C++ installation is called rogue_lite in HPS jargon. Here are the steps to compile it:

git clone https://github.com/slaclab/rogue rogue_lite
cd rogue_lite
git checkout v5.8.0
mkdir lib
cd lib
cmake .. -DNO_PYTHON=1
make install

Once this is compiled and installed, in order to setup rogue location in the environment, just:

source setup_rogue.sh

This will setup ROGUE_DIR folder.

Compiling and installing the event_builder

The event builder is located in the heavy-photon-daq repository. As of June 2021, we are using the Run2021 branch.
https://github.com/slaclab/heavy-photon-daq
Under software/event_builder
It needs to be compiled against the rogue_lite version

cd heavy-photon-daq/software/event_builder
source src/scripts/setup_eb.sh
BuildHps

Bring the event_builder and rogue_lite libraries in coda

The event_builder and rogue_lite libraries are just copied in coda

rm -rf /pathTo-coda/Linux_x86_64/lib/librogue*
rm -rf /pathTo-coda/Linux_x86_64/lib/libhps*
\cp -a -v /pathTo-rogue_lite/lib/* pathTo-coda/Linux_x86_64/lib/
\cp -a -v /pathTo-heavy-photon-daq/software/rogue_coda/install/x86_64-linux/lib/* pathTo-coda/Linux_x86_64/lib/

At Jlab, this file is important:

bravo@clonfarm2:/usr/clas12/release/1.4.0/slac_svt$ cat copy_libraries.sh
#!/bin/bash
rm -rf /usr/clas12/release/1.4.0/coda/Linux_x86_64/lib/librogue*
rm -rf /usr/clas12/release/1.4.0/coda/Linux_x86_64/lib/libhps*
\cp -a -v /usr/clas12/release/1.4.0/slac_svt/rogue_lite/lib/* /usr/clas12/release/1.4.0/coda/Linux_x86_64/lib/
\cp -a -v /usr/clas12/release/1.4.0/slac_svt/heavy-photon-daq/software/rogue_coda/install/x86_64-linux/lib/* /usr/clas12/release/1.4.0/coda/Linux_x86_64/lib/

Setting up a VNC for SVT DAQ work - DEPRECATED

SVT DAQ system implies running a gui for configuration and data taking. Setting up a VNC can speed up significantly work time.
It is suggested to setup a VNC server on rdsrv305 as that is directly accessible via internet.
General informations on how to setup a VNC are given here:
VNC on Unix

This is how you can setup VNC on rdsrv305.

ssh username@rdsrv305.slac.stanford.edu

In the case you don't have access to your home area (try to touch something) try:

kinit
aklog

Then setup a vncserver

vncserver :1 -geometry 2880x1620 # Use whatever resolution your home monitor has

#Fix the permissionsfs setacl ~/.vnc system:slac none
fs setacl ~/.vnc system:authuser noneexit #logout
ssh username@rdsrv305.slac.stanford.edu -L9999:localhost:5901 #port is 590X where X is the monitor number :X
#If you see a black screen with checkmarks to select but you don't see a monitor, take this vncstartup file:
cp ~bareese/.vnc/xstartup ~/.vnc/

You should be able to connect to the vncserver via ssh tunnel through localhost:9999

Setting up a VNC for CODA Running and sharing work

The server machine rdsrv309 has a vncserver running as a service on desktop :2
It is run under clasrun user and it is bound to localhost. In order to access it from remote, an ssh tunnel needs to be setup on the client machine.

ssh -L <localClientPort>:localhost:5902 -C -N -l clasrun rdsrv309.slac.stanford.edu

Where <localClientPort> can be whatever port that is not used on the client side. To connect to the vncserver just connect to localhost:<localClientPort> after opening the ssh tunnel.

Basic SVT DAQ instructions for the test bed

Before starting working make sure that the heatsink of the FEB is facing upward and the fan is pointing to it. If the FEB overheats it will reset itself (you will see all the leds halfway up).

Start working/developing:

Connect to the rdusr219 machine
Setup the general environment:
```
source /u1/hps/setup_env.sh
```
Start the conda environment:
```
conda activate rogue_5_8_0
```
Run the software from this folder.
```
cd /u1/hps/server/heavy-photon-daq/software/scripts
python SvtDaqGui.py --local --env SLAC21BOT [--epicsEn]
```
The env command line is to tell which network configuration to load. The network configuration that gets loaded is stored in python/hps/constants.py
Check the FebLinkStatus. In the RCE test, you should see FebFpga[0] and check the Link[0] state. If False, Read the variables again (click Read). If now True, means we are talking to the Feb. After the link is established one can load the configuration.

Load the configuration for the system configuration (back end: ATCA (advanced telecommunication computing architecture) and front end (FEB+Hybrid+APVs)) . Click on the HpsSvtDaqRoot tab and load settings
```
/u1/hps/daq/heavy-photon-daq/software/config/rce-test.yml
```
There is a copy of this configuration file in /u1/hps/server/heavy-photon-daq/software/config/

Find the FEB Link

It might be unclear which link channel goes from the RTM to the FEB(s). In the test area, where a single feb is attached to the RTM, one can put in the link map the fill list of RTM channels (12) giving the full list of tuples.
Each index of the list is the feb address of the feb array and in each tuple (X,Y) X is the cob number (0,1) and Y is the link number 0-11.
Example:
Provide

[(0,1),(0,2),...]

To activate all the links and then check which one reads back to True when reading the variables. The channel mapping between the flange boards and the RTM has been performed on 8th April 2021 and is summarised here:
Flange Board to RTM Slow Control Channel Mapping

Setting on the Hybrids

In Variables tab navigate to the FebArray→FebCore→FebConfig and HybridPwrEn the 0 and 1 (the power supply will not be able to sustain 3 hybrids, but should be OK for 2).
After turning on the Hybrids, load the configuration again to be sure you are sending the right config to them.

Sending reset commands to the Hybrids

Navigate to "Commands" tab and in PcieTiDtmArray click the sequence:

1) ApvClkAlign (to set all the APVs to the same phase)
2) ApvReset101. (reset and start to wait for a trigger signal)

Start local svt data taking

With the hybrids configured and synced.

1) Go to HpsSvtDaqRoot tab→Browse and then set the name of the file ( it will set the output name with data and time automatically)
2) Cick on Open. You will see File Open → True when ready.
3) Set run Rate to something reasonable (10Hz should be OK)
4) Click on Run State and select "Running". When successfully running you'll see Run Count going up.
5) When done with local data taking, first set Run State to "Stopped" and Close the file.

Configuration files used to configure rogue.
The 2019 CODA (central DAQ software at jLab) configuration files for HPS DAQ are stored on jLab machines at

/usr/clas12/release/1.4.0/parms/trigger/HPS/Run2019/

The general configuration for the DAQ usually ends in .trg, the relevant block for SVT is under the #SVT Config block in one of those files.
This should point to something like ..../svt/svt_config.cnf, where the configuration should be pointed.

To access this information, one needs to ssh to the clonfarm machines. In particular, clonfarm2 and clonfarm3 are the SVT DAQ machines.

Running the SVT Daq using CODA

A setup is available to run the SVT DAQ using CODA. As of 25th of June the following configuration has been tested and it is working:

Running using rdsrv309 only
1 FEB ON but Hybrids OFF
Threshold are not loaded as Daq mapping needs to be taken into account
Random trigger rate at 5 kHz

Here are the instructions on how to setup CODA

ssh as clasrun to rdsrv309
The default shell for clasrun is tcsh. In order to start rogue one has to change to /bin/bash. Then setup the conda environment as shown in "Basic SVT DAQ instructions for the test bed"
```
/bin/bash
source /u1/hps/setup_env.sh
conda activate rogue_5_8_0
cd /u1/hps/server/heavy-photon-daq/software/scripts
```
Run the Rogue instance for data taking. Epics connection has not been tested yet. So is kept off at the moment. in order for the coda system to run, both COBs need to be enabled. So use the full configuration even if a single FEB is attached. In the case the second COB is not populated, the system is smart enough to figure it out

python SvtCodaRun.py --local --env SLAC21

This command will bring up the Rogue Gui and start the rogue server

Open a new terminal as clasun. To start CODA it's sufficient to type

runcontrol -rocs

This will bring up the GUI interface. The following setup will guide you through the various transitions.

DEBUG:

Connect via minicom to the crate controller:

rdusr219 >$ minicom usb2

It will dump the info every couple of minutes. Here below is an example of a BOOT failure :

================================================================================
DPM0 RCE0 BSI:
--------------------------------------------------------------------------------
000: 02 00 00 00 01 00 00 00  08 00 56 00 46 07 00 00 
010: 00 00 00 00 72 63 65 2d  76 32 30 31 36 2e 35 00 
020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
030: 00 00 00 00 35 31 30 34  00 00 00 00 00 00 00 00 
040: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
050: 00 00 00 00 00 14 b0 fe  72 08 20 80 07 46 00 00 
060: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
070: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
080: ...
140: 70 f5 04 7f 00 00 00 67  00 00 02 01 6c 62 6e 65 
150: 31 5f 30 31 30 32 00 00  00 00 00 00 00 00 00 00 
160: 00 00 00 00 00 00 00 00  00 00 00 00 03 00 00 00 
170: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
180: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
190: ...
7f0: 00 00 00 00 00 00 00 00  00 00 00 00 ee 00 00 00 

--------------------------------------------------------------------------------
DPM0 RCE0
  BSI Status: 0xee
  BSI version number: 0x00000002
  UBOOT version:   rce-v2016.5
  DAT version:     5104
  Zynq Device DNA: 0x80200872 feb01400
  Zynq eFUSE User: 0x00000000 00004607
  PHY Type: 0x00000001
    Lane 0: (1) 1G
    Lane 1: (0) Unused
    Lane 2: (0) Unused
    Lane 3: (0) Unused
  RCE MAC: 07:46:00:56:00:08
  CMB ID: 670000007f04f570
  Cluster Address: 0x01020000
      Version    : 0x01
      Cluster No.: 0x02
      Bay No.    : 0x00
      Element No.: 0x00
  Cluster Group Name: lbne1_0102
  External Interconnect Type: 0x00000003
DPM0 RCE2 BSI:
--------------------------------------------------------------------------------
000: 02 00 00 00 01 00 00 00  08 00 56 00 46 08 00 00 
010: 00 00 00 00 72 63 65 2d  76 32 30 31 36 2e 35 00 
020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
030: 00 00 00 00 35 31 30 34  00 00 00 00 00 00 00 00 
040: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
050: 00 00 00 00 00 1c b0 fe  72 08 20 80 08 46 00 00 
060: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
070: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
080: ...
140: 70 f5 04 7f 00 00 00 67  02 00 02 01 6c 62 6e 65 
150: 31 5f 30 31 30 32 00 00  00 00 00 00 00 00 00 00 
160: 00 00 00 00 00 00 00 00  00 00 00 00 03 00 00 00 
170: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
180: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
190: ...
7f0: 00 00 00 00 00 00 00 00  00 00 00 00 ee 00 00 00 

--------------------------------------------------------------------------------
DPM0 RCE2
  BSI Status: 0xee
  BSI version number: 0x00000002
  UBOOT version:   rce-v2016.5
  DAT version:     5104
  Zynq Device DNA: 0x80200872 feb01c00
  Zynq eFUSE User: 0x00000000 00004608
  PHY Type: 0x00000001
    Lane 0: (1) 1G
    Lane 1: (0) Unused
    Lane 2: (0) Unused
    Lane 3: (0) Unused
  RCE MAC: 08:46:00:56:00:08
  CMB ID: 670000007f04f570
  Cluster Address: 0x01020002
      Version    : 0x01
      Cluster No.: 0x02
      Bay No.    : 0x00
      Element No.: 0x02
  Cluster Group Name: lbne1_0102
  External Interconnect Type: 0x00000003
DPM1 RCE0 BSI:
--------------------------------------------------------------------------------
000: 02 00 00 00 01 00 00 00  08 00 56 00 45 ff 00 00 
010: 00 00 00 00 72 63 65 2d  76 32 30 31 36 2e 35 00 
020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
030: 00 00 00 00 35 31 30 34  00 00 00 00 00 00 00 00 
040: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
050: 00 00 00 00 00 1c b0 c1  0b 8c b0 80 ff 45 00 00 
060: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
070: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
080: ...
140: 70 31 c8 7e 00 00 00 a8  00 01 02 01 6c 62 6e 65 
150: 31 5f 30 31 30 32 00 00  00 00 00 00 00 00 00 00 
160: 00 00 00 00 00 00 00 00  00 00 00 00 03 00 00 00 
170: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
180: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
190: ...
7f0: 00 00 00 00 00 00 00 00  00 00 00 00 ee 00 00 00 

--------------------------------------------------------------------------------
DPM1 RCE0
  BSI Status: 0xee
  BSI version number: 0x00000002
  UBOOT version:   rce-v2016.5
  DAT version:     5104
  Zynq Device DNA: 0x80b08c0b c1b01c00
  Zynq eFUSE User: 0x00000000 000045ff
  PHY Type: 0x00000001
    Lane 0: (1) 1G
    Lane 1: (0) Unused
    Lane 2: (0) Unused
    Lane 3: (0) Unused
  RCE MAC: ff:45:00:56:00:08
  CMB ID: a80000007ec83170
  Cluster Address: 0x01020100
      Version    : 0x01
      Cluster No.: 0x02
      Bay No.    : 0x01
      Element No.: 0x00
  Cluster Group Name: lbne1_0102
  External Interconnect Type: 0x00000003
DPM1 RCE2 BSI:
--------------------------------------------------------------------------------
000: 02 00 00 00 01 00 00 00  08 00 56 00 46 00 00 00 
010: 00 00 00 00 72 63 65 2d  76 32 30 31 36 2e 35 00 
020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
030: 00 00 00 00 35 31 30 34  00 00 00 00 00 00 00 00 
040: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
050: 00 00 00 00 00 1c b0 c1  0b 8c b0 80 00 46 00 00 
060: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
070: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
080: ...
140: 70 31 c8 7e 00 00 00 a8  02 01 02 01 6c 62 6e 65 
150: 31 5f 30 31 30 32 00 00  00 00 00 00 00 00 00 00 
160: 00 00 00 00 00 00 00 00  00 00 00 00 03 00 00 00 
170: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
180: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
190: ...
7f0: 00 00 00 00 00 00 00 00  00 00 00 00 ee 00 00 00 

--------------------------------------------------------------------------------
DPM1 RCE2
  BSI Status: 0xee
  BSI version number: 0x00000002
  UBOOT version:   rce-v2016.5
  DAT version:     5104
  Zynq Device DNA: 0x80b08c0b c1b01c00
  Zynq eFUSE User: 0x00000000 00004600
  PHY Type: 0x00000001
    Lane 0: (1) 1G
    Lane 1: (0) Unused
    Lane 2: (0) Unused
    Lane 3: (0) Unused
  RCE MAC: 00:46:00:56:00:08
  CMB ID: a80000007ec83170
  Cluster Address: 0x01020102
      Version    : 0x01
      Cluster No.: 0x02
      Bay No.    : 0x01
      Element No.: 0x02
  Cluster Group Name: lbne1_0102
  External Interconnect Type: 0x00000003
DPM2 RCE0 BSI:
--------------------------------------------------------------------------------
000: 02 00 00 00 01 00 00 00  08 00 56 00 45 b1 00 00 
010: 00 00 00 00 72 63 65 2d  76 32 30 31 36 2e 35 00 
020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
030: 00 00 00 00 35 31 30 34  00 00 00 00 00 00 00 00 
040: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
050: 00 00 00 00 00 5c b0 71  48 ca 08 80 b1 45 00 00 
060: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
070: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
080: ...
140: 70 f9 c6 7e 00 00 00 34  00 02 02 01 6c 62 6e 65 
150: 31 5f 30 31 30 32 00 00  00 00 00 00 00 00 00 00 
160: 00 00 00 00 00 00 00 00  00 00 00 00 03 00 00 00 
170: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
180: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
190: ...
7f0: 00 00 00 00 00 00 00 00  00 00 00 00 ee 00 00 00 

--------------------------------------------------------------------------------
DPM2 RCE0
  BSI Status: 0xee
  BSI version number: 0x00000002
  UBOOT version:   rce-v2016.5
  DAT version:     5104
  Zynq Device DNA: 0x8008ca48 71b05c00
  Zynq eFUSE User: 0x00000000 000045b1
  PHY Type: 0x00000001
    Lane 0: (1) 1G
    Lane 1: (0) Unused
    Lane 2: (0) Unused
    Lane 3: (0) Unused
  RCE MAC: b1:45:00:56:00:08
  CMB ID: 340000007ec6f970
  Cluster Address: 0x01020200
      Version    : 0x01
      Cluster No.: 0x02
      Bay No.    : 0x02
      Element No.: 0x00
  Cluster Group Name: lbne1_0102
  External Interconnect Type: 0x00000003
DPM2 RCE2 BSI:
--------------------------------------------------------------------------------
000: 02 00 00 00 01 00 00 00  08 00 56 00 45 b2 00 00 
010: 00 00 00 00 72 63 65 2d  76 32 30 31 36 2e 35 00 
020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
030: 00 00 00 00 35 31 30 34  00 00 00 00 00 00 00 00 
040: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
050: 00 00 00 00 00 54 b0 71  48 ca 08 80 b2 45 00 00 
060: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
070: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
080: ...
140: 70 f9 c6 7e 00 00 00 34  02 02 02 01 6c 62 6e 65 
150: 31 5f 30 31 30 32 00 00  00 00 00 00 00 00 00 00 
160: 00 00 00 00 00 00 00 00  00 00 00 00 03 00 00 00 
170: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
180: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
190: ...
7f0: 00 00 00 00 00 00 00 00  00 00 00 00 ee 00 00 00 

--------------------------------------------------------------------------------
DPM2 RCE2
  BSI Status: 0xee
  BSI version number: 0x00000002
  UBOOT version:   rce-v2016.5
  DAT version:     5104
  Zynq Device DNA: 0x8008ca48 71b05400
  Zynq eFUSE User: 0x00000000 000045b2
  PHY Type: 0x00000001
    Lane 0: (1) 1G
    Lane 1: (0) Unused
    Lane 2: (0) Unused
    Lane 3: (0) Unused
  RCE MAC: b2:45:00:56:00:08
  CMB ID: 340000007ec6f970
  Cluster Address: 0x01020202
      Version    : 0x01
      Cluster No.: 0x02
      Bay No.    : 0x02
      Element No.: 0x02
  Cluster Group Name: lbne1_0102
  External Interconnect Type: 0x00000003
DPM3 RCE0 BSI:
--------------------------------------------------------------------------------
000: 02 00 00 00 01 00 00 00  08 00 56 00 46 01 00 00 
010: 00 00 00 00 72 63 65 2d  76 32 30 31 36 2e 35 00 
020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
030: 00 00 00 00 35 31 30 34  00 00 00 00 00 00 00 00 
040: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
050: 00 00 00 00 00 14 b0 fe  72 08 60 80 01 46 00 00 
060: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
070: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
080: ...
140: 70 3e aa 7e 00 00 00 1e  00 03 02 01 6c 62 6e 65 
150: 31 5f 30 31 30 32 00 00  00 00 00 00 00 00 00 00 
160: 00 00 00 00 00 00 00 00  00 00 00 00 03 00 00 00 
170: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
180: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
190: ...
7f0: 00 00 00 00 00 00 00 00  00 00 00 00 ee 00 00 00 

--------------------------------------------------------------------------------
DPM3 RCE0
  BSI Status: 0xee
  BSI version number: 0x00000002
  UBOOT version:   rce-v2016.5
  DAT version:     5104
  Zynq Device DNA: 0x80600872 feb01400
  Zynq eFUSE User: 0x00000000 00004601
  PHY Type: 0x00000001
    Lane 0: (1) 1G
    Lane 1: (0) Unused
    Lane 2: (0) Unused
    Lane 3: (0) Unused
  RCE MAC: 01:46:00:56:00:08
  CMB ID: 1e0000007eaa3e70
  Cluster Address: 0x01020300
      Version    : 0x01
      Cluster No.: 0x02
      Bay No.    : 0x03
      Element No.: 0x00
  Cluster Group Name: lbne1_0102
  External Interconnect Type: 0x00000003
DPM3 RCE2 BSI:
--------------------------------------------------------------------------------
000: 02 00 00 00 01 00 00 00  08 00 56 00 46 02 00 00 
010: 00 00 00 00 72 63 65 2d  76 32 30 31 36 2e 35 00 
020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
030: 00 00 00 00 35 31 30 34  00 00 00 00 00 00 00 00 
040: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
050: 00 00 00 00 00 1c b0 fe  72 08 20 80 02 46 00 00 
060: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
070: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
080: ...
140: 70 3e aa 7e 00 00 00 1e  02 03 02 01 6c 62 6e 65 
150: 31 5f 30 31 30 32 00 00  00 00 00 00 00 00 00 00 
160: 00 00 00 00 00 00 00 00  00 00 00 00 03 00 00 00 
170: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
180: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
190: ...
7f0: 00 00 00 00 00 00 00 00  00 00 00 00 ee 00 00 00 

--------------------------------------------------------------------------------
DPM3 RCE2
  BSI Status: 0xee
  BSI version number: 0x00000002
  UBOOT version:   rce-v2016.5
  DAT version:     5104
  Zynq Device DNA: 0x80200872 feb01c00
  Zynq eFUSE User: 0x00000000 00004602
  PHY Type: 0x00000001
    Lane 0: (1) 1G
    Lane 1: (0) Unused
    Lane 2: (0) Unused
    Lane 3: (0) Unused
  RCE MAC: 02:46:00:56:00:08
  CMB ID: 1e0000007eaa3e70
  Cluster Address: 0x01020302
      Version    : 0x01
      Cluster No.: 0x02
      Bay No.    : 0x03
      Element No.: 0x02
  Cluster Group Name: lbne1_0102
  External Interconnect Type: 0x00000003
 DTM RCE0 BSI:
--------------------------------------------------------------------------------
000: 02 00 00 00 01 00 00 00  08 00 56 00 46 0e 00 00 
010: 00 00 00 00 72 63 65 2d  76 32 30 31 36 2e 35 00 
020: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
030: 00 00 00 00 35 31 30 34  00 00 00 00 00 00 00 00 
040: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
050: 00 00 00 00 00 14 b0 fe  72 20 31 80 0e 46 00 00 
060: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
070: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
080: ...
140: 70 95 90 7e 00 00 00 83  00 04 02 01 6c 62 6e 65 
150: 31 5f 30 31 30 32 00 00  00 00 00 00 00 00 00 00 
160: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
170: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
180: ...
240: 02 00 00 00 02 00 00 00  02 00 00 00 02 00 00 00 
250: 02 00 00 00 02 00 00 00  02 00 00 00 02 00 00 00 
260: 02 00 00 00 06 00 01 00  06 00 01 00 06 00 01 00 
270: 06 00 01 00 06 00 01 00  06 00 01 00 06 00 01 00 
280: 06 00 01 00 06 00 01 00  06 00 01 00 06 00 01 00 
290: 06 00 01 00 06 00 01 00  02 00 00 00 06 00 00 00 
2a0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
2b0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
2c0: ...
300: 01 00 00 80 02 00 00 00  c0 a8 02 6e c0 a8 02 dc 
310: ff ff ff 00 00 00 00 00  00 00 00 80 00 00 00 00 
320: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
330: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
340: ...
400: 01 8c 11 8d 21 8e 31 8f  41 90 51 91 61 92 71 93 
410: 81 94 91 95 a1 89 b1 8a  c1 8b d1 9f e1 9f 00 00 
420: 00 8c 10 8d 20 8e 30 8f  40 90 50 91 60 92 70 93 
430: 80 94 90 95 a0 89 b0 8a  c0 8b d0 9f e0 9f 00 00 
440: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
450: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
460: ...
600: 01 00 00 80 00 00 00 00  00 00 00 00 00 00 00 00 
610: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
620: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
630: ...
7f0: 00 00 00 00 00 00 00 00  00 00 00 00 ee 00 00 00 

--------------------------------------------------------------------------------
 DTM RCE0
  BSI Status: 0xee
  BSI version number: 0x00000002
  UBOOT version:   rce-v2016.5
  DAT version:     5104
  Zynq Device DNA: 0x80312072 feb01400
  Zynq eFUSE User: 0x00000000 0000460e
  PHY Type: 0x00000001
    Lane 0: (1) 1G
    Lane 1: (0) Unused
    Lane 2: (0) Unused
    Lane 3: (0) Unused
  RCE MAC: 0e:46:00:56:00:08
  CMB ID: 830000007e909570
  Cluster Address: 0x01020400
      Version    : 0x01
      Cluster No.: 0x02
      Bay No.    : 0x04
      Element No.: 0x00
  Cluster Group Name: lbne1_0102
  External Interconnect Type: 0x00000000
--------------------------------------------------------------------------------
 Port | (     Raw) - Group            Type |
--------------------------------------------------------------------------------
   0  | (00000002) -     0        1G-Lane0 | 
   1  | (00000002) -     0        1G-Lane0 | 
   2  | (00000002) -     0        1G-Lane0 | 
   3  | (00000002) -     0        1G-Lane0 | 
   4  | (00000002) -     0        1G-Lane0 | 
   5  | (00000002) -     0        1G-Lane0 | 
   6  | (00000002) -     0        1G-Lane0 | 
   7  | (00000002) -     0        1G-Lane0 | 
   8  | (00000002) -     0        1G-Lane0 | 
   9  | (00010006) -     1            XAUI | 
  10  | (00010006) -     1            XAUI | 
  11  | (00010006) -     1            XAUI | 
  12  | (00010006) -     1            XAUI | 
  13  | (00010006) -     1            XAUI | 
  14  | (00010006) -     1            XAUI | 
  15  | (00010006) -     1            XAUI | 
  16  | (00010006) -     1            XAUI | 
  17  | (00010006) -     1            XAUI | 
  18  | (00010006) -     1            XAUI | 
  19  | (00010006) -     1            XAUI | 
  20  | (00010006) -     1            XAUI | 
  21  | (00010006) -     1            XAUI | 
  22  | (00000002) -     0        1G-Lane0 | 
  23  | (00000006) -     0            XAUI | 
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
 Shelf IP Information - Done: 1
--------------------------------------------------------------------------------
      Port Membership: 80000001
  Port Ingress Policy: 00000002
           Group Base: 192.168.2.110
            Group End: 192.168.2.220
          Subnet Mask: 255.255.255.0
              Gateway: 0.0.0.0
--------------------------------------------------------------------------------
|-----------------------------------------------------|
| Fabric Map - Local Slot 2  - HUB: No                |
|-----------------------------------------------------|
| Local Slot-Ch -> Remote  | Local Slot-Ch -> Remote  |
|--------------------------|--------------------------|
| S01-C01 (P12) -> S02-C01 | S01-C02 (P13) -> S02-C02 |
| S01-C03 (P14) -> S02-C03 | S01-C04 (P15) -> S02-C04 |
| S01-C05 (P16) -> S02-C05 | S01-C06 (P17) -> S02-C06 |
| S01-C07 (P18) -> S02-C07 | S01-C08 (P19) -> S02-C08 |
| S01-C09 (P20) -> S02-C09 | S01-C10 (P21) -> S02-C10 |
| S01-C11 (P09) -> S02-C11 | S01-C12 (P10) -> S02-C12 |
| S01-C13 (P11) -> S02-C13 | S01-C14 (P31) -> S02-C14 |
| S01-C15 (P31) -> S02-C15 |
|--------------------------|--------------------------|
| S02-C01 (P12) -> S01-C01 | S02-C02 (P13) -> S01-C02 |
| S02-C03 (P14) -> S01-C03 | S02-C04 (P15) -> S01-C04 |
| S02-C05 (P16) -> S01-C05 | S02-C06 (P17) -> S01-C06 |
| S02-C07 (P18) -> S01-C07 | S02-C08 (P19) -> S01-C08 |
| S02-C09 (P20) -> S01-C09 | S02-C10 (P21) -> S01-C10 |
| S02-C11 (P09) -> S01-C11 | S02-C12 (P10) -> S01-C12 |
| S02-C13 (P11) -> S01-C13 | S02-C14 (P31) -> S01-C14 |
| S02-C15 (P31) -> S01-C15 |
|--------------------------|--------------------------|
================================================================================
================================================================================
| SFP   |LOS| Vendor           | Part             |
--------------------------------------------------------------------------------
| Ch. 0 | Y | FS               | SFP-GB-GE-T      |
| Ch. 4 | Y | Fiberstore       | SFP-10GSR-85     |
================================================================================
================================================================================
|  BAY |Pre|Ena|Vok| Voltage | Current |  Power  | Alloc | ID               |
--------------------------------------------------------------------------------
| DPM0 | Y | Y | Y | 11.898V |  1653mA | 19.667W |  0.0W | 670000007f04f570 |
| DPM1 | Y | Y | Y | 11.910V |  1680mA | 20.008W | 35.0W | a80000007ec83170 |
| DPM2 | Y | Y | Y | 11.910V |  1623mA | 19.329W | 35.0W | 340000007ec6f970 |
| DPM3 | Y | Y | Y | 11.982V |  1680mA | 20.129W |  0.0W | 1e0000007eaa3e70 |
|  DTM | Y | Y | Y | 11.892V |   686mA |  8.157W | 15.0W | 830000007e909570 |
|  RTM | Y | Y |   | 11.898V |   550mA |  6.543W | 25.0W | c0000001270ad170 |
|  CEN | Y | Y | Y | 11.976V |  2046mA | 24.502W | 24.0W | 690000007e910c70 |
================================================================================
================================================================================
|       RCE  |Ena|Rst|Rdy|Dne|Vok|BTemp|JTemp|      RCE State |
--------------------------------------------------------------------------------
| DPM0: RCE0 | Y | N | Y | Y | Y | 39C | 59C |ee   NOT_BOOTED |
|       RCE2 | Y | N | Y | Y | Y | 44C |     |ee   NOT_BOOTED |
| DPM1: RCE0 | Y | N | Y | Y | Y | 47C | 66C |ee   NOT_BOOTED |
|       RCE2 | Y | N | Y | Y | Y | 51C | 67C |ee   NOT_BOOTED |
| DPM2: RCE0 | Y | N | Y | Y | Y | 39C | 67C |ee   NOT_BOOTED |
|       RCE2 | Y | N | Y | Y | Y | 48C | 62C |ee   NOT_BOOTED |
| DPM3: RCE0 | Y | N | Y | Y | Y | 36C | 59C |ee   NOT_BOOTED |
|       RCE2 | Y | N | Y | Y | Y | 43C |     |ee   NOT_BOOTED |
|  DTM: RCE0 | Y | N | Y | Y | Y | 36C | 47C |ee   NOT_BOOTED |
================================================================================
================================================================================
| COB\GPIO | DET | RST | RDY | DNE |  VOK |  PWR |
--------------------------------------------------------------------------------
|     DPM0 | 0x5 | 0xa | 0x5 | 0x5 | 0xff | 0x00 |
|     DPM1 | 0x5 | 0xa | 0x5 | 0x5 | 0xff | 0xaf |
|     DPM2 | 0x5 | 0xa | 0x5 | 0x5 | 0xff | 0xaf |
|     DPM3 | 0x5 | 0xa | 0x5 | 0x5 | 0xff | 0x00 |
|      DTM | 0x1 | 0xe | 0x1 | 0x1 | 0xff | 0x4b |
|      CEN |     |     |     |     | 0xff | 0x78 |
================================================================================
================================================================================
| RTM | Pre | Ena | Alloc |   Type   | BTemp | JTemp |
--------------------------------------------------------------------------------
| RTM |  Y  |  Y  | 25.0W | 00000003 |   35C |       |
| RMB |  N  |  N  |       |          |       |       |
================================================================================
================================================================================
| CEN | Pre | Ena | VOK | BTemp | JTemp |
--------------------------------------------------------------------------------
| CEN |  Y  |  Y  |  Y  |   27C |   71C |
================================================================================

Connect via minicom to the dtm:

rdusr219>$ minicom usb0

Dump the registers of the crate

rdusr219>$ cob_dump --all dev-crate1-sm

Setup of RCE bootloader for diskless booting: (based on this page)

ssh (or minicom via usb) to DTM (192.168.2.221)

minicom bay<bay>.<RCE> (bay is 0-3 <4 is the DTM> and RCE is 0 or 2 <NOT 1>)

From rdusr219 (with SDK env): cob_rce_reset 192.168.1.254/2/<bay>/<RCE>

Interrupt boot sequence by pressing any key in minicom

Apply the default u-boot environment:

zynq-uboot> env default -a

zynq-uboot> setenv modeboot nfsboot

zynq-uboot> setenv bootdelay 3

zynq-uboot> saveenv

zynq-uboot> reset

In order to down/upload to github, set up a tunnel between rhel6-64 and rdusr219. See Proto-DUNE: How to setup a temporary tunnel for github for a recipe.

The HPS RCEs boot in "diskless mode" via NFS. For info about this, see RCE Diskless Node (NFS).

RCE network names:

data DPMs: eth1_test_dpm<1-6>

control DPM: eth1_test_dpm7

DTM: eth1_test_dtm1

TO DO:

Upgrade sw to rogue, requires minor tweaks to fw (Ben)

Update to new config format used by rogue (Cameron and Omar)

Switch to JLab supported TI interface to use event builder supported at JLab and respin RTM (Ryan)

Build sw to take event blocks from data DPM fw and build EVIO blocks for CODA (JJ)

Install CODA at SLAC (Cameron and Omar)

Write EPICS sw (Cameron and Omar)

Search/Navigation:

Related: