GT Readout Platform: Testing and Bring-Up

← GT Readout Platform

Resistance measurements with cable attached

• Cable was produced in-house, see TIDIDECS-114 - Getting issue details... STATUS
• There are no serial number on the cables, so for now only using one and assigning it "Cable 01"
• Measuring between banana connector on the cable and test points on the board

Resistance measurements with analog and digital boards mated

Measuring on test points on the analog board.

Voltage measurements with analog and digital boards mated

• FPGA is not programmed.
• All power rails are not powered up yet on the digital board, see Issue #3.
• All power rails on the analog board are disabled by default due to the pull-down resistor R21.

Simplified virtual I/O gateware for the FPGA

• Vivado v2023.1 full project: gt-readout-vio-test.zip
• Bitfile: gt-readout-vio-test.bit.zip
• Debug file: gt-readout-vio-test.ltx

Only digital board

• Before programming the FPGA
  • Current consumption: 327 mA at 24 V on P24V0D
  • Temperature: Stable at around 28C with fan blowing on the bottom of the board (not FPGA side)
• After programming the FPGA
  • Current consumption: 336 mA at 24 V on P24V0D
  • Temperature: Stable at around 29C with fan blowing on the bottom of the board (not FPGA side)
• Virtual IO
  • prsntAnalogL and prsntCarrierL both are high indicating no boards are plugged in
  • hwRev is correct: 0b0001
  • hardwareIo[7:4] works

Digital and analog board

• After programming the FPGA
  • Current consumption: 338 mA at 24 V on P24V0D
  • Temperature stable at around 32C now due to analog board in the way of the airflow to the digital board
• prsntAnalogL is now low indicating that the analog board is plugged on
• Toggling pwrEnable HIGH should turn on all the power supplies on the analog board
  • This includes supplies on the P24V0A power rail also so this must be provided from the external supply
  • Toggling the pwrEnable HIGH
  • Current consumption:
    • 440 mA at 24 V on P24V0D
    • 47 mA at 24 V on P24V0A
  • Probing on analog board:
    • P5V0A FIRST = 4.978 V
    • P5V0D FIRST = 5.013 V
    • P1V8A FIRST = 1.816 V
    • P1V8D FIRST = 1.803 V
    • P1V8D = 0.494 V
    • P3V3D = 0 V
    • P3V3A = 0 V
    • P1V8A = 0 V
    • P2V5 AVDD = 0 V
    • P2V5A SINK = 0 V
    • P0V8 DC BIAS = 0 V
  • → Something is not working on the analog board and the power sequencing!
    • Issue identified, see Issue #1
    • Gave all three analog boards to the workshop on 2024-04-26 for rework
• Reworked analog board received back on 2024-04-29
  • Testing 38-C00-01 and 39-C00-01
    • Current consumption after programming and enabling power to analog board:
      • 446 mA at 24 V on P24V0D
      • 192 mA at 24 V on P24V0A
    • Probing on analog board:
      • P5V0A FIRST = 5.003 V
      • P5V0D FIRST = 4.990 V
      • P1V8A FIRST = 1.807 V
      • P1V8D FIRST = 1.801 V
      • P1V8D = 1.808 V
      • P3V3D = 3.280 V
      • P3V3A = 3.294 V
      • P1V8A = 1.800 V
      • P2V5 AVDD = 2.517 V
      • P2V5A SINK = 2.529 V
      • P0V8 DC BIAS = 0.807 V
      • P1V3 A1VDD = 1.303 V
      • P1V3 A2VDD = 1.303 V
      • P1V3 A3VDD = 1.305 V
      • P1V3 A4VDD = 1.302 V
      • P0V6 A1SINK = 0.616 V
      • P0V6 A2SINK = 0.624 V
      • P0V6 A4SINK = 0.616 V
      • P0V6 A3SINK = 0.619 V
    • → All looks good!
    • Temperature stable at around 36C with fan blowing on the top of the analog board, not directly on the FPGA

Digital, analog and carrier board

• prsntCarrierL is now low indicating that the carrier board is plugged on

Mounted LEAP transceiver

• Mounted a LEAP transceiver on board 38-C00-01 on 2024-04-30
  • Instructions from Detector assembly instructions#Connectingtransceiver
• Only powering up the digital board by itself:
  • FPGA is not programmed
  • 481 mA at 24 V on P24V0D
    • Compare this to 327 mA without the transceiver

Mounted fiber optical pigtail

• Mounted a fiber optical pigtail on board 38-C00-01 on 2024-05-01
  • Instructions from Detector assembly instructions#Connectingthetransceivercable

Mounted boards to cooling block

• Mounted digital board 38-C00-01 and analog board 39-C00-01 on 2024-05-1
• Started with the analog board:
  • Cut two pieces of 1 mm thick thermal pad that covered most of the components
  • Added additional 1 mm thick thermal pad over the regulators on the left side of the board as shown below
  • Added some 3 mm thick thermal pad over the regulators on the right side of the board
• Screwed the analog board into the cooling block
• Then the digital board:
  • Cut one 1 mm thick thermal pad for the FPGA
  • Cut one 1 mm thick thermal pad for the big and tall regulator
  • Cut one 1 mm thick thermal pad covering all the other regulators on the left side of the board as shown below
  • Cut some 3 mm thick thermal pads and placed over clock components on the right side of the board
• Carefully plugged in the digital board into the analog board through the connector
• Tightened the screws on the digital board but not all the way since the distance is smaller between the two boards on the cooling block
  • Tightening more would probably cause a lot of stress on the connector which sets the height of 17.5 mm between the board
• Powered everything up
• LEAP transceiver gets too warm after only about 2 min of being on even with a fan blowing above
  • This needs to be connected to the cooling block somehow
  • Could also use the LEAP with a heatsink for use in the lab with a fan
• The original cooling plate for the LEAP transceiver does not fit as is because it was based on the previous board design
  • Hacked it together by drilling and tapping a hole into the copper cooling block
  • Added a 3 mm thermal pad between the LEAP transceiver and the plate to fill out the gap
  • Seems to work OK, with the plate heating up slowly and the LEAP not feeling too hot

First full system test

• Testing started on 2024-05-02
• Running the simple virtual I/O gateware first to see where the internal FPGA temperature stabilizes at
• Power draw after enabling power to the analog board:
  • 609 mA at 24 V on P24V0D → 14.6 W
  • 193 mA at 24 V on P24V0A → 4.6 W
• Temperature stable at around 34°C
• Switching to real gateware: https://github.com/slaclab/epix-uhr-100kHz-dev
  • Temperature stable at around 36°C
  • 585 mA at 24 V on P24V0D → 14 W
  • 0 mA at 24 V on P24V0A → Power enable signal is low by default
  • There's no connection between the software and the hardware... that would have been too lucky!
  • Moving on to check PCIe card and fibers

Test of PCIe card and fibers

• Using a KCU105 and the example PGP project: https://github.com/slaclab/Simple-PGPv4-KCU105-Example
  • Connected up to the first fiber in the breakout
  • Working!
• Moving the KCU105 fiber pair to another pair on the PCIe does not work.
  • It seems that for this example, the first fiber lane of the PCIe card must be connected to the KCU105
  • Can't find any information on how the PCIe card manages multiple fiber lanes and how this is configured...
  • Ideally it would be useful to test all 8 possible fiber lanes on the PCIe card to make sure the fiber connections are correct
  • → Errata: this can be changed with the "--lane" argument to the devGui.py software, see https://github.com/slaclab/Simple-PGPv4-KCU105-Example/blob/v2.7.0/software/scripts/devGui.py#L42
• Warning message when running PgpMonitor.py 

  • WARNING: pciServer.AxiPcieCore.numDmaLanes = 4 != pciServer.AxiPcieCore.AxiVersion.DMA_SIZE_G = 8

  • The app also only displays 4 lanes, while we should have 8 because of the two QSFP and 8 fiber pairs
  • There is a parameter that defaults to 4, change this when running the PgpMonitor:

    • python PgpMonitor.py --numLane 8

• Looking at the different lanes in the PGP monitor showed that none of them were showing a good link
• With the help from Dawood, it was shown that the MTP adapter that was used is not the correct one
  • The one that was used flipped the male MTP-24 cable from the LEAP relative to the female MTP-24
    • "Opposed Key" adapter (black): https://www.fs.com/products/48508.html?attribute=6117&id=109153
  • He had a correct adapter where it just goes straight through
    • "Aligned Key" adapter (gray): https://www.fs.com/products/12072.html?attribute=6118&id=109156
  • After replacing the adapter we can now see lane 5 is working and the application GUI is able to connect and read register values from the GT Readout FPGA!
• Values of some registers:
  • LeapXcvr:
    • Root.Core.LeapXcvr.TxLower.TxTemp = 52.3
    • Root.Core.LeapXcvr.TxLower.TxVcc3p3 = 3.2968
    • Root.Core.LeapXcvr.RxUpperPage01.InputOpticalPowerMonitor[0] = -16.0
    • Root.Core.LeapXcvr.RxLower.RxVcc3p3 = 3.306
  • SysMon
    • Root.Core.AxiSysMonUltraScale.Temperature = 35.397
    • Root.Core.AxiSysMonUltraScale.VccInt = 0.848
    • Root.Core.AxiSysMonUltraScale.VccAux = 1.798
• Enabling the GtReadoutBoardCtrl module
  • No carrier board is installed
  • All values look good
  • pwrEnableAnalogBoard works
    • 702 mA at 24 V on P24V0D → 16.85 W
    • 193 mA at 24 V on P24V0A → 4.6 W
  • Reading out the board serial IDs is working as well

GTY transceiver tests with fiber loopback on single channel

The aim of this test is to verify the data rate that can be achieved with the FPGA GTY transceivers that are connected to the Leap transceiver. An external fiber connection is used to loop back the transmitted data back into the received data of the same channel. A simplified gateware is used for the FPGA that contains the IBERT for UltraScale/UltraScale+ GTY Transceivers and interacts with the Vivado Serial I/O Analyzer. A similar setup was used during the testing of the SparkPix-IO ASIC. The 156.25 MHz clock connected to MGTREFCLK1 on bank 133 is used as the reference clock for the GTs and the IBERT.

Using digital board 38-C00-01. Only channel 0 (GT_RX_0 / GT_TX_0) is tested for now because of lack of fiber loopbacks. The loopback is done after the 24-MTP to 12-LC-duplex using a LC-simplex fiber. The temperature is the Internal FPGA temperature read from the system monitor. A 31-bit PRBS is used and the eye is scanned with the highest possibly resolution after the temperature has stabilized inside the FPGA. A full test should have all 12 GTY channels running over a long time period.

• Vivado project with pre-built bitfiles: gt-readout-gty-test.zip

Note: The IBERT module only works up until 28.2 Gbit/s.

GTY transceiver tests with fiber loopback on all channels

A MTP-24 loopback cable is attached directly on the Leap fiber pigtail.

Remote control of Rohde & Schwarz HMP4030 power supply

• See Github repository: https://github.com/slaclab/hmp4030

Connection to chiller

A ThermoTek T257P chiller is connected up to the cooling block. The temperature is set to 21.5°C and the chiller was topped up with distilled water. After turning on the chiller the temperature inside the FPGA dropped and is kept at a stable 23°C with the epix-uhr-100kHz-dev firmware running on it. Current consumption for P24V0D is at 0.578 A (= 13.87 W).

Testing with 25 Gbit/s fiber loopback

Using the 25gbps firmware from the earlier test above (see GTY transceiver tests with fiber loopback on all channels) to push more heat out of the FPGA and see how the cooling behaves. Previous test showed a stable temperature around 49°C with a 21.6 W power draw from P24V0D. Temperature now with the chiller is stable at around 29°C, a significant drop!