← ePixUHR35kHz - Megapixel Cameras

The readout electronics for the 3x2 sensor module is split into two parts; the ASIC carrier and the readout board. This page describe these parts in more detail.


Useful resources

Table of contents


Components

ASICs

ePixUHR Throughput Calculations

From ASIC to FPGA:

  • 168*192*12(bit)*14/12(encoding) = 451584 bits/frame
  • @35 kHz frame rate: 451584 bits/frame * 35 kHz = 15.8 Gbit/s per ASIC
    • 15.8/8 = 1.975 Gbit/s per data output
  • @59 kHz frame rate: 451584 bits/frame * 59 kHz = 26.6 Gbit/s per ASIC
    • 26.6/8 = 3.33 Gbit/s per data output
  • @100 kHz frame rate: 451584 bits/frame * 100 kHz = 45.2 Gbit/s per ASIC
    • 45.2/8 = 5.65 Gbit/s per data output

FPGA

The FPGA that will be used is XCKU15P-2FFVA1760E from AMD/Xilinx. Same as used in the GT Readout Platform.

  • Package mechanical drawing
  • FPGA measured 3D model
  • Difference between the drawing and 3D model used:
    • Total height (A = A1 + A2):
      • 3D model is 3.86 mm
      • Drawing is 3.71 mm (max)
    • Solder ball height (A1):
      • 3D model is 0.6 mm
      • Drawing is 0.6 mm (max)
    • Package height (A2):
      • 3D model is 3.26 mm
      • Drawing is 3.21 mm (max)
    • → It seems that the 3D model of the package we have is not 100% accurate, but the difference (3.86-3.71 = 0.15 mm) is negligible
    • Any thermal interface between the package and the cooling block should be able to "absorb" this difference

Optical transceiver

Page about the LEAP transceiver from other project: https://confluence.slac.stanford.edu/display/ppareg/GS-12-1396

Clocks

There are four main clock sources in the system:

  • 371.428571 MHz oscillator
  • 156.25 MHz oscillator
  • LMK61E2 (default 156.25 MHz output) programmable clock
  • 48 MHz crystal
  • Recovered clock from optical GT links

These are routed to various peripherals in the FPGA as shown in the diagram below. The Si5345B is a 10-channel clock multiplier with jitter cleaning features that can have different frequencies on each output. The input to this device can be selected from the 48 MHz crystal, the LMK61E2, a local feedback or from the FPGA and feeds the internal DSPLL. The main limitation to consider is that only one input can be selected to drive the DSPLL. The output of the DSPLL is branched out five MultiSynth blocks that can divide and multiply the clock. These blocks go through a crosspoint switch which allows any of the 10 output channels to be driven from any of the five MultiSynth blocks.

clock-diagram

SI5345B crystal layout and routing

The layout and routing of the 48 MHz crystal feeding the SI5345B is critical to maximize the jitter performance of the devices. It is documented in the reference manual [local pdf] (section 12, page 62). Also see comments in the following Jira issue.

Firmware clock tree

firmware-clock-tree


Board-to-board connector

SAMTEC SEAF8/SEAM8 series connector will be used with 10x40=400 pins in one connector.


Readout board connectorCarrier board connector
3D model

 

Photo of sample
(50 column version)

Part numberSEAF8-40-1-S-10-2-RASEAM8-40-S02.0-S-10-3
Product pagehttps://www.samtec.com/products/seaf8-40-1-s-10-2-rahttps://www.samtec.com/products/seam8-40-s02.0-s-10-3
Catalog[online version] - [local pdf][online version] - [local pdf]
Drawing[online version] - [local pdf][online version] - [local pdf]
Footprint[online version] - [local pdf][online version] - [local pdf]
STEP 3D modelSEAF8-40-1-S-10-2-RA.stpSEAM8-40-S02.0-S-10-3.stp

50 column


Readout board connectorCarrier board connector
3D model

Photo of sample

Part numberSEAF8-50-1-S-10-2-RASEAM8-50-S02.0-S-10-3
Product pagehttps://www.samtec.com/products/seaf8-50-1-s-10-2-rahttps://www.samtec.com/products/seam8-50-s02.0-s-10-3
Catalog[online version] - [local pdf][online version] - [local pdf]
Drawing[online version] - [local pdf][online version] - [local pdf]
Footprint[online version] - [local pdf][online version] - [local pdf]
STEP 3D modelSEAF8-50-1-S-10-2-RA.stpSEAM8-50-S02.0-S-10-3.stp

50 column with guide posts


Readout board connectorCarrier board connector
3D model

Part numberSEAF8-50-1-S-10-2-RA-GPSEAM8-50-S02.0-S-10-3-GP
Product pagehttps://www.samtec.com/products/seaf8-50-1-s-10-2-ra-gphttps://www.samtec.com/products/seam8-50-s02.0-s-10-3-gp
Catalog[online version] - [local pdf][online version] - [local pdf]
Drawing[online version] - [local pdf][online version] - [local pdf]
Footprint[online version] - [local pdf][online version] - [local pdf]
STEP 3D modelSEAF8-50-1-S-10-2-RA-GP.stpSEAM8-50-S02.0-S-10-3-GP.stp

Propagation delay

Note: Due to the use of a right-angle connector there will be different path lengths for signals in different rows. See High Speed Characterization Report from Samtec.

Table 16 on page 38 shows the propagation delay of the first row A (~100 ps) to the last row K (~180 ps) for different signal configurations. These propagation delay values have been assigned to the right-angle connector footprint pads for each row and will therefore be included in the propagation delay calculation in Altium when a trace is routed.

The P and N signal of differential pairs should be placed in the same row to avoid skew between them. Timing critical signals should take into account the different propagation delays for the rows.

Propagation delays, cells with yellow color have been interpolated from the data in the report.

Row

Single-ended: 1:1 S/GSingle-ended: 2:1 S/GDifferential: Optimal HorizontalAssigned in Altium
A96 ps103 ps94 ps100 ps
B106 ps111 ps103 ps109 ps
C115 ps118 ps112 ps118 ps
D125 ps128 ps120 ps127 ps
E135 ps137 ps129 ps136 ps
F144 ps147 ps137 ps144 ps
G153 ps156 ps146 ps153 ps
H162 ps166 ps154 ps162 ps
J172 ps176 ps165 ps171 ps
K182 ps186 ps174 ps180 ps
Average difference:9.6 ps9.2 ps8.9 ps8.9 ps

Timing

External TTL timing in/out

ADC/DAC

Only slow-speed ADC.

Temperature

NHQM103B375T10 10k NTC type thermistors are used at:

  • One on carrier board going to ADC
  • One on readout board going to the ADC
  • One on carrier board going to external connector

Humidity

One HIH-5031-001 sensor mounted close to the carrier connector.

LDO monitoring


ASIC monitoring


Various peripherals

NameDescription
IDA DS2411R+T&R chip is located on the readout board to provide a unique ID that can be read out by the FPGA. Another ID chip is also place on the carrier board.
FlashUsed to store the FPGA configuration.
EEPROMCan be used to store operational settings and parameters.
JTAGA 6-pin Tag-Connect for Xilinx footprint is located on the bottom of the board to not interfere with the cold plate on the top.

Block diagrams

3x2-readout-board-overview

3x2-carrier-board-overview


Power

A GUI from Linear Technologies (now Analog Devices) called LTPowerPlanner has been used to calculate all the required currents and voltages in the system. It also calculates the estimated losses and efficiency of the system.

Power block diagramLTPowerPlannerDatasheets

Board power maps

The figures below give an approximation of where the power is consumed on the readout and ASIC carrier boards. The numbers are from the LTPowerPlanner block diagram above.

Readout board power mapCarrier board power map

DrawIO source file: power-maps.drawio

FPGA

SummaryGraphs

LEAP transceivers

Running at 3.3V, see datasheet on restricted OneDrive for all values. Max power condition of 7.9 W (2.4 A @ 3.3 V) is used in the calculations below.

ASICs


Analog

Digital and I/O

Analog test system

Net

G_AS

G_DS

G_AS_2V5

Voltage

1.3 V

1.3 V

2.5 V

Required current per ASIC

1.85 A ≈ 1.9 A

0.468 A ≈ 0.5 A

0.01 A

System requirement with 6 ASICs

(+30% current for PVT variation)

6*1.9*1.3 = 14.82 A

+1.3 V @ 15 A

1.3*15 = 19.5 W

6*0.5*1.3 = 3.9 A

+1.3 V @ 4.0 A

1.3*4 = 5.2 W

6*0.01*1.3 = 0.078 A

+2.5 V @ 0.5 A

2.5*0.5 = 1.25 W

Power rail current draw

Using worst-case values below for conservative estimate of the currents needed for the different supplies

Power supply railPartQuantityMax current
P0V85DFPGA VCCINT/VCCINT_IO17 A
Total:7 A
P0V8D_ASICASIC GT bias6*2< 0.1 A
Total:< 0.1 A
P0V9DFPGA MGTAVCC44 A
Total:4 A
P1V2DFPGA MGTAVTT48 A
Total:8 A
P1V3DSN74AXC4T774RSVR44*26 uA = 104 uA
Total:< 0.1 A
P1V3D_ASIC[1..6]ASIC digital and IO supply10.5*1.3 = 0.65 A
Total:< 0.7 A
P1V8DSN74AXC4T774RSVR44*26 uA = 104 uA
SI5345A-D-GM1260 mA
FPGA VCCAUX1~500 mA
FPGA VCCAUX_IO1~100 mA
FPGA MGTVCCAUX1~300 mA
FPGA VCCO 01< 100 mA
FPGA XADC_VCC1~10 mA
FPGA VCCO 64/65/66/70/71/911< 100 mA
MT25QU01GBBB8E12-0SIT155 mA
AD5541ABRMZ2< 1 mA
Total:< 1.5 A
P2V5DXLL726371.428571I144 mA
536FB156M250DG198 mA
AT24C64D-MAHM-T13 mA
FPGA VCCO 901< 100 mA
DS2411R2< 10 uA
Total:< 0.3 A
P3V3DSN74AXC4T774RSVR44*26 uA = 104 uA
SI5345A-D-GM1130 mA
LMK61E2BAA-SIAT1196 mA
FPGA VCCO 93/941< 50 mA
SN74AHC1G04DBVR33*4 = 12 mA
SN74LVC3G34DCUR3~3*1 = 3 mA
LEAP transceiver12.4 A
Total:< 3 A
P1V3A_ASIC[1..6]ASIC analog supply11.9*1.3=2.47 A
Total:< 2.5 A
P2V5A_ASICASIC analog test system66*10*1.3 = 78 mA
Total:< 0.1 A
P3V3AADS1217IPFBT11.325 mA
HIH-5031-00110.5 mA
ADR360BUJZ1< 1 mA
ADR361BUJZ1< 1 mA
OPA2626IDGKR22*2*3.1 mA = 12.4 mA
AD5541ABRMZ2< 1 mA
MAX4781ETE???< 1 mA
Total:< 0.1 A

Different options for power components

See Power supply modules and regulators page for details on interesting power supplies and regulators.

power-components-info


Option A
Same components as used in the GT readout platform
Common ASIC supplies

Option B
New components
Separate ASIC supplies

Option C
LT3071 instead of LTM4709
Common ASIC supplies

Diagram

power-supply-options

power-supply-option-b

power-supply-option-c

Area

  • 5x LT8638S: 5*69 = 345 mm2
  • 1x LMZ31520: 246 mm2
  • 2x TPSM5D1806: 2*44 = 88 mm2
  • 19x LT3086EFE: 19*35 = 665 mm2
  • Total: 1344 mm2
  • 27 components
  • 3x LTM4676A: 3*256 = 768 mm2
  • 1x LTM8060: 190 mm2
  • 2x LT3045EDD: 2*9 = 18 mm2
  • 4x LTM4709: 4*72 = 288 mm2
  • 1x LTM8080: 56 mm2
  • Total: 1320 mm2
  • 11 components
  • 3x LTM4676A: 3*256 = 768 mm2
  • 1x LTM8060: 190 mm2
  • 6x LT3071: 6*20 = 120 mm2
  • 1x LT3045EDD: 9 mm2
  • 1x LTM4709: 72 mm2
  • Total: 1159 mm2
  • 12 components

Cost

January 2024

  • 5x LT8638S: 5*13= $65
  • 1x LMZ31520: $20
  • 2x TPSM5D1806: 2*24=$48
  • 19x LT3086EFE: 19*9=$171
  • Total: $304
  • 3x LTM4676A: 3*65=$195
  • 1x LTM8060: $34
  • 2x LT3045EDD: 2*8=$16
  • 4x LTM4709: 4*35=$140
  • 1x LTM8080: $25
  • Total: $410
  • 3x LTM4676A: 3*65=$195
  • 1x LTM8060: $34
  • 6x LT3071: 6*10=$60
  • 1x LT3045EDD: $8
  • 1x LTM4709: $35
  • Total: $332

Noise performance

  • LT8638S:
    • <10 mV pk-pk output ripple
  • LMZ31520:
    • ±1.8% total output voltage variation
    • 1% output voltage ripple @ 20 MHz bandwidth
  • TPSM5D1806:
    • ±1% voltage reference accuracy
  • LT3086EFE:
    • ±2% tolerance over line, load and temperature
    • 40 μVRMS output noise (10 Hz to 100 kHz)
  • LTM4676A:
    • ±0.5% DC output error over temperature
    • 10 mV pk-pk output voltage ripple
  • LTM8060:
    • 0.05% line regulation
    • 0.1% load regulation
    • 10 mV output RMS ripple
  • LT3045EDD:
    • 0.8 µVRMS output noise (10 Hz to 100 kHz)
  • LTM4709:
    • ±1.5% output voltage regulation over line, load,
      and temperature
    • 1.3 μVRMS output noise (10 Hz to 100 kHz)
  • LTM8080:
    • <1 μVRMS (10 Hz to 100 kHz)
  • LT3071:
    • ±1% accuracy over line, load and temperature,

    • 25 μVRMS output noise (10 Hz to 100 kHz)


I/O needs

The sections below indicate how many inputs and outputs (I/Os) are needed for the main blocks of the system.

GT transceiver signals

The FPGA has two types of GT quads:

  • GTY capable at bitrates from 0.5 Gbit/s to 32.75 Gbit/s
    • 8 quads with 4 TX/RX in each for a total of 32 transceivers
  • GTH capable at bitrates from 0.5 Gbit/s to 16.3 Gbit/s
    • 11 quads with 4 TX/RX in each for a total of 44 transceivers

LEAP transceivers

The LEAP transceiver can operate up to 25 Gbit/s and therefore require GTY transceivers. There are 12 TX and 12 RX channels in total.

ASICs

There are 8 high-speed data outputs from each ASIC that can operate up to ~6 Gbit/s. For each ASIC there is also one GT clock input which the FPGA has to provide a clock to (6 Gbit/s = 3 GHz clock). In total each ASIC needs 8 RX and 1 TX channel of the FPGA. There are 6 ASICs in total.

  • 6*8 = 48 RX channels
  • 6*1 = 6 TX channels

This is more than what is available in GTH or GTY separately, which means that some ASIC GT channels will be in GTH and some in GTY. It would also be beneficial to have the TX and RX channels separated to avoid constraining the resources and there are enough FPGA GTs available for this. As shown in the table below, there are 8x ASIC RX channels that are placed in two GTY quads.

Total data rates

  • 6x ASICs, 8x data outputs per ASIC, up to 6 Gbit/s per data output: 6*8*6=288 Gbit/s
  • 12x LEAP transceiver channels up to 25 Gbit/s:
    • 1 channel will be used for timing
    • Data rate left: 11*25=275 Gbit/s
      • 275/(6*8) ≈ 5.7 Gbit/s per ASIC in this case (+encoding)
    • Running PGPv4 at 25 Gbit/s has not been proven yet!
  • From FPGA to PC:

    • 168*192*12(bit)*66/64(PGP encoding) = 399168 bits/frame/ASIC
    • @35 kHz frame rate: 6*399168*35e3 = 83.82528 Gbit/s ≈ 83.8 Gbit/s
      • 83.82528/11 = 7.62048 Gbit/s ≈ 7.6 Gbit/s per LEAP channel (11 channels in total)
    • @100 kHz frame rate: 6*399168*100e3 = 239.5008 Gbit/s ≈ 239.5 Gbit/s
      • 239.5008/11 = 21.7728 Gbit/s ≈ 21.8 Gbit/s per LEAP channel (11 channels in total)

Reverse calculation assuming the current PGPv4 running at 15 Gbit/s:

  • 11 channels at 15 Gbit/s: 11*15*64/66(PGP encoding) = 160 Gbit/s
  • 6 ASICs: 160/6*12/14(encoding) = 22.85714286 ≈ 22.9 Gbit/s per ASIC
    • 22.9/8 = 2.8625 Gbit/s per data output
    • 22.85714286e9/(168*192*12) = 59051.39834449 ≈ 59 kHz frame rate

Reverse calculation assuming the current PGPv4 running at 10 Gbit/s:

  • 11 channels at 10 Gbit/s: 11*10*64/66(PGP encoding) ≈ 106.67 Gbit/s
  • 6 ASICs: 106.67/6*12/14(encoding) = 15.23857143 ≈ 15.24 Gbit/s per ASIC
    • 15.24/8 = 1.905 Gbit/s per data output
    • 15.23857143e9/(168*192*12) = 39368.82913256 ≈ 39 kHz frame rate

GT reference clocks

Quads can source their reference clock from a quad that is at most two quads away. This is to ensure the best jitter performance.

In the table below there are the following reference clock inputs (marked with X and purple/blue):

  • One for the LEAP channels for the GTY quads 0 to 2
  • One for the ASIC clock channels for the GTY quads 4 and 5
  • One for the ASIC data channels for the GTY quads 6 and 7
  • One for the ASIC data channels for the GTH quads 0 to 4
  • One for the ASIC data channels for the GTH quads 5 to 9

This means at least one reference clock is needed for the LEAP channels and four reference clocks are needed for the ASIC channels. See Clocks above for more details on the system clock structure.

Important note on clocking of GTY transceivers from UG578 page 48: "QPLL0 must use GTREFCLK0 and QPLL1 must use GTREFCLK1 when the channel is operating above 16.375 Gb/s"

Summary


Quad 0Quad 1Quad 2Quad 3Quad 4Quad 5Quad 6Quad 7Quad 8Quad 9Quad 10

TXRXTXRXTXRXTXRXTXRXTXRXTXRXTXRXTXRXTXRXTXRX
GTY4x LEAP4x LEAP4x LEAP4x LEAP4x LEAP4x LEAP

N/C4x ASICN/C4x ASIC3x ASICN/C3x ASICN/C





Clock|←← X →→|
| X →→|| X →→|


GTHN/C4x ASICN/C4x ASICN/C4x ASICN/C4x ASICN/C4x ASICN/C4x ASICN/C4x ASICN/C4x ASICN/C4x ASICN/C4x ASIC

Clock|←-← X →-→||←-← X →-→|

There is one GTY quad and one GTH quad left over.

  • No labels