Tip
The readout electronics for the 3x2 sensor module is split into two parts; the ASIC carrier and the readout board. This page describe these parts in more detail.

Components

ASICs

ePixUHR Throughput Calculations

From ASIC to FPGA:

168*192*12(bit)*14/12(encoding) = 451584 bits/frame
@35 kHz frame rate: 451584 bits/frame * 35 kHz = 15.8 Gbit/s per ASIC
- 15.8/8 = 1.975 Gbit/s per data output
@59 kHz frame rate: 451584 bits/frame * 59 kHz = 26.6 Gbit/s per ASIC
- 26.6/8 = 3.33 Gbit/s per data output
@100 kHz frame rate: 451584 bits/frame * 100 kHz = 45.2 Gbit/s per ASIC
- 45.2/8 = 5.65 Gbit/s per data output

FPGA

The FPGA that will be used is XCKU15P-2FFVA1760E from AMD/Xilinx. Same as used in the GT Readout Platform.

Expand

Package mechanical drawing
FPGA measured 3D model
Difference between the drawing and 3D model used:
- Total height (A = A1 + A2):
  - 3D model is 3.86 mm
  - Drawing is 3.71 mm (max)
- Solder ball height (A1):
  - 3D model is 0.6 mm
  - Drawing is 0.6 mm (max)
- Package height (A2):
  - 3D model is 3.26 mm
  - Drawing is 3.21 mm (max)
- → It seems that the 3D model of the package we have is not 100% accurate, but the difference (3.86-3.71 = 0.15 mm) is negligible
- Any thermal interface between the package and the cooling block should be able to "absorb" this difference

Optical transceiver

Page about the LEAP transceiver from other project: https://confluence.slac.stanford.edu/display/ppareg/GS-12-1396

Clocks

There are four main clock sources in the system:

371.428571 MHz oscillator
LMK61E2 (default 156.25 MHz) programmable clock
48 MHz crystal
Recovered clock from optical GT links

These are routed to various peripherals in the FPGA as shown in the diagram below. The Si5345B is a 10-channel clock multiplier with jitter cleaning features that can have different frequencies on each output. The input to this device can be selected from the 48 MHz crystal, the LMK61E2, a local feedback or from the FPGA and feeds the internal DSPLL. The main limitation to consider is that only one input can be selected to drive the DSPLL. The output of the DSPLL is branched out five MultiSynth blocks that can divide and multiply the clock. These blocks go through a crosspoint switch which allows any of the 10 output channels to be driven from any of the five MultiSynth blocks.

TODO: Do we need a 156.25 MHz clock directly to the FPGA?

Gliffy Diagram

size	600
displayName	clock-diagram
name	clock-diagram
pagePin	14

Firmware clock tree

Gliffy Diagram

size	300
displayName	firmware-clock-tree
name	firmware-clock-tree
pagePin	23

Timing

External TTL timing in/out

ADC/DAC

Only slow-speed ADC.

Temperature

NHQM103B375T10 10k NTC type thermistors are used at:

One on carrier board going to ADC
One on readout board going to the ADC
One on carrier board going to external connector

Humidity

One HIH-5031-001 sensor mounted close to the carrier connector.

LDO monitoring

ASIC monitoring

Various peripherals

Name	Description
ID	A DS2411R+T&R chip is located on the readout board to provide a unique ID that can be read out by the FPGA. Another ID chip is also place on the carrier board.
Flash	Used to store the FPGA configuration.
EEPROM	Can be used to store operational settings and parameters.
JTAG	A 6-pin Tag-Connect for Xilinx footprint is located on the bottom of the board to not interfere with the cold plate on the top.

Block diagrams

Gliffy Diagram

displayName	3x2-readout-board-overview
name	3x2-readout-board-overview
pagePin	17

Gliffy Diagram

macroId	e1bf8b77-0d8e-42d9-8a9d-a8334dc511dd
displayName	3x2-carrier-board-overview
name	3x2-carrier-board-overview
pagePin	6

Power

A GUI from Linear Technologies (now Analog Devices) called LTPowerPlanner has been used to calculate all the required currents and voltages in the system. It also calculates the estimated losses and efficiency of the system.

Power block diagram	LTPowerPlanner	Datasheets
	Total input power: 73 W Total output power: 58 W Total power loss: 15 W Total efficiency: 79% Source file: ePixUHR35kfps-3x2-readout-board-lt-power-planner.ltp3 NOTE: This includes the power that is consumed by the ASICs also	LTM4676A LTM8060 LTM8080 LTM4709 LT3045

Board power maps

The figures below give an approximation of where the power is consumed on the readout and ASIC carrier boards. The numbers are from the LTPowerPlanner block diagram above.

Readout board power map	Carrier board power map

DrawIO source file: power-maps.drawio

FPGA

https://www.xilinx.com/products/technology/power/xpe.html
UltraScalePlus_XPE_2023_1_2.xlsm
- The numbers in this spreadsheet are estimates based on the epix-uhr-dev and epix-hr-m-320k firmwares
- Note that the majority of the power is in the GTs, which means that the logic and other components can be roughly estimated

Summary	Graphs

LEAP transceivers

Running at 3.3V, see datasheet on restricted OneDrive for all values. Max power condition of 7.9 W (2.4 A @ 3.3 V) is used in the calculations below.

ASICs

ASIC Version 2 - 2023: ePixUHR-100kHz: power consumption ← Using these numbers in the table below
ASIC Version 1 - 2022: ePixUHR: power consumption

	Analog	Digital and I/O	Analog test system
Net	G_AS	G_DS	G_AS_2V5
Voltage	1.3 V	1.3 V	2.5 V
Required current per ASIC	1.85 A ≈ 1.9 A	0.468 A ≈ 0.5 A	0.01 A
System requirement with 6 ASICs (+30% current for PVT variation)	61.91.3 = 14.82 A +1.3 V @ 15 A 1.315 = 19.5 W*	60.51.3 = 3.9 A +1.3 V @ 4.0 A 1.34 = 5.2 W*	60.011.3 = 0.078 A +2.5 V @ 0.5 A 2.50.5 = 1.25 W*

Power rail current draw

Using worst-case values below for conservative estimate of the currents needed for the different supplies

Power supply rail	Part	Quantity	Max current
P0V85D	FPGA VCCINT/VCCINT_IO	1	7 A
P0V85D	Total:		7 A
P0V8D_ASIC	ASIC GT bias	6*2	< 0.1 A
P0V8D_ASIC	Total:		< 0.1 A
P0V9D	FPGA MGTAVCC	4	4 A
P0V9D	Total:		4 A
P1V2D	FPGA MGTAVTT	4	8 A
P1V2D	Total:		8 A
P1V3D	SN74AXC4T774RSVR	4	4*26 uA = 104 uA
P1V3D	Total:		< 0.1 A
P1V3D_ASIC[1..6]	ASIC digital and IO supply	1	0.5*1.3 = 0.65 A
P1V3D_ASIC[1..6]	Total:		< 0.7 A
P1V8D	SN74AXC4T774RSVR	4	4*26 uA = 104 uA
	SI5345A-D-GM	1	260 mA
	FPGA VCCAUX	1	~500 mA
	FPGA VCCAUX_IO	1	~100 mA
	FPGA MGTVCCAUX	1	~300 mA
	FPGA VCCO 0	1	< 100 mA
	FPGA XADC_VCC	1	~10 mA
	FPGA VCCO 64/65/66/70/71/91	1	< 100 mA
	MT25QU01GBBB8E12-0SIT	1	55 mA
	AD5541ABRMZ	2	< 1 mA
	Total:		< 1.5 A
P2V5D	XLL726371.428571I	1	44 mA
	AT24C64D-MAHM-T	1	3 mA
	FPGA VCCO 90	1	< 100 mA
	DS2411R	2	< 10 uA
	Total:		< 0.2 A
P3V3D	SN74AXC4T774RSVR	4	4*26 uA = 104 uA
	SI5345A-D-GM	1	130 mA
	LMK61E2BAA-SIAT	1	196 mA
	FPGA VCCO 93/94	1	< 50 mA
	SN74AHC1G04DBVR	3	3*4 = 12 mA
	SN74LVC3G34DCUR	3	~3*1 = 3 mA
	LEAP transceiver	1	2.4 A
	Total:		< 3 A
P1V3A_ASIC[1..6]	ASIC analog supply	1	1.9*1.3=2.47 A
P1V3A_ASIC[1..6]	Total:		< 2.5 A
P2V5A_ASIC	ASIC analog test system	6	6101.3 = 78 mA
P2V5A_ASIC	Total:		< 0.1 A
P3V3A	ADS1217IPFBT	1	1.325 mA
	HIH-5031-001	1	0.5 mA
	ADR360BUJZ	1	< 1 mA
	ADR361BUJZ	1	< 1 mA
	OPA2626IDGKR	2	223.1 mA = 12.4 mA
	AD5541ABRMZ	2	< 1 mA
	MAX4781ETE	???	< 1 mA
	Total:		< 0.1 A

Different options for power components

See Power supply modules and regulators page for details on interesting power supplies and regulators.

Gliffy Diagram

displayName	power-components-info
name	power-components-info
pagePin	4

Option A
Same components as used in the GT readout platform
Common ASIC supplies

Option B
New components
Separate ASIC supplies

Option C
LT3071 instead of LTM4709
Common ASIC supplies

Diagram

Gliffy Diagram

size	200
displayName	power-supply-option-a
name	power-supply-options
pagePin	59

Gliffy Diagram

size	200
displayName	power-supply-option-b
name	power-supply-option-b
pagePin	75

Gliffy Diagram

size	200
displayName	power-supply-option-c
name	power-supply-option-c
pagePin	3

Area

5x LT8638S: 5*69 = 345 mm²
1x LMZ31520: 246 mm²
2x TPSM5D1806: 2*44 = 88 mm²
19x LT3086EFE: 19*35 = 665 mm²
Total: 1344 mm²
27 components

3x LTM4676A: 3*256 = 768 mm²
1x LTM8060: 190 mm²
2x LT3045EDD: 2*9 = 18 mm²
4x LTM4709: 4*72 = 288 mm²
1x LTM8080: 56 mm²
Total: 1320 mm²
11 components

3x LTM4676A: 3*256 = 768 mm²
1x LTM8060: 190 mm²
6x LT3071: 6*20 = 120 mm²
1x LT3045EDD: 9 mm²
1x LTM4709: 72 mm²
Total: 1159 mm²
12 components

Cost

January 2024

5x LT8638S: 5*13= $65
1x LMZ31520: $20
2x TPSM5D1806: 2*24=$48
19x LT3086EFE: 19*9=$171
Total: $304

3x LTM4676A: 3*65=$195
1x LTM8060: $34
2x LT3045EDD: 2*8=$16
4x LTM4709: 4*35=$140
1x LTM8080: $25
Total: $410

3x LTM4676A: 3*65=$195
1x LTM8060: $34
6x LT3071: 6*10=$60
1x LT3045EDD: $8
1x LTM4709: $35
Total: $332

Noise performance

LT8638S:
- <10 mV pk-pk output ripple
LMZ31520:
- ±1.8% total output voltage variation
- 1% output voltage ripple @ 20 MHz bandwidth
TPSM5D1806:
- ±1% voltage reference accuracy
LT3086EFE:
- ±2% tolerance over line, load and temperature
- 40 μV_RMS output noise (10 Hz to 100 kHz)

LTM4676A:
- ±0.5% DC output error over temperature
- 10 mV pk-pk output voltage ripple
LTM8060:
- 0.05% line regulation
- 0.1% load regulation
- 10 mV output RMS ripple
LT3045EDD:
- 0.8 µV_RMS output noise (10 Hz to 100 kHz)
LTM4709:
- ±1.5% output voltage regulation over line, load,
  and temperature
- 1.3 μV_RMS output noise (10 Hz to 100 kHz)
LTM8080:
- <1 μVRMS (10 Hz to 100 kHz)

LT3071:
- ±1% accuracy over line, load and temperature,
- 25 μV_RMS output noise (10 Hz to 100 kHz)

I/O needs

The sections below indicate how many inputs and outputs (I/Os) are needed for the main blocks of the system.

GT transceiver signals

The FPGA has two types of GT quads:

GTY capable at bitrates from 0.5 Gbit/s to 32.75 Gbit/s
- 8 quads with 4 TX/RX in each for a total of 32 transceivers
GTH capable at bitrates from 0.5 Gbit/s to 16.3 Gbit/s
- 11 quads with 4 TX/RX in each for a total of 44 transceivers

LEAP transceivers

The LEAP transceiver can operate up to 25 Gbit/s and therefore require GTY transceivers. There are 12 TX and 12 RX channels in total.

ASICs

There are 8 high-speed data outputs from each ASIC that can operate up to ~6 Gbit/s. For each ASIC there is also one GT clock input which the FPGA has to provide a clock to (6 Gbit/s = 3 GHz clock). In total each ASIC needs 8 RX and 1 TX channel of the FPGA. There are 6 ASICs in total.

6*8 = 48 RX channels
6*1 = 6 TX channels

This is more than what is available in GTH or GTY separately, which means that some ASIC GT channels will be in GTH and some in GTY. It would also be beneficial to have the TX and RX channels separated to avoid constraining the resources and there are enough FPGA GTs available for this. As shown in the table below, there are 8x ASIC RX channels that are placed in two GTY quads.

Total data rates

6x ASICs, 8x data outputs per ASIC, up to 6 Gbit/s per data output: 6*8*6=288 Gbit/s
12x LEAP transceiver channels up to 25 Gbit/s:
- 1 channel will be used for timing
- Data rate left: 11*25=275 Gbit/s
  - 275/(6*8) ≈ 5.7 Gbit/s per ASIC in this case (+encoding)
- Running PGPv4 at 25 Gbit/s has not been proven yet!
  - 15 Gbit/s according to this page https://confluence.slac.stanford.edu/display/ppareg/GS-12-1396
  - Main limitation is the 64-bit data width of PGPv4 which would result in a 25000/64 ≈ 390 MHz fabric clock
  - 11*15=165 Gbit/s
  - 165/(6*8) = 3.4375 Gbit/s per ASIC in this case (+encoding)
From FPGA to PC:
- 168*192*12(bit)*66/64(PGP encoding) = 399168 bits/frame/ASIC
- @35 kHz frame rate: 6*399168*35e3 = 83.82528 Gbit/s ≈ 83.8 Gbit/s
  - 83.82528/11 = 7.62048 Gbit/s ≈ 7.6 Gbit/s per LEAP channel (11 channels in total)
- @100 kHz frame rate: 6*399168*100e3 = 239.5008 Gbit/s ≈ 239.5 Gbit/s
  - 239.5008/11 = 21.7728 Gbit/s ≈ 21.8 Gbit/s per LEAP channel (11 channels in total)

Reverse calculation assuming the current PGPv4 running at 15 Gbit/s:

11 channels at 15 Gbit/s: 11*15*64/66(PGP encoding) = 160 Gbit/s
6 ASICs: 160/6*12/14(encoding) = 22.85714286 ≈ 22.9 Gbit/s per ASIC
- 22.9/8 = 2.8625 Gbit/s per data output
- 22.85714286e9/(168*192*12) = 59051.39834449 ≈ 59 kHz frame rate

GT reference clocks

Quads can source their reference clock from a quad that is at most two quads away. This is to ensure the best jitter performance.

In the table below there are the following reference clock inputs (marked with X and purple/blue):

One for the LEAP channels for the GTY quads 0 to 2
One for the ASIC clock channels for the GTY quads 4 and 5
One for the ASIC data channels for the GTY quads 6 and 7
One for the ASIC data channels for the GTH quads 0 to 4
One for the ASIC data channels for the GTH quads 5 to 9

This means at least one reference clock is needed for the LEAP channels and four reference clocks are needed for the ASIC channels. See Clocks above for more details on the system clock structure.

Note
Important note on clocking of GTY transceivers from UG578 page 48: "QPLL0 must use GTREFCLK0 and QPLL1 must use GTREFCLK1 when the channel is operating above 16.375 Gb/s"

Summary

	Quad 0		Quad 1		Quad 2		Quad 3		Quad 4		Quad 5		Quad 6		Quad 7		Quad 8		Quad 9		Quad 10
	TX	RX	TX	RX	TX	RX	TX	RX	TX	RX	TX	RX	TX	RX	TX	RX	TX	RX	TX	RX	TX	RX
GTY	4x LEAP	4x LEAP	4x LEAP	4x LEAP	4x LEAP	4x LEAP			N/C	4x ASIC	N/C	4x ASIC	3x ASIC	N/C	3x ASIC	N/C
Clock	\|←		← X →		→\|				\| X →		→\|		\| X →		→\|
GTH	N/C	4x ASIC	N/C	4x ASIC	N/C	4x ASIC	N/C	4x ASIC	N/C	4x ASIC	N/C	4x ASIC	N/C	4x ASIC	N/C	4x ASIC	N/C	4x ASIC	N/C	4x ASIC
Clock	\|←		-		← X →		-		→\|		\|←		-		← X →		-		→\|

There is one GTY quad and one GTH quad left over.

Space shortcuts

Confluence Content

Child pages

Versions Compared

Old Version 309

New Version 310

Key

Useful resources

Table of contents

Components

ASICs

ePixUHR Throughput Calculations

FPGA

Optical transceiver

Clocks

Firmware clock tree

Timing

ADC/DAC

Temperature

Humidity

LDO monitoring

ASIC monitoring

Various peripherals

Block diagrams

Power

Board power maps

FPGA

LEAP transceivers

ASICs

Power rail current draw

Different options for power components

I/O needs

GT transceiver signals

LEAP transceivers

ASICs

Total data rates

GT reference clocks

Summary

Space shortcuts

Confluence Content

Child pages

Page History

Versions Compared

Old Version 309

New Version 310

Key

Useful resources

Table of contents

Components

ASICs

ePixUHR Throughput Calculations

FPGA

Optical transceiver

Clocks

Firmware clock tree

Timing

ADC/DAC

Temperature

Humidity

LDO monitoring

ASIC monitoring

Various peripherals

Block diagrams

Power

Board power maps

FPGA

LEAP transceivers

ASICs

Power rail current draw

Different options for power components

I/O needs

GT transceiver signals

LEAP transceivers

ASICs

Total data rates

GT reference clocks

Summary