Jira tasks for the project

• Jira Kanban board for this project

ASIC level requirements summary

Requirement	ePixUHR	SparkPix-S	SparkPix-ED
Frame rate	100kfps	1Mfps	1Mfps
Power supplies	2.5V Analog 1.3V (AS/DS/IO)	2.5V Analog 1.3V (AS/DS/IO) 0.6V (Current sink!)	1.3V (AS/DS/IO)
Power for each supply	ePixUHR - 35 kHz	SparkPix-S: supply/ground and power consumption	t.b.d.
Number of GT IOs per ASIC	8 (outputs) 1 clock in	8 (outputs) 1 clock in	t.b.d (The current agreement is to have 8 outputs)
Expected I/O speed	5.25 Gbit/s	5.25 Gbit/s	10 Gbit/s
Total data bandwidth	42 Gbit/s	42 Gbit/s	80 Gbit/s

Target Cameras

There are three targeted cameras for this project:

• 2x2 ePix/SparkPix
• 1M ePix to later be used as building block for 16M camera
• 2M SparkPix

Block diagrams

2x2 ePix/SparkPix

• ePixHR/UHR 2x2 would be 140k pixels
• SparkPix S would be 500k pixels

1M ePix

• ePixHR/UHR in a 6x6 configuration would be 1.1M pixels
• ePixHR/UHR in a 6x5 configuration would be 1M pixels

2M SparkPix S

• SparkPix S in a 4 x 4 configuration would be 2M pixels

Parameter overview

Anchor
asic-power-requirements asic-power-requirements

ASIC Power Requirements

FPGA Selection

*Done considering 1% Occupancy instead of maxing out the transceivers

Summarizing Table

Expand
title FPGA Size Comparison

FPGA KU15P (-A1760) Kintex U+ voltage rails

From Kintex UltraScale+ FPGAs Data Sheet: DC and AC Switching Characteristics (DS922)

Power

System Power Consumption Breakdown

Anchor
dc-dc-converters dc-dc-converters

DC/DC converters

Texas Instruments WEBENCH

https://webench.ti.com/power-designer/switching-regulator

LMZ31520 - digital board - 5.0V to 0.85V

• Datasheet
• Link to WEBENCH with all parameters set (login required)
• WEBENCH Design Report

TPSM5D1806 - digital board - 5.0V to 1.8V and 0.9V

• Datasheet
• Link to WEBENCH with all parameters set (login required)
• WEBENCH Design Report

TPSM5D1806 - digital board - 5.0V to 1.2V

• Datasheet
• Link to WEBENCH with all parameters set (login required)
• WEBENCH Design Report

Anchor
power-graph power-graph

Power Graph

Optional bypass of LDOs for ASIC

There are optional bypass jumpers on the analog board, as shown in the diagram above, for providing power to the ASIC directly from the LT8638S switching regulators. This will bypass the LDOs and can be used to test the performance of the ASIC when it's powered from a "dirtier" power source.

Paralleling of DC/DC and LDOs from data sheets

System level

General I/O

Other components

Transceiver

• Ideal is to reuse the 300 Gbit/s Leap On transceiver from Amphenol, unless we find a replacement that operates with single mode power supply.
• Amphenol Transceivers Product Presentation
• About the stock (at 12 December 2023):
  • interconnect that is soldered on the board we have in stock :
  • 10140369-201LF (qty.25)
  • 10140369-101LF (qty.4)

  • (I checked the parts and both of them say -101LF on the component. 201 is different only on the numbering on the label. So in total we have 29)

  • Laser transceiver part:
    10124588-410 (no heatsink) (qty.5)
    10124588-411 (low profile heatsink)(qty.15)
    (Online you cannot find description of -410 and -411. I am pretty sure they are equivalent to the -310 and -311) 
  • Optical cable:
    Needs to be custom ordered

Board material 

• Needs to be FR408HR or better

Anchor
connectors connectors

Connectors

Cooling block

Anchor
grounding grounding

Grounding

• Separate ground come into the system through the TFM power connector on the digital board
• The HV supply ground and the Analog supply ground are connected to the same analog ground net
• The digital supply is connected to the digital ground net
• The analog and digital grounds are connected together on the digital board near the TFM power connector

Clocks

There are four main clock sources:

• Oscillator for GTY transceivers for LCLS-II timing
• Oscillator for GTY transceivers for PGP communication
• Jitter attenuator with various input choices for FPGA logic and GTY transceivers
• Jitter attenuator with various input choices for FPGA logic and GTH transceivers

The SI53340 clock buffer only buffers the input clock source into four output clocks with low amount of jitter added. The SI5345B is a jitter attenuator and an "any-frequency" multiplier where one input is selected that is fed to a PLL which then feeds multipliers for the individual outputs. Each output can therefore be programmed to different frequencies, which are synchronous with the selected input clock.

Clock structure diagram

ASIC GT clock generation

The gigabit transceivers (GTs) in the ASIC require a high-frequency clock (2-3 GHz) with low jitter. There is no PLL inside the ASIC so it has to be provided from an external source (see SparkPix-IOs: fast I/Os prototype (~5 Gb/s) on TSMC 130nm). The proposed architecture will use the GTH transceiver outputs of the Kintex UltraScale+ to generate a "clock" from a static "101010..." bit pattern.

ASIC GT AC-coupling

There are high-speed signals in both directions between the ASICs and the FPGA:

• Clock signals from the FPGA GTH TX to the ASIC GT CLK
• Data signals from the ASIC GT TX to the FPGA GTH RX

These could either be DC-coupled or AC-coupled, which means that a series capacitor in the order of ~100nF is placed in series with the signal somewhere along the path. It is also called DC-blocking capacitors as described by Dr. Howard Johnson in an article on his website. The question is, where to place the capacitors? Close to the transmitter, in the middle, close to the receiver or somewhere else? In Johnson's article he argues that the main effect of these capacitors will be an impedance mismatch because the package of the capacitor will be bigger than the trace itself and we end up with an impedance that is less than the nominal impedance Z0 of the transmission line. This will result in negative reflections. He then argues that the effect of such a reflection has to be considered in relationship to the symbol baud interval and if it's much less than 1/2 of the baud interval it will have a minimal effect. We are working with up to 6 Gbit/s which gives a baud interval of 167ps. With a propagation delay of around 150 ps/inch (source) it means that to have minimal impact it has to be much less than 1/4 inch (~6mm), somewhere in the order 1/20 of an inch (~1.2mm) which is not practical on a PCB with IC packages, passives and routing.

For these high-speed cases it is therefore not the distance of the capacitor from the transmitter that matters, but instead the layout and routing of the traces around the capacitor.The goal is instead to minimize the effect of the capacitor on the impedance of the trace. One example Johnson gives is to reduce the parasitic capacitance underneath the body of the capacitor with a keep-out region in the reference plane underneath. A similar concept is shown in a layout design guide (local pdf copy, see page 30) for an Intel Stratix 10 device.

If we look at the LEAP transceiver that we are using in this project we can see that it has the AC-coupling capacitors on both RX and TX inside the package itself. This probably ensures that the effect of the capacitors is reduced as much as possible and they might use very small package sizes.

In our case it's probably a question of physical constraints that define where we can place the capacitors. For all four ASICs we would need 4*8*2+4*2=72 capacitors for data and clocks. Placing these on the carrier board might be tricky. Having them on the analog board would be a tight fit due to all the power supplies it has. That leaves the digital board which should hopefully have enough space to make a "clean" implementation of these capacitors.

ASIC timing and control signals - voltage level translation

The ASIC timing (SRO, R0, ACQ and INJ) and control (GR_N, SACI_SEL[3..0], SACI_CLK, SACI_CMD and SACI_RSP) are 1.3V CMOS signals in the ASIC that has be be interfaced with the FPGA banks. In previous systems (e.g. ePixHR10k 2M digital board (TXI)) these were 2.5V CMOS and were driven directly from HD banks in the Kintex UltraScale+ (except for SACI_SEL and SACI_RSP that were driven via MAX3002 from 1.8V HP banks).

In short, there are two types of voltage level translators: auto-sensing bidirectional ones with transmission gates that will pull up or down when a rising or falling edge are detected; directional ones which has input receivers in one voltage domain and output drivers in the other voltage domain. The auto-sensing ones seem to lack information about the timing characteristics in the datasheet which is probably because it depends on how it is used and the device that is driving the pins of the translator. The directional ones are probably a better fit for this application where we know the direction and want as high drive strength as possible to propagate from the digital board all the way to the ASIC on the carrier board.

Resources

• TI Voltage Translation Buying Guide
• TI Supporting Time and Skew Sensitive Interfaces with TI's TXV Level-Shifter Portfolio

Anchor
board-design board-design

Board design

Altium 365 project folder: https://stanford-linear-accelerator-center.365.altium.com/designs/folder-077F97F3-88BC-43E1-A2A7-66F200D82CFA

ePixUHR100k 2x2 implementation

Previous boards

• Based mechanically on the boards from ePixHR250M_2x2_Camera_Documentation
  • Carrier board (PC_261_101_24_C00)
  • Digital board (PC_261_101_25_C00)
  • Power & communication board (PC_261_101_26_C00)
• Also using some components as used in ePixHR10k Shingled Camera - 2.2MPix - with jaw movement
  • Carrier board (PC_261_101_20_C00)
  • Digital board (PC_261_101_23_C00)
  • Analog board (PC_261_101_22_C00)
• Changes done to the system architecture compared to the boards in the camera above:
  • The digital board with the FPGA will be moved to the bottom of the stack with the optical transceiver
  • The analog board is at the top and interfaces between the carrier and the digital board

Anchor
altium-missing-components altium-missing-components

Altium missing components

The components listed in the expansion box below are currently missing from the SLAC Altium library located on OneDrive (Altium_Yee_lib).

Block diagrams

Digital board

The digital board contains the FPGA, supporting ICs and the Amphenol optical transceiver module. The DC/DC on this board are the ones related to the components located here.

FPGA design resources

• Kintex UltraScale+ Documentation

• UltraScale+ Device Packaging and Pinouts Product Specification User Guide (UG575)

  • Note: The PDF download has better resolution than the web version
  • See page 322 - FFVA1760 (XCKU15P)
• XTP427 - UltraScale+ FPGA and Zynq UltraScale+ MPSoC and RFSoC Schematic Review Checklist (v1.14) (login required)
• UltraScale and UltraScale+ Package Device Pinout Files
  • XCKU15P-FFVA1760: https://www.xilinx.com/content/dam/xilinx/support/packagefiles/usapackages/xcku15pffva1760pkg.txt

Analog board

The analog board contains all the DC/DC converters that are needed to support the different power rails of the ASICs (see above). The data and control signals between the ASICs and the FPGA are routed through this board between the two high-density connectors.

Carrier board

The carrier board contains the specific ASICs and any passive components that are needed.

Simulations

GT to the ASICs

TODO

GT to the transceivers

TODO

Power drops

TODO

Simulation of sinking power regulator at 0.6V output potential

Presentations by Pietro

Simulation by Pietro

• Source files: LT3091 test.zip
• A LT1965 LDO feeds a 2.5V (Vin) voltage to two LT3091 in parallel
• At the outputs of the LT3091 there is a 2.4A current source with a 1.2V voltage source (will be 1.3V in final application) in series that emulate the ASIC front-end
• The SET pins have a 50uA reference current source in LT3091
  • Since there are two in this configuration the total current is 100uA
  • This means that the voltage is V=I*R=100e-6*19e3=1.9V
  • This voltage is relative to the "GND" pins which are at a 2.5V potential
  • The potential at the output is therefore 2.5-1.9=0.6V
  • Closest standard value resistor is 19.1k that results in V=1.91V → Output is at 0.59V

Updated simulation with LT3806 and ballast resistor compensation

• The LT3086 feeds the two LT3091's with 2.5V
• The LT3086 is fed with a 5V supply
• The Imonn signal is tapped off at a voltage divider which compensates for the two 10mR ballast resistors at the outputs of the LT3091's
• Simulation file: LT3091_sinking_with_LT3086_and_compensation.asc