Page History

Introduction

Modern High Energy Physics and Photon Science experiments now require high-speed serial data links of 10 Gbps and higher for their data acquisition systems. These links must be lightweight, with low protocol overhead and small FPGA resource utilization. Often these links will multiplex data from several sources, so the link protocol should also support the idea of multiple “Virtual Channels” per physical link. Based on our experience developing the PGP2 protocol, supported link rates of up to 6 Gbps using 8b/10b encoding, we have now developed the PGP3 protocol to support link rates in excess of 10 Gbps using 64b/66b encoding.

Link Layer

PGP3 uses 64b/66b encoding to achieve DC balance of the serial data stream. Each 64-bit word is scrambled with a source synchronous scrambler with polynomial G(x)=x⁵⁸+x²⁹+1. Two bits are then appended to each word, 0b01 to mark regular data, and 0b10 to mark control characters (K-Codes). This ensures that a transition between 0 and 1 at least once every 66 bits. It is also used for word alignment. These 66-bit words can then be serialized and deserialized using the high speed transceivers found in modern FPGAs. The protocol does not specify any link rates, and any link speed may be targeted provided that the link medium and FPGA on each side can support it.

K-Codes

The protocol defines several K-Codes to indicate data framing, flow-control, opcodes, and other metadata. For all K-Codes, the most significant 8 bits of the 64-bit word indicate which code it is. This is known as the Block Type Field (BTF). The lower 56 bits are then specified differently depending on the K-Code.

LINKINFO Structure

Flow control in performed on a per virtual channel basis. Each received Virtual Channel is expected to be separately buffered in external logic, with PAUSE and OVERFLOW signals from the buffer fed back to the PGP3 block. The PAUSE signal indicates that the buffer has less than Cell (128 words) of space remaining. The receive buffer status of all Virtual Channels is grouped into a 40-bit LINKINFO structure, which is included in each IDLE, SOF and SOC code that is transmitted. The maximum cell size of 128 words grantees that any change in buffer fill status will be transmitted back upstream within at most 128 word-clock cycles

K-Code: SKIP

SKIP codes are sent once every 5000 words. They are used to mitigate clock drift between the oscillators on either side of the link. SKIP characters are not written into the elastic buffer in the receive logic, allowing the buffer to avoid overflows when the transit clock on one side of the link is slightly faster than the receive clock on the other.  The lower 56-bit data field is called "RemoteLinkData" and used to publish status data between the two end-point with high latency with no guarantee of transmission.  RemoteLinkData is intended for high level, slow changing status bit communication (Example: Board ID Number).

K-Code: IDLE

IDLE codes are sent when the transmit logic has nothing else to send, or when flow control indicates that the downstream side is unable to receive data.

K-Code: SOF/SOC

SOF (Start of Frame) or SOC (Start of Cell) codes are sent when the start of data payload transmission

K-Code: EOF/EOC

EOF (End of Frame) or EOC (End of Cell) codes are sent when the start of data payload transmission

Data Cells