PGP version

Gen 1 hardware implements revision 1 of the PGP protocol. PGP firmware for Petacache RCEs and slices returns decimal 206 as the PGP firmware version.

Number of instances per RCE

Petacache RCEs have four PGP cores named A through D while LCLS RCEs have three named A through C.

Core device registers

Each PGP core has four Device Control Registers, which give information about each PGP link as seen from the "FPGA" (the RCE). The register base addresses are:

PGP core	First DCR addr
A	0x2e0
B	0x2e4
C	0x2e8
D (petacache only)	0x2ec

The individual registers for each core:

Offset	Name	Description
0	Control	Selects and displays the mode of operation.
1	Status	Summarizes the current status of the link.
2	Count0	A set of 4-bit counters.
3	Count1	More 4-bit counters.

In the register descriptions that follow the bit numbering used is the same as in the VHDL file: the most significant bit is numbered 31 and the least significant is numbered zero.

The names of some of the register fields use "pib" to mean the remote PGP or the link to it; perhaps "rmt" would have been a better choice.

Control register

Bit	Name	Purpose
31-20		Ignored on write. Read back as zeros.
19	countReset	As long as this bit is 1 all the counters for this core will be reset to and held at zero. Clear the bit when you want the counters to become active again.
18	mgtLoopback	Set this bit to place the core's Multi-Gigabit Transceiver (MGT) into loopback mode, so that any data sent is received locally instead of by the remote PGP.
17	pibRelink	Set this bit to set the link state to ST_LOCK (see below) and hold it there. Clear the bit again to try to reestablish the link.
16	enableDrop	This bit controls the what the core will do when the PGP transmit buffer is "almost full". When it's set the core will discard any new frame data coming from the assigned PEB. When it's clear the core will cause the PEB to stop sending new data. In either case normal behavior will resume when enough buffer space becomes available.
15-1		Ignored on write. Read back as zeros.
0	pgpSeqError	Ignored on write. When read a value of 1 means that there has been a PGP sequencing error for this core. This occurs when the ACK FIFO becomes full or the frame sequence number FIFO becomes empty. This condition will persist until the link is dropped and reestablished (see pibRelink above).

After a system reset both loopback and drop mode are disabled.

If you change the value of mgtLoopback or enableDrop you must also reset the link using pibRelink in order for the new operating mode to go into effect. For example, in order to set loopback mode for link A you must first write 0x00060000 to DCR 0x2e0 and then write 0x00040000.

Status register

Writing to this register has no effect. It may be read to get a summary of the core's current state.

Bit	Name	Purpose
31-24	vc3RemBuffFull	Each of these single-bit flags shows the status of the remote PGP's receive buffer for a virtual channel.
	vc3RemBuffAFull	Each such buffer may be asserting "full" and/or "almost full".
	vc2RemBuffFull
	vc2RemBuffAFull
	vc1RemBuffFull
	vc1RemBuffAFull
	vc0RemBuffFull
	vc0RemBuffAFull
23-22	pibLock	The state of the MGT's phase-locked loops (PLLs); the high-order bit is for the transmit (Tx) PLL and the other is for the receive (Rx) PLL. A value of 1 means that the PLL is locked.
21	mgtInverted	A value of 1 means that the MGT is inverting the signals it receives and transmits, something that happens automatically if needed to communicate with the remote PGP.
20	pibFail	A value of 1 means that the link state is ST_FAIL (see below).
19	pibLinkReady	A value of 1 means that the link state is ST_EN or ST_SKP (see below).
18-16	pibState	The state of the link to the remote PGP. See the table below.
15-8	remoteVersion	The version number of the remote PGP core. This is reliable only if the link is ready.
7-0	localVersion	The version number of this PGP core.

A finite state machine controls the state of the link to the remote PGP. The possible states are:

pibState value	State name	Meaning
1	ST_LOCK	The initial state. Waiting for the PLL to lock.
2	ST_IDLE	Transmitting and receiving IDLE sequences.
3	ST_TRAIN	Transmitting and receiving training sequences.
4	ST_VER	Transmitting and receiving PGP core version numbers.
5	ST_EN	Ready to transmit (or already transmitting) cell data.
6	ST_SKP	Transmitting the Skip sequence that must follow every cell sent.
7	ST_FAIL	Failed to reach the state ST_EN.

The normal sequence of states is ST_LOCK, ST_IDLE, ST_TRAIN, ST_VER, ST_EN, ST_SKP, ST_EN, ST_SKP, ST_EN, etc. If any of the steps leading to state ST_EN fails then the link enters the state ST_FAIL and remains there until forced back to the ST_LOCK state. ST_LOCK will be entered upon system reset, if the remote PGP sends IDLE sequences or if the pibRelink bit is set in the control register (see above). Generally failure occurs when the expected type of sequences are not seen coming from the remote PGP; the ST_VER step will in addition fail if the remote version isn't the same as the local version.

Immediately after a system reset the link goes into the ST_LOCK state with all the other state information set accordingly, after which the local PGP core tries to bring the link up.

Count0 register

Writing to this register has no effect. Reading it yields a number of 4-bit counters which count up to 15; they don't roll over to zero again once they've reached that maximum.

Bits	Counter name	What it counts
31-28	cntFlowError	The PIB FIFO condition changes from not-full to full.
27-24	cntStatError	The PEB's internal 16-element completion-status FIFO couldn't accept a new value.
23-20	cntShiftError	A firmware error where data is presented to the receiver when it isn't ready for it.
19-16	cntNack	The link was up and we received the first of a run of NACKs.
15-12	cntLinkDown	The link was up and went down.
11-8	cntLinkError	The link was up and the first of a run of link errors occurred.
7-4	cntCellError	The link was up and the first of a run of cell reception errors occurred.
3-0	cntPllLock	The link PLL went from unlocked to locked.

Immediately after a system reset all counters are set to zero; some may become nonzero due to the first attempt to establish the link.

Count1 register

Writing to this register has no effect. Reading it yields a number of 4-bit counters which count up to 15; they don't roll over to zero again once they've reached that maximum.

Bits	Counter name	What it counts
31-28	cntExpFull	The remote reception buffers for one or more VCs became full.
27-24	cntExpAFull	The remote reception buffers for one or more VCs became almost-full.
23-20	cntImpAFull	The PIB FIFO depth went above almost-full or the FLB FIFO depth went below almost-empty.
19-16	cntImpEofe	An end-of-frame-due-to-error (EOFE) sequence was received.
15-12		Unused. Always zero.
11-8	cntExpReset	The PEB was reset.
7-4	cntImpReset	The PIB was reset.
3-0	cntFrameDrop	A frame was dropped either because drop mode was enabled or due to a firmware error where data was presented to the transmitter when it wasn't ready for it.

Immediately after a system reset all counters are set to zero; some may become nonzero due to the first attempt to establish the link.

Slice link status on the front panel

The front panel shows the status of each PGP link from the point of view of the "slice" Sn (FMC and related firmware) where n runs from 1 to 8. Each display shows a link status letter and (in hex) a count of frames received modulo 16.

Display assignments:

Sn	RCE	PGP link
S1	left/upper	A
S2	left/upper	B
S3	left/upper	C
S4	left/upper	D

S5	right/lower	A
S6	right/lower	B
S7	right/lower	C
S8	right/lower	D

Status letters:

Letter	Meaning
L	Good link
N	No link (PLL lock failed)
E	Link error seen, persists until next reset
P	Awaiting PLL lock
F	Link failure, local and remote PGP versions don't match

The E status is latched when an operating link has seen one of the following:

The reception of a NACK.
A CRC error.
A reception decoding error (8b/10b code).
An incorrect frame sequence number.

PIC block assignments

The actual PGP data transfers are scheduled and tracked through Plugin Interface Core (PIC) firmware blocks. There are at most 16 of any type of block and within each type the blocks are assigned numbers starting at zero. Each PGP core needs the use of a Pending Export Block (PEB) for the scheduling of transmissions, an Export Completion Block (ECB) to track the completion of transmissions, a Free List Block to track available buffers for reception and a Pending Import Block (PEB) to track buffers that have data from reception. Each core has its own PEB and PIB but share a common ECB and FLB. The lowest-numbered PIC blocks of each type are used for ethernet ports; one port on Petacache RCEs and two on LCLS RCEs.

An additional PIC block called the Interrupt Summary Block tracks the state of the fault and event flags for all the other PIC blocks in the RCE.

LCLS RCE:

Core	ECB	FLB	PEB	PIB
A	2	2	2	2
B			3	3
C			4	4

Petacache RCE:

Core	ECB	FLB	PEB	PIB
A	1	1	1	1
B			2	2
C			3	3
D			4	4

PIC DCR addresses

Each PIC block is assigned four consecutive DCR addresses in order by block number within type:

Block type	DCR addresses
ISB fault flags	0x2fc-0x2fd
ISB event flags	0x2fe-0x2ff
PEB	0x300-0x33f
ECB	0x340-0x37f
FLB	0x380-0x3bf
PIB	0x3c0-0x3ff

PIC block FIFO threshold registers

The CEM document says that the DCR at offset 3 for each PIC block is reserved. It's now a write-only register used to set the adjustable FIFO thresholds. The almost-empty and almost-full thresholds are settable for any PIC block; the FLB has an additional threshold specific to its function.

Bits	Block types	Threshold
26-18	FLB	resume
17-9	All	almost-full
8-0	All	almost-empty

These are the thresholds that are set after a PIC block is reset:

Block type	almost-empty	almost-full	resume
PEB	2	510	n/a
ECB	5	507	n/a
FLB	5	507	13
PIB	2	510	n/a

The local PGP's reception-almost-full signals are sent to the remote PGP (and the local cntImpAFull is incremented) when either of the following conditions occurs:

The PIB FIFO depth is at or above its almost-full threshold.
The FLB FIFO depth is at or below its almost-empty threshold.

The signals remain asserted until the PIB FIFO depth falls below almost-full AND the FLB FIFO depth is at or above resume. Since a reset of the FLB empties its FIFO and the default values of almost-empty and resume are non-zero, the signals will be asserted and must be cleared either by lowering the resume threshold or by queuing enough TDEs in the FLB FIFO to reach the current resume threshold.

Software port numbers

For convenience, library and application software often refer to PGP "ports" where core A corresponds to port 0, etc. This port number is the same as the type sequence number assigned to PGP plugins by class PortList.

Child pages

PGP core implementation on RCE405 (Gen 1)