PGP version
Gen 1 hardware implements PGP version 1.
Number of instances per RCE
Petacache RCEs have four PGP cores named A through D while LCLS RCEs have three named A through C.
Core device registers
Each PGP core has four Device Control Registers, which give information about each PGP link as seen from the "FPGA" (the RCE). The register base addresses are:
PGP core |
First DCR addr |
---|---|
A |
0x2e0 |
B |
0x2e4 |
C |
0x2e8 |
D (petacache only) |
0x2ec |
The individual registers for each core:
Offset |
Name |
Description |
---|---|---|
0 |
Control |
Selects and displays the mode of operation. |
1 |
Status |
Summarizes the current status of the link. |
2 |
Count0 |
A set of 4-bit counters. |
3 |
Count1 |
More 4-bit counters. |
In the register descriptions that follow the bit numbering used is the same as in the VHDL file: the most significant bit is numbered 31 and the least significant is numbered zero.
The names of some of the register fields use "pib" to mean the remote PGP or the link to it; perhaps "rmt" would have been a better choice.
Control register
Bit |
Name |
Purpose |
---|---|---|
31-20 |
|
Ignored on write. Read back as zeros. |
19 |
countReset |
As long as this bit is 1 all the counters for this core will be reset to and held at zero. Clear the bit when you want the counters to become active again. |
18 |
mgtLoopback |
Set this bit to place the core's Multi-Gigabit Transceiver (MGT) into loopback mode, so that any data sent is received locally instead of by the remote PGP. |
17 |
pibRelink |
Set this bit to set the link state to ST_LOCK (see below) and hold it there. Clear the bit again to try to reestablish the link. |
16 |
enableDrop |
This bit controls the what the core will do when the PGP transmit buffer is "almost full". When it's set the core will discard any new frame data coming from the assigned PEB. When it's clear the core will cause the PEB to stop sending new data. In either case normal behavior will resume when enough buffer space becomes available. |
15-1 |
|
Ignored on write. Read back as zeros. |
0 |
pgpSeqError |
Ignored on write. When read a value of 1 means that there has been a PGP sequencing error for this core. This occurs when the ACK FIFO becomes full or the frame sequence number FIFO becomes empty. This condition will persist until the link is dropped and reestablished (see pibRelink above). |
After a system reset both loopback and drop mode are disabled.
If you change the value of mgtLoopback or enableDrop you must also reset the link using pibRelink in order for the new operating mode to go into effect. For example, in order to set loopback mode for link A you must first write 0x00060000 to DCR 0x2e0 and then write 0x00040000.
Status register
Writing to this register has no effect. It may be read to get a summary of the core's current state.
Bit |
Name |
Purpose |
---|---|---|
31-24 |
vc3RemBuffFull |
Each of these single-bit flags shows the status of the remote PGP's receive buffer for a virtual channel. |
|
vc3RemBuffAFull |
Each such buffer may be asserting "full" and/or "almost full". |
|
vc2RemBuffFull |
|
|
vc2RemBuffAFull |
|
|
vc1RemBuffFull |
|
|
vc1RemBuffAFull |
|
|
vc0RemBuffFull |
|
|
vc0RemBuffAFull |
|
23-22 |
pibLock |
The state of the MGT's phase-locked loops (PLLs); the high-order bit is for the transmit (Tx) PLL and the other is for the receive (Rx) PLL. A value of 1 means that the PLL is locked. |
21 |
mgtInverted |
A value of 1 means that the MGT is inverting the signals it receives and transmits, something that happens automatically if needed to communicate with the remote PGP. |
20 |
pibFail |
A value of 1 means that the link state is ST_FAIL (see below). |
19 |
pibLinkReady |
A value of 1 means that the link state is ST_EN or ST_SKP (see below). |
18-16 |
pibState |
The state of the link to the remote PGP. See the table below. |
15-8 |
remoteVersion |
The version number of the remote PGP core. This changes value when a valid version sequence is received when this core is not transmitting its own version number. |
7-0 |
localVersion |
The version number of this PGP core. |
A finite state machine controls the state of the link to the remote PGP. The possible states are:
pibState value |
State name |
Meaning |
---|---|---|
1 |
ST_LOCK |
The initial state. Waiting for the PLL to lock. |
2 |
ST_IDLE |
Transmitting and receiving IDLE sequences. |
3 |
ST_TRAIN |
Transmitting and receiving training sequences. |
4 |
ST_VER |
Transmitting and receiving PGP core version numbers. |
5 |
ST_EN |
Ready to transmit (or already transmitting) cell data. |
6 |
ST_SKP |
Transmitting the Skip sequence that must follow every cell sent. |
7 |
ST_FAIL |
Failed to reach the state ST_EN. |
The normal sequence of states is ST_LOCK, ST_IDLE, ST_TRAIN, ST_VER, ST_EN, ST_SKP, ST_EN, ST_SKP, ST_EN, etc. If any of the steps leading to state ST_EN fails then the link enters the state ST_FAIL and remains there until forced back to the ST_LOCK state. ST_LOCK will be entered upon system reset, if the remote PGP sends IDLE sequences or if the pibRelink bit is set in the control register (see above). Generally failure occurs when the expected type of sequences are not seen coming from the remote PGP; the ST_VER step will in addition fail if the remote version isn't the same as the local version.
Immediately after a system reset the link goes into the ST_LOCK state with all the other state information set accordingly, after which the local PGP core tries to bring the link up.
Count0 register
Writing to this register has no effect. Reading it yields a number of 4-bit counters which count up to 15; they don't roll over to zero again once they've reached that maximum.
Bits |
Counter name |
What it counts |
---|---|---|
31-28 |
cntFlowError |
The PIB FIFO condition changes from not-full to full. |
27-24 |
cntStatError |
The PEB's internal 16-element completion-status FIFO couldn't accept a new value. |
23-20 |
cntShiftError |
A firmware error where data is presented to the receiver when it isn't ready for it. |
19-16 |
cntNack |
The link was up and we received the first of a run of NACKs. |
15-12 |
cntLinkDown |
The link was up and went down. |
11-8 |
cntLinkError |
The link was up and the first of a run of link errors occurred. |
7-4 |
cntCellError |
The link was up and the first of a run of cell reception errors occurred. |
3-0 |
cntPllLock |
The link PLL went from unlocked to locked. |
Immediately after a system reset all counters are set to zero; some may become nonzero due to the first attempt to establish the link.
Count1 register
Writing to this register has no effect. Reading it yields a number of 4-bit counters which count up to 15; they don't roll over to zero again once they've reached that maximum.
Bits |
Counter name |
What it counts |
---|---|---|
31-28 |
cntExpFull |
The remote reception buffers for one or more VCs became full. |
27-24 |
cntExpAFull |
The remote reception buffers for one or more VCs became almost-full. |
23-20 |
cntImpAFull |
The PIB FIFO depth went above almost-full or the FLB FIFO depth went below almost-empty. |
19-16 |
cntImpEofe |
An end-of-frame-due-to-error (EOFE) sequence was received. |
15-12 |
|
Unused. Always zero. |
11-8 |
cntExpReset |
The PEB was reset. |
7-4 |
cntImpReset |
The PIB was reset. |
3-0 |
cntFrameDrop |
A frame was dropped either because drop mode was enabled or due to a firmware error where data was presented to the transmitter when it wasn't ready for it. |
Immediately after a system reset all counters are set to zero; some may become nonzero due to the first attempt to establish the link.
Slice link status on the front panel
The front panel shows the status of each PGP link from the point of view of the "slice" Sn (FMC and related firmware) where n runs from 1 to 8. Each display shows a link status letter and (in hex) a count of frames received modulo 16.
Display assignments:
Sn |
RCE |
PGP link |
---|---|---|
S1 |
left/upper |
A |
S2 |
left/upper |
B |
S3 |
left/upper |
C |
S4 |
left/upper |
D |
|
|
|
S5 |
right/lower |
A |
S6 |
right/lower |
B |
S7 |
right/lower |
C |
S8 |
right/lower |
D |
Status letters:
Letter |
Meaning |
---|---|
L |
Good link |
N |
No link (PLL lock failed) |
E |
Link error seen, persists until next reset |
P |
Awaiting PLL lock |
F |
Link failure, local and remote PGP versions don't match |
The E status is latched when an operating link has seen one of the following:
- The reception of a NACK.
- A CRC error.
- A reception decoding error (8b/10b code).
- An incorrect frame sequence number.
PIC block assignments
The actual PGP data transfers are scheduled and tracked through Plugin Interface Core (PIC) firmware blocks. There are at most 16 of any type of block and within each type the blocks are assigned numbers starting at zero. Each PGP core needs the use of a Pending Export Block (PEB) for the scheduling of transmissions, an Export Completion Block (ECB) to track the completion of transmissions, a Free List Block to track available buffers for reception and a Pending Import Block (PEB) to track buffers that have data from reception. Each core has its own PEB and PIB but share a common ECB and FLB. The lowest-numbered PIC blocks of each type are used for ethernet ports; one port on Petacache RCEs and two on LCLS RCEs.
An additional PIC block called the Interrupt Summary Block tracks the state of the fault and event flags for all the other PIC blocks in the RCE.
LCLS RCE:
Core |
ECB |
FLB |
PEB |
PIB |
---|---|---|---|---|
A |
2 |
2 |
2 |
2 |
B |
|
|
3 |
3 |
C |
|
|
4 |
4 |
Petacache RCE:
Core |
ECB |
FLB |
PEB |
PIB |
---|---|---|---|---|
A |
1 |
1 |
1 |
1 |
B |
|
|
2 |
2 |
C |
|
|
3 |
3 |
D |
|
|
4 |
4 |
PIC DCR addresses
Each PIC block is assigned four consecutive DCR addresses in order by block number within type:
Block type |
DCR addresses |
---|---|
ISB fault flags |
0x2fc-0x2fd |
ISB event flags |
0x2fe-0x2ff |
PEB |
0x300-0x33f |
ECB |
0x340-0x37f |
FLB |
0x380-0x3bf |
PIB |
0x3c0-0x3ff |
PIC block FIFO threshold registers
The CEM document says that the DCR at offset 3 for each PIC block is reserved. It's now a write-only register used to set the adjustable FIFO thresholds. The almost-empty and almost-full thresholds are settable for any PIC block; the FLB has an additional threshold specific to its function.
Bits |
Block types |
Threshold |
---|---|---|
26-18 |
FLB |
resume |
17-9 |
All |
almost-full |
8-0 |
All |
almost-empty |
These are the thresholds that are set after a PIC block is reset:
Block type |
almost-empty |
almost-full |
resume |
---|---|---|---|
PEB |
2 |
510 |
n/a |
ECB |
5 |
507 |
n/a |
FLB |
5 |
507 |
13 |
PIB |
2 |
510 |
n/a |
The local PGP's reception-almost-full signals are sent to the remote PGP (and the local cntImpAFull is incremented) when either of the following conditions occurs:
- The PIB FIFO depth is at or above its almost-full threshold.
- The FLB FIFO depth is at or below its almost-empty threshold.
The signals remain asserted until the PIB FIFO depth falls below almost-full AND the FLB FIFO depth is at or above resume. Since a reset of the FLB empties its FIFO and the default values of almost-empty and resume are non-zero, the signals will be asserted and must be cleared either by lowering the resume threshold or by queuing enough TDEs in the FLB FIFO to reach the current resume threshold.
Software port numbers
For convenience, library and application software often refer to PGP "ports" where core A corresponds to port 0, etc. This port number is the same as the type sequence number assigned to PGP plugins by class PortList.