PGP version
Gen 1 hardware implements PGP version 1.
Number of instances per RCE
Petacache RCEs have four PGP cores named A through D while ATLAS TDAQ RCEs have three named A through C.
Core device registers
Each PGP core has four Device Control Registers. The register base addresses are:
PGP core |
First DCR addr |
---|---|
A |
0x2e0 |
B |
0x2e4 |
C |
0x2e8 |
D (petacache only) |
0x2ec |
The individual registers for each core:
Offset |
Name |
Description |
---|---|---|
0 |
Control |
Selects and displays the mode of operation. |
1 |
Status |
Summarizes the current status of the link. |
2 |
Count0 |
A set of 4-bit counters. |
3 |
Count1 |
More 4-bit counters. |
In the register descriptions that follow the bit numbering used is the same as in the VHDL file: the most significant bit is numbered 31 and the least significant is numbered zero.
Control register
Bit |
Name |
Purpose |
---|---|---|
31-20 |
|
Ignored on write. Read back as zeros. |
19 |
countReset |
As long as this bit is 1 all the counters for this core will be reset to and held at zero. Clear the bit when you want the counters to become active again. |
18 |
mgtLoopback |
Set this bit to place the core's Multi-Gigabit Transceiver (MGT) into loopback mode, so that any data sent is received locally instead of by the remote PGP. |
17 |
pibRelink |
Set this bit to set the link state to ST_LOCK (see below) and hold it there. Clear the bit again to try to reestablish the link. |
16 |
enableDrop |
This bit controls the what the core will do when the PGP transmit buffer is "almost full". When it's set the core will discard any new frame data coming from the assigned PEB. When it's clear the core will cause the PEB to stop sending new data. In either case normal behavior will resume when enough buffer space becomes available. |
15-1 |
|
Ignored on write. Read back as zeros. |
0 |
pgpSeqError |
Ignored on write. When read a value of 1 means that there has been a PGP sequencing error for this core. This occurs when the ACK FIFO becomes full or the frame sequence number FIFO becomes empty. This condition will persist until the link is dropped and reestablished (see pibRelink above). |
After a system reset both loopback and drop mode are disabled.
Status register
Writing to this register has no effect. It may be read to get a summary of the core's current state.
Bit |
Name |
Purpose |
---|---|---|
31-24 |
vc3RemBuffFull |
These bits show the status of the remote PGP's receive buffer for each virtual channel. |
|
vc3RemBuffAFull |
Each such buffer may be asserting "full" and/or "almost full". |
|
vc2RemBuffFull |
|
|
vc2RemBuffAFull |
|
|
vc1RemBuffFull |
|
|
vc1RemBuffAFull |
|
|
vc0RemBuffFull |
|
|
vc0RemBuffAFull |
|
23-22 |
pibLock |
The state of the MGT's phase-locked loops (PLLs); the high-order bit is for the transmit (Tx) PLL and the other is for the receive (Rx) PLL. A value of 1 means that the PLL is locked. |
21 |
mgtInverted |
A value of 1 means that the MGT is inverting the signals it receives and transmits, something that happens automatically if needed to communicate with the remote PGP. |
20 |
pibFail |
A value of 1 means that the link state is ST_FAIL (see below). |
19 |
pibLinkReady |
A value of 1 means that the link state is ST_EN or ST_SKP (see below). |
18-16 |
pibState |
The state of the link to the remote PGP. See the table below. |
15-8 |
remoteVersion |
The version number of the remote PGP core. This changes value when a valid version sequence is received when this core is not transmitting its own version number. |
7-0 |
localVersion |
The version number of this PGP core. |
A finite state machine controls the state of the link to the remote PGP. The possible states are:
pibState value |
State name |
Meaning |
---|---|---|
1 |
ST_LOCK |
The initial state. Waiting for the PLL to lock. |
2 |
ST_IDLE |
Transmitting and receiving IDLE sequences. |
3 |
ST_TRAIN |
Transmitting and receiving training sequences. |
4 |
ST_VER |
Transmitting and receiving PGP core version numbers. |
5 |
ST_EN |
Ready to transmit (or already transmitting) cell data. |
6 |
ST_SKP |
Transmitting the Skip sequence that must follow every cell sent. |
7 |
ST_FAIL |
Failed to reach the state ST_EN. |
The normal sequence of states is ST_LOCK, ST_IDLE, ST_TRAIN, ST_VER, ST_EN, ST_SKP, ST_EN, ST_SKP, ST_EN, etc.
If any of the steps leading to state ST_EN fails then the link enters the state ST_FAIL and remains there until forced back to the ST_LOCK state. ST_LOCK will be entered upon system reset, if the remote PGP sends IDLE sequences or if the pibRelink bit is set in the control register (see above). Generally failure occurs when the expected type of sequences are not seen coming from the remote PGP; the ST_VER step will in addition fail if the remote version isn't the same as the local version.
Immediately after a system reset the link goes into the ST_LOCK state with all the other state information set accordingly, after which the local PGP core tries to bring the link up.
Count0 register
Writing to this register has no effect. Reading it yields a number of 4-bit counters which count up to 15; they don't roll over to zero again once they've reached that maximum.
Bits |
Counter name |
What it counts |
---|---|---|
31-28 |
cntFlowError |
PIB FIFO transitions from not-full to full. |
27-24 |
cntStatError |
The PEB's internal 16-element completion-status FIFO couldn't accept a new value. |
23-20 |
cntShiftError |
A firmware error where data is presented to the receiver when it isn't ready for it. |
19-16 |
cntNack |
The link was up and we received the first of a run of NACKs. |
15-12 |
cntLinkDown |
The link was up and went down. |
11-8 |
cntLinkError |
The link was up and the first of a run of link errors occurred. |
7-4 |
cntCellError |
The link was up and the first of a run of cell reception errors occurred. |
3-0 |
cntPllLock |
The link PLL went from unlocked to locked. |
Immediately after a system reset all counters are set to zero; some may become nonzero due to the first attempt to establish the link.w
Count1 register
Writing to this register has no effect. Reading it yields a number of 4-bit counters which count up to 15; they don't roll over to zero again once they've reached that maximum.
Bits |
Counter name |
What it counts |
---|---|---|
31-28 |
cntExpFull |
The remote reception buffers for one or more VCs became full. |
27-24 |
cntExpAFull |
The remote reception buffers for one or more VCs became almost-full. |
23-20 |
cntImpAFull |
The PIB FIFO depth went above almost-full or the FLB FIFO depth went below almost-empty. |
19-16 |
cntImpEofe |
An end-of-frame-due-to-error (EOFE) sequence was received. |
15-12 |
|
Unused. Always zero. |
11-8 |
cntExpReset |
The PEB was reset. |
7-4 |
cntImpReset |
The PIB was reset. |
3-0 |
cntFrameDrop |
A frame was dropped either because drop mode was enabled or due to a firmware error where data was presented to the transmitter when it wasn't ready for it. |
Immediately after a system reset all counters are set to zero; some may become nonzero due to the first attempt to establish the link.
PIC block assignments
The actual PGP data transfers are scheduled and tracked through Plugin Interface Core (PIC) firmware blocks. There are at most 16 of any type of block and within each type the blocks are assigned numbers starting at zero. Each PGP core needs the use of a Pending Export Block (PEB) for the scheduling of transmissions, an Export Completion Block (ECB) to track the completion of transmissions, a Free List Block to track available buffers for reception and a Pending Import Block (PEB) to track buffers that have data from reception. Each core has its own PEB and PIB but share a common ECB and FLB. The lowest-numbered PIC blocks of each type are used for ethernet ports; one port on Petacache RCEs and two on ATLAS RCEs.
An additional PIC block called the Interrupt Summary Block tracks the state of the fault and event flags for all the other PIC blocks in the RCE.
ATLAS TDAQ RCE:
Core |
ECB |
FLB |
PEB |
PIB |
---|---|---|---|---|
A |
1 |
1 |
2 |
2 |
B |
|
|
3 |
3 |
C |
|
|
4 |
4 |
Petacahe RCE:
Core |
ECB |
FLB |
PEB |
PIB |
---|---|---|---|---|
A |
1 |
1 |
1 |
1 |
B |
|
|
2 |
2 |
C |
|
|
3 |
3 |
D |
|
|
4 |
4 |
PIC DCR addresses
Each PIC block is assigned four consecutive DCR addresses in order by block number within type:
Block type |
DCR addresses |
---|---|
ISB fault flags |
0x2fc-0x2fd |
ISB event flags |
0x2fe-0x2ff |
PEB |
0x300-0x33f |
ECB |
0x340-0x37f |
FLB |
0x380-0x3bf |
PIB |
0x3c0-0x3ff |
PIC block FIFO threshold registers
The CEM document says that the DCR at offset 3 for each PIC block is reserved. It's now a write-only register used to set the adjustable FIFO thresholds. The almost-empty and almost-full thresholds are settable for any PIC block; the FLB has an additional threshold specific to its function.
Bits |
Block types |
Threshold |
---|---|---|
26-18 |
FLB |
resume |
17-9 |
All |
almost-full |
8-0 |
All |
almost-empty |
The FLB resume threshold has no effect on the transfer of data or on the setting of the FLB event or fault flags. It only affects the behavior or the cntImpAFull counter (see above). It ensures that if the counter is incremented due to the FLB FIFO's becoming almost-empty then the counter can't be incremented again until the FLB FIFO depth reaches the resume threshold.
Software port numbers
For convenience, library and application software often refer to PGP "ports" where core A corresponds to port 0, etc. This port number is the same as the type sequence number assigned to PGP plugins by class PortList.