Publications

Requirements

  • Support targeting "point-to-point"
    • But network mode is possible
  • 3 layer of Ethernet:
    • Layer1: PHY
    • Layer2: MAC/FEC
    • Layer3: HTSP

  • Layer 3 Ethernet framing

    • Wrapper on the Ethernet MAC
    • Targeting 100G Hard IP MAC
      • 512-bit AXI Stream interface
      • IEEE 802.3bj Reed-Solomon Forward Error Correction (RS-FEC)
      • Frame Check Sequence (FCS) checking, adding and deleting
  • 512-bit header and 48-bit footer
    • Header and payload always 512-bit wide
    • Simplify the logic for 100GbE MAC interface (no AxiStreamShifting required because matched to MAC's AXI stream 512-bit width)
  • Chuck up the stream to MAX_SIZE (8196B default)
    • 98.9% efficient = 8196B/(8196B + 4B FCS + 12B intergap + 8B preamble + 64B HTSP Ethernet Header + 4B HTSP Ethernet Footer)
    • Smaller MAX_SIZE lower the latency for publishing the virtual channel "pause" status to the remote side but increases overhead
  • Use the MAC's FCS for error checking
  • Virtual channel flow control done at the L3 layer (instead of ETH pause)
  • Support up to 16 Virtual channels on the same link
    • Same as PGPv4


Word#Word's BYTETypeNameDescriptionNote
0[5:0]HeaderDestMacDestination MAC
  • Network mode only
0[11:6]HeaderSrcMacSource MAC
  • Network mode only
0[13:12]HeaderEtherTypeTBD Value
  • Network mode only
014HeaderVersion0x1
  • Hard coded
015HeaderTIDTransaction ID
  • Increments once per packet
  • Used for out-of-order frame reorganizing in network mode
0[17:16]HeaderPauseVirtual Channel Pause
  • 1 bit per Virtual Channel
018HeaderVCVirtual Channel Index
  • BIT[3:0]: Virtual Channel Index
  • BIT[7:4]: Reserved
019HeadertUserFirstfirst 8-bits of tUser
020HeaderOpCodeEnOP-code Enable
0[29:20]HeaderReservedReserved
  • Zeros
0[31:30]HeaderHdrXsum16-bit Header Checksum
  • Header Only
0[47:32]HeaderOpCodeData128-bit OP-code Data
  • Supports 64-bit timestamp + more information
0[63:48]HeaderUserData128-bit User Data
  • Sampled every packet sent
1[63:0]PayloadAXIS DataUser Payload Data
...............
(MAX_SIZE/64)[63:0]PayloadAXIS DataUser Payload Data
(MAX_SIZE/64)+10FootertKeepLastTKEEP on last payload word
  • Converted from byte map to an integer
(MAX_SIZE/64)+11FooterEOF/EOFEEOF and EOFE marker
  • BIT0 = EOF, BIT1 = EOFE
(MAX_SIZE/64)+1[3:2]FooterPauseVirtual Channel Pause
  • paused bits are sampled and latched during the payload transport
(MAX_SIZE/64)+1[5:4]FooterPayloadSizeNumber of bytes in payload
  • Units of bytes


Performance

Going "point-to-point" we were measuring 97.3 Gb/s bandwidth for 1MB frames using 8196B HTSP burst size.  97.3 Gb/s is < 98.9% theoretical limit, which we think is related to some unaccounted inefficiency in the CMAC4 hard IP core that we used in this testing.  The DMA bandwidth limit in this test was 101 Gb/s, which means the DMA was not back pressuring on the HTSP link.  Auto-polling register channel access on the HTSP link on a different virtual channel was enabled in this testing.


  • No labels