Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Draft 2

The firmware for a channel determines the header size and maximum payload size for the channel's frames. Normally this will be a function of the type of channel and the firmware version. These numbers describe the frames as received from the medium; the buffers allocated for the frames will also include space for system overhead.

The same software may be able to handle several different combinations of firmware version, header size and max payload size so the same code container name may appear in more than one table entry. The code is loaded only once but a separate factory instance is created for each entry

The Data Paths table

The lanes (MGTs) allocated to a channel will all feed a particular output (pipe) on the backplane. Those channels that have no lanes will not appear in this table.

Table of Content Zone
locationtop
typelist

Concepts

The low-level interface to an RCE's protocol plug-ins uses abstractions called ports, frames, frame buffers, channels, lanes, pipes and factories.

A port is the RCE end of a communications link similar in concept to a BSD socket. Ports are globally visible but not MT-safe; at a given time at most one task may be waiting for data or receiving data from any given port.

Ports deliver data as frames. The exact content of a frame depends on the transmission protocol being used but the the API recognizes a broad division into header and payload. One frame corresponds to one message on the I/O medium and is delivered in a single frame buffer. In other words all ports implement datagram rather than byte-stream protocols. It's up to higher-level software such as a TCP stack to provide any operations that cross frame boundaries. Each port contains a queue of frames which have arrived and have not yet been consumed by the application.

A channel is a hardware I/O engine capable of DMA to and from system RAM, i.e., a protocol plug-in. An RCE has at most eight channels. Most channels will make use of one or more of the Multi-Gigabit Tranceiver modules (lanes) available in firmware, though some may simply offer access to on-board resources such as DSPs. Each channel has its own port space where each port is identified by a 16-bit unsigned integer. Each port represents a different source of incoming data such as a UDP port or a Petacache flash lane. The actual number of ports available size of the port number space on a channel depends on the channel type and may be as low as one.

With one exception there There is a one-to-one correspondence between the pairs (channel, port no.) and port objects. The exception is a "catch-all" port, which receives data If a channel receives a frame that doesn't " belong " to any other port in the system. There is a catch-all pseudo-channel available with a limit of one port; creating a port on this channel will create a catch-all port. If no catch-all port exists then an orphan frame is dropped and an error message is placed in the system logopen port of the channel, the channel object just counts it before reclaiming the frame buffer.

All the lanes (if any) of a given channel go to one the outputs (pipes) of the backplane. This mapping is fixed by hardware.

Each type of channel has both an offical (unsigned) number and an official short name such as "eth" or "config". Either may be used to look up the corresponding Channel object after system startup. Channels that differ only in the number of lanes they use, e.g., 10 Gb/s ethernet (4) and a slower ethernet (1) will have the same channel type, in this case "eth".

The factory code that creates the right kind of Channel objects for a given type of channel may be already part of the system core or it may be in a container in configuration flash. In the latter case the code must be loaded, relocated and bound to the system core before its first use. An entry point is called in each such loaded factory code module which will register a channel factory object in a central table using a function registry object exported by the core for this purpose. Pre-loaded code must also be registered using this function. Factory code returns values of Channel* though the actual objects pointed to are of derived classes for the specific channel type.

The information needed to initialize the channels find the factory code modules and create the the channel objects is found in configuration flash and/or by probing the hardware.

Channel information in configuration flash

Data container zero in the configuration flash contains tables providing all the information needed to make all channels ready. This includes references to type-of information about the hardware and firmware; this information can't be gotten directly from the Virtex-4 hardware and firmware. It includes references to channel-type specific factory code but not the code itself; other containers will hold that. In addition the tables provide some extra information not actually needed for setup but required to print a summary of what the protocol plug-ins provide.

The tables are called Channels, Factories, Data Paths , Buffers and Strings. Except for Strings each table is an array of plain-old-data structs with the first field being a key value normally equal to the array index. The keys are there to allow tables to refer to one another's entries. A key equal to 0xffffffff signifies the end of the table in which case none of the other fields in the struct have any guaranteed values and should never be read. Thus you'll generate end-of-table sentinels automatically when you erase a flash block before writing tables into it, provided you leave at least one 32-bit word of unwritten space at the end of each table. If you write all the tables in one go then you must provide the end-of-table markers explicitly.

The Strings table is like an ELF string table; NUL-terminated ASCII strings laid end to end. The other tables refer to a string by giving the offset of its first character in the Strings table.

General layout of the container contents

The first words of the container are the 32-bit offsets in the container to the starts of the tables in the following order:

Offset of Channels

Offset of Factories

Offset of Data PathsOffset of Buffers

Offset of Strings

Each of the offsets should be divisible by four.

After the offsets come the tables themselves. No particular order is required, though since String table entries have variable length it's most convenient to place it last.

To make alignment easier we use 32-bit fields wherever possible, even for 16-bit quantities such as port numbers. All of the declarations for the configuration tables are in namespace RCE::config.

Code Block
none
none
typedef uint32_t StringOff; // Offset within the Strings table.

The Channels table

Most of this table will specify the firmware resources used by the channel. The channel type and the number of ports supported are also given.

Code Block
none
Code Block
nonenone
struct Channel {
    uint32_t  key;
    unit32uint32_t  officialTypefactoryKey;     // TheRef. officialto channelFactories type numbertable.
    unit32_t  lanesUsed;      // Bit-mask of lanes used.
    uint32_t  pibsUsed;       // Bit-mask of Pending Import Blocks used.
    uint32_t  pebsUsed;       // Bit-mask of Pending Export Blocks used.
    uint32_t  ecbsUsed;       // Bit-mask of Event Completion Blocks used.
    uint32_t  flbsUsed;       // Bit-mask of Free List Blocks used.
    uint32_t  numPorts;       // Size of port-number space.
    StringOff description;
};

The Factories table

In the Factories table a container name of 0xffffffff marks pre-loaded factory code; otherwise the name is used to find the required container as specified in the RCE document.

Code Block
nonenone

struct Factory {
    uint32_t  key;
    uint32_t  channelTypeNum;
    uint32_t  containerName;
    StringOff description;
};
Code Block
none
none
struct DataPath { Factory {
    uint32_t  key;
    unit32_t  typeNumber;      // The official channel type number.
    uint32_t  key;firmwareVersion; // Firmware version number.
    uint32_t  channelKey;headerSize;      // Bytes.
    uint32_t  pipeNummaxPayloadSize;    // WhereBytes.
  the channel's signals "come out." uint32_t  containerName;
};

The

Buffers

Data Paths table

Each channel comes with a recommendation for the number and size (in bytes) of buffers to be allocated in non-cached memory. Those recommendations are in this table. Note that these are only recommendations ; the system initialization procedure needs to consider all the recommendations together along with the amount of memory actually available.The lanes (MGTs) allocated to a channel will all feed a particular output (pipe) on the backplane. Those channels that have no lanes will not appear in this table.

note
Code Block
none
none
struct BufferDataPath {
    uint32_t  key;
    uint32_t  channelKey;
    uint32_t  bufferCount;
    uint32_t bufferSize;pipeUsed;    // Where the channel's signals "come out."
};

You should give buffer sizes large enough for the largest frame expected on the medium. The system will add to each size some space for overhead.

Example configuration tables

Here's what the tables would look like for an RCE that defines nothing but an ethernet LAN and configuration flash, which are standard for all RCEs. We assume that the ethernet uses lanes 0-3, PIB 0, PEB 0, ECB 0, FLB0 and feeds pipe 54.

The Channels table:

KeyOfficial

type no.Factory key

Lanes

PIBs

PEBs

ECBs

FLBs

Ports

FLBsFirmware version

Description

0

0

0xf

1

1

1

1

65536

1

"LAN"

1

1

0

0

0

0

0

1

1

"Configuration flash"

2

2

0

0

0

0

0

1

1

"Catch-all channel"

Ethernet allows 65536 ports so as not to restrict the port space of UDP or TCP.

The Factories table:

KeyOfficial type no.

Channel type no.

Firmware version

Header size

Max payload size

Container name

Description

0

0

1

xxx

xxx

<name1>

"Virtex-4 ethernet"

1

1

1

xxx

xxx

0xffffffff

"Virtex-4 configuration flash"

2

2

0xffffffff"

"Virtex-4 catch-all channel"

The Data Paths table:

Key

Channel key

Pipe no.

0

0

54

The Buffers table:

Key

Channel key

No. of buffers

Buffer size

0

0

32

1518

1

1

16

2048

Here we assume that the firmware delivers all ethernet framing bytes as well as the payload and that jumbo frames are not allowed. We assume that the formware delivers configuration flash data in units of pages and that a page is 2K bytes long.

How the tables appear in RAM

We copy the tables without alteration and set the members of an instance of the following structure:

Code Blocknonenone

struct Tables {
    unsigned numChannels;
    Channel *channel;      // Pointer to first element of table.
    unsigned numFactories;
    Factory *factory;
    unsigned numDataPaths;
    DataPath *dataPath;
    unsigned numBuffers;
    Buffer *buffer;
};

The Data Paths table:

Key

Channel key

Pipe no.

0

0

54

How the tables appear in RAM

We copy the tables without alteration and set the members of an instance of the following structure:

Code Block
none
none

struct Tables {
    unsigned numChannels;
    Channel *channel;      // Pointer to first element of table.
    unsigned numFactories;
    Factory *factory;
    unsigned numDataPaths;
    DataPath *dataPath;
};

Use case: System startup

We'll need to use a quick and dirty way to read the configuration tables from flash; either that or just use faked-up configuration tables to create the config. flash Channel object. Once we can have that we can create the rest of the Channel objects.

  1. Boot code
    1. Loads and starts the system core.
  2. System core
    1. Initializes the CPU.
    2. Initializes RTEMS.
    3. Initializes any extra C++ support.
    4. Sets up the MMU's TLB and enables the MMU.
    5. Uses a basement level technique to read the tables from configuration flash.
    6. Creates and registers the channel factory for configuration flash.
    7. Creates and registers the configuration flash channel.
    8. Allocates frame buffers in uncached RAM for config. flash pages.
    9. Gives the frame buffers to the configuration flash channel and enables it.
    10. Creates the default instance of the dynamic linker.
    11. Loads and links the factory code marked as loadable in the Factories table.
    12. Calls the entry points in the factory code modules just loaded.
    13. Iterates on the Channels table and creates all Channel objects.
    14. Iterates over all Channel objects (except config. flash) to get their recommended buffer allocations.
    15. Allocate the buffers in uncached memory.
    16. Assigns buffers to channels.
    17. Enables all Channel objects not already enabled.
    18. Loads and links the application code using the default dynamic linker.
    19. Initializes the network.
      1. Initializes each ethernet channel.
      2. Initializes IP, UDP, TCP and BSD sockets.
      3. Gets a DHCP lease if required.
    20. Calls the application entry point.
  3. Factory code entry point.
    1. Creates a channel factory object using the given configuration table info as input.
    2. Registers the object with the core.

Use case: Frame import

TBD.

Use case: Frame export

TBD.

Low-level API

Official RCE channel type numbers

These are given in a header file made available to both core and application code. The numbers are members of an enumeration.

Code Block
none
none

namespace RCE::chantype {

    enum {
        ETHERNET,
        CONFIG_FLASH,
        CATCH_ALL,
        ...
    } Number;
}

Channel factory registry

An instance of this class will be exported by the core code using some design pattern such as Singleton or functional equivalent. This instance is created during system startup and lasts until system shutdown. Assignment or copying of instances is forbidden.

Code Block
none
none

class ChannelFactoryRegistry {

    ChannelFactoryRegistry();

    // Register a factory and assign it a sequence number i starting from zero.
    // The argument may not be a null pointer.
    void addFactory(ChannelFactory*);

    int numFactories();

    // If i is a valid sequence number then return the i'th factory
    // else return a null pointer.
    ChannelFactory* factory(int i);

);

Channel registry

An instance of this class will be exported by the core code using some design pattern such as Singleton or functional equivalent. This instance is created during system startup and lasts until system shutdown. Assignment or copying of instances is forbidden.

Code Block
none
none

class ChannelRegistry {

    ChannelRegistry();

    // Register a channel and assign it a sequence number i starting from zero.
    // The argument may not be a null pointer.
    void addChannel(Channel*);

    int numChannels();

    // If i is a valid sequence number then return the i'th channel
    // else return a null pointer.
    Channel* factory(int i);

);

Use case: System startup

  1. Boot code
    1. Loads and starts the system core.
  2. System core
    1. Initializes the CPU.
    2. Initializes RTEMS.
    3. Initializes any extra C++ support.
    4. Sets up the MMU's TLB and enables the MMU.
    5. Registers the factories for configuration flash and catch-all.
    6. Creates the configuration flash channel.
    7. Allocates uncached RAM for config. flash pages.
    8. Enables the configuration flash channel.
    9. Copies the Channels table and related info from configuration flash into RAM.
    10. Creates the default instance of the dynamic linker.
    11. Loads and links the factory code marked as loadable in the Factories table.
    12. Calls all factory code entry points so that all factories are registered.
    13. Walks the Channels table and creates all Channel objects.
    14. Walks the Buffers table and allocates uncached memory for I/O buffers.
    15. Assigns buffers to channels.
    16. Enables all Channel objects not already enabled.
    17. Loads and links the application code using the default dynamic linker.
    18. Initializes the network.
      1. Initializes each ethernet channel.
      2. Initializes IP, UDP, TCP and BSD sockets.
      3. Gets a DHCP lease if required.
    19. Calls the application entry point.
  3. Factory code entry point.
    1. Registers with the core an object that creates the right kind of Channel objects.

Use case: Frame import

TBD.

Use case: Frame export

TBD.

Low-level API

Official RCE channel type numbers and names

These are given in a header file made available to both core and application code. The numbers are members of an enumeration and the names+descriptions are given in a static array.

Code Block
nonenone

namespace RCE::chantype {

    enum {
        ETHERNET,
        CONFIG_FLASH,
        CATCH_ALL,
        ...
    } Numbers;

    const char *name = {
        "eth",
        "config",
        "catch-all"
        ...
    };

    const char *description = {
        "ethernet",
        "configuration flash",
        "catch-all channel",
        ...
    };
}

Channel factory

The abstract base class for objects that manufacture Channel instances. Channel factories are created during system startup and last until system shutdown. Assignment or copying of instances is forbidden.

Code Block
none
none
class ChannelFactory {
public:

    ChannelFactoryRCE::chantype::Number typeNumber() const;

    virtualconst Channelchar* createChanneltypeName(
        unsigned ident,) const;

    unsigned firmwareVersion() const;

    unsigned headerSize() const;

  // Assigned by init. code.unsigned maxPayloadSize() const;

    unsigned numBuffersAdvised() const;

    unsigned channelTableKey,codeContainerName() const;

    const    RCE::config::Tables&) = 0;
char* description() const;

    virtual ~ChannelFactory();
};

Channel type registry

An instance of this class will be exported by the core code using some design pattern such as Singleton or functional equivalent. This instance is created during system startup and lasts until system shutdown. Assignment or copying of instances is forbidden.

Code Block
nonenone

class ChanneTypeRegistry {

    ChannelTypeRegistry();

    // Register a Channel factory using its official RCE type number.
    void registerChannelFactoryChannel* createChannel(
        unsigned channelTableKey,
        RCE::config::Tables&
    ) = 0;

protected:

    ChannelFactory(
        unsigned officialNumberfactoryTableKey,
         ChannelFactory const RCE::config::Tables&,
	const char*   );

    ChannelFactory& channelFactory(unsigned officialNumbertypeName,
	unsigned numBuffersAdvised,
	const char *description
    );

    unsignedvirtual officialNumber(const char *officialName~ChannelFactory();
)};

Channel

A channel object manages a set of I/O buffers and a particular instance of a protocol plug-in. At the beginning of each buffer the channel will build a SuperFrame FrameBuffer object. Each channel object is created at system startup, initially in a disabled state. Afterward the system startup code will allocate the buffers, feed them to the channel objects and then enable them. Channel objects live until system shutdown.

Each channel creates and destroys Port objects on demand. Each port is assigned a unique ID numberone of the channel's legal port numbers not used by any other port. The client code may request a particular ID in the legal range port number for the type of channel, e.g., a well-known TCP port number. The client may also allow the channel to assign an ID not currently in use by any port.

A channel knows how to derive a port number from the header of a frame it has received.

Channel and its derived classes do not allow copying or assignment of their instances.

Code Block
none
none
class Channel {
public:

    ChannelFactory* factory() const; // Set the system's channel ID and the channel type number. Leaves
  // the Channel in the disabled state.
  Channel(unsigned systemAssignedId, unsigned typeNum);

  unsigned idThe factory that created me.

    unsigned lanesUsed() const;

    unsigned pibsUsed() const;

    unsigned pebsUsed() const;

    unsigned ecbsUsed() const;

    unsigned flbsUsed() const;

    unsigned pipeUsed() const;

    const char* description() const;


    unsigned typeNumorphanFrameCount() const;

    // Create a Port with a specified port number.
    // Returns 0null if the number is already taken.
    virtual Port* createPort(int portNum) = 0;

    // Create a Port with a number assigned by the channel.
    // Returns 0null if no port number is free.
    virtual Port* createPort() = 0;

    virtual void destroyPort(Port *) = 0;

    // Return 0null or the Port with the given number.
    Port *findPortgetPort(unsigned portNum) const;

  // Manage a new buffer.
  void addBuffer(void *);

  // Reclaim a buffer that was given out by// oneStart ofmanaging thea Ports
new buffer //or createdreclaim byan thisold channelone.
    void reclaimBufferclaimBuffer(void *);

    void enable();

    void disable();

    void isEnabled() const;

    // Return null or a pointer to the Port that should
    // Returnreceive -1 or the number of the port to which the frame should go,
  // assuming the frame belongs to this channel.
  virtual int getPortNumber(void *frameHeader) const;

  virtual unsigned frameHeaderSize() const;

protected:

  // Does nothing because Channels last until the next system shutdown.the data.
    virtual Port* deducePort(void *frameHeader) const;

protected:
    // Create a Channel which is initially in the disabled state.
    Channel(
        unsigned channelTableKey,
        RCE::config::Tables&,
	ChannelFactory *,
    );

  virtual ~Channel();

};

Port

Remembers the Channel that created it.

Allows client code to wait for new frames.

Reclaims frames frame buffers that the client code has finished using AND that were provided by this particular port(or has just
created).

Code Block
none
none
class Port {
public:
  Port(Channel &*, unsigned portNumportNumber);
  Channel&* channel() const;
  unsigned portNumportNumber() const;
  virtual void* wait(); // Returns a pointer to the frame header when one becomes available.
  void reclaimBufferclaimBuffer(void *frameHeader) const;
};

API implementation classes

These classes are not exposed to the client but are used internally by the API.

Class

SuperFrame

FrameBuffer

A SuperFrame FrameBuffer instance is placed at the beginning of a buffer that will also hold the frame proper immediately after the SuperFrame. The instances are created at system startup and live until system shutdown.

A SuperFrame FrameBuffer contains regions of fixed size used in the management of the frame itself and its buffer:

  • One or more links used to make singly-linked lists.
  • The firmware's in-memory descriptor. The firmware TDE for the frame points to the descriptor.
  • Status. The firmware writes operation completion status here.

Immediately following the end of the SuperFrame FrameBuffer buffer in we have first the frame's header and then its payload.

The links, descriptor and status areas have the same sizes for all channel types.

The descriptor must begin on a 64-byte boundary; for ease of layout the entire buffer also has that alignment. The buffer should also begin on a cache line boundary in order to reduce cache thrashing but since a cache line is only 32 bytes long this requirement is already met.

SuperFrame is used internally by the API; no client code is ever given any of its instancesFrameBuffer instances may not be copied or assigned.

Code Block
none
none
class SuperFrameFrameBuffer {
public:
    SuperFrameFrameBuffer(void *bufferAddress);
    SuperFrameFrameBuffer* link(unsigned linkNum) const;
    const Descriptor& descriptor() const;
    const OpStatus& status() const;
    char* header() const;
    char* payload() const;
private:
    SuperFrameFrameBuffer* link[N];
    Descriptor descr;
    OpStatus status;
    // The frame header starts here.
};