Concepts

RCE code is divided into three layers:

1. The core is the same for all RCEs of a given generation. It contains low-level PowerPC code, RTEMS, generic C/C++ support libraries, etc.
2. The protocol plug-in (PPI) handling code, which is recorded somewhere on the RCE, e.g., in configuration flash.
3. Application code which is entered only after the first two layers are fully initialized.

The low-level interface to an RCE's protocol plug-ins uses abstractions called ports, frames, frame buffers, channels, lanes, pipes and factories.

A port is the RCE end of a two-way communications link similar in concept to a BSD socket. Ports are globally visible but not MT- interface to an RCE's protocol plug-ins uses abstractions called ports, frames, frame buffers, channels, lanes, pipes and factories.A port is the RCE end of a two-way communications link similar in concept to a BSD socket. Ports are globally visible but not MT-safe; at a given time at most one task may be waiting for data from, receiving data from or delivering data to any given port.

Ports take and deliver data as frames. The exact content of a frame depends on the transmission protocol being used. but the the API recognizes a broad division into header and payload. One One frame corresponds to one message on the I/O medium and is delivered in a single frame buffer. In other words all ports implement datagram rather than byte-stream protocols. It's up to higher-level software such as a TCP stack to provide any operations that cross frame boundaries. Each port contains a queue of frames which have arrived and have not yet been consumed by the application.

A channel represents a hardware I/O engine capable of DMA to and from system RAM, i.e., a protocol plug-in. An RCE has at most eight channels. Most channels will make use of one or more of the Multi-Gigabit Tranceiver Transceiver modules (lanes) available in firmware, though some may simply offer access to on-board resources such as DSPs. Each channel has its own port space where each port is identified by a 16-bit unsigned integer. Each port represents a different source of incoming data such as a UDP port or a Petacache flash lane. The size of the port number space on a channel depends on the channel type and may be as low as one.

There is a one-to-one correspondence between the pairs (channel, port no.) and port objects. If a channel receives a frame that doesn't belong to any open port of the channel, the channel object just counts it before reclaiming the frame buffer.

All the lanes (if any) of a given channel go to one the outputs (pipes) of the backplane. This mapping is fixed by hardware.

Each type of channel has both an offical official (unsigned) number and an official short name such as "eth" or "config". Channels that differ only in the number of lanes they use, e.g., 10 Gb/s ethernet (4) and a slower ethernet (1) will have the same channel type, in this case "eth".

The factory code that creates the right kind of Channel objects for a given type of channel may be already part of the system core or it may be in a container in is in configuration flash. In the latter case the It code must be loaded, relocated and bound to the system core before its first use. An entry point is called in each such loaded factory code module which will register create a channel factory object in a central table using a registry object exported by the core for this purpose. Pre-loaded code must also be registered. Factory code returns . Factory code returns values of Channel* though the actual objects pointed to are of derived classes for the specific channel type.

The information needed to find the factory code modules and create the the channel objects is found in configuration flash.

Interface

Both PPI hardware and the channel factory modules both have version numbers which will allow some measure of compatibility checking. Any Incompatibility detected will will cause an exception to be thrown.

Interface

The C++ class declarations given here contain only those members intended for the user.

Channel/plug-in type numbers

These are given in a header file made available to both core and application code. The numbers are members of an enumeration.

namespace RCE::chantype {

    enum {
        ETHERNET,
        CONFIG_FLASH,
        ...
    } Number;
}

Channel

list

This class is a Borg-type singleton; the default constructor gets you a reference to the one instance. You can get a count of the number of channels or the first channel on the list (the list can't be empty). The dump() member function will print informational messages in the system log which show the contents of the channel list.
A particular channel may be looked up in several different ways:

• By its global ID number, assigned in sequence starting from zero as channels are created.
• By its type number and the creation sequence number within the type, e.g., (ETHERNET,0), (ETHERNET,1), etc.
• By the number of the pipe the channel is connected to.

Lookup methods return a null pointer if the search fails.

class ChannelList {

public:

    ChannelList();

    int numChannels();

    Channel* head();

    Channel* lookup(unsigned type, unsigned instance);

    Channel* lookup(unsigned id);

    Channel* lookupByPipe(unsigned pipe

An instance of this class will be exported by the core code using some design pattern such as Singleton or functional equivalent. This instance is created during system startup and lasts until system shutdown. Assignment or copying of instances is forbidden.

class ChannelRegistry {

    ChannelRegistry();

    // Register a channel and assign it a sequence number i starting from zero.
    // The argument may not be a null pointer.
    void addChannel(Channel*);

    intvoid numChannelsdump() const;

    // If i is a valid sequence number then return the i'th channel
    // else return a null pointer.
    Channel* factory(int i);

);

Channel

A channel object manages a set of I/O buffers and a particular instance of a protocol plug-in. At the beginning of each buffer the channel will build a FrameBuffer object. Each channel object is created at system startup, initially in a disabled state. Afterward the system startup code will allocate the buffers, feed them to the channel objects and then enable them. Channel objects live until system shutdown.

);

Channel

A channel object represents a particular instance of a protocol plug-in. Each channel object is created at system startup. Channel objects live until system shutdown and may not be copied or assigned.

Each channel creates and destroys Port objects on demand. Each port is assigned one of the channel's legal port numbers not used by any other port. The client code may request a port number for the type of channel, e.g., a well-known TCP port number. The client may Each channel creates and destroys Port objects on demand. Each port is assigned one of the channel's legal port numbers not used by any other port. The client code may request a port number for the type of channel, e.g., a well-known TCP port number. The client may also allow the channel to assign an ID not currently in use by any port.

A channel knows how to derive a port number from the header of a frame it has received.

Channel and its derived classes do not allow copying or assignment of their instances.

Every channel object is a member of the linked list accessed though class ChannelList and may not be removed from the list. Use the next() member function to iterate over the list.

A short name for the type and a short description of the channel are also provided.

If a frame comes in which doesn't belong to any of the channel's ports (an orphan frame) then a count is incremented for the channel and the frame buffer is reclaimed.

Other information provided:

• The ID of the pipe associated with the channel.
• The hardware (firmware) version number.
• The software version of the channel factory.

The dump() member function produces detailed description about the channel in the system log.

class Channel {
public:

    ChannelFactoryPort* factoryallocate(int portNum) const;

 // The factory that created me.

virtual Port* allocate() = 0;

    virtual unsignedvoid lanesUseddeallocate(Port *) = const0;

    unsigned pibsUsedorphans() const;

    unsigned pebsUsedid() const;

    unsigned ecbsUsedtype() const;

    unsigned flbsUsedconst char* name() const;

    unsigned pipeUsedinstance() const;

    const char* descriptionunsigned pipe() const;

    unsigned orphanFrameCountversionHard() const;

    // Create a Port with a specified port number.
    // Returns null if the number is already taken.unsigned versionSoft() const;

    Channel* next() const;

    virtualconst Portchar* createPortdescription(int portNum) = 0const;

    // Create a Port with a number assigned by the channel.
    // Returns null if no port number is free.
    virtual Port* createPort() = 0;

    virtual void destroyPort(Port *) = 0;

    // Return null or the Port with the given number.
    Port *getPort(unsigned portNum) const;

    // Start managing a new buffer or reclaim an old one.
    void claimBuffer(void *)void dump() const;

};

Port

Each port is created by a channel and is assigned a unique ID in the channel's port number space.

A port queues frames directed to it by the channel that created it. These frames may be waited for and retrieved using the wait() member function. Client code will normally keep a frame for a short time then give it back to the port they got it from using the port's giveBack() member function. It's an error to try to give back a frame to a port which didn't produce it; doing so will produce undefined behavior.

A port takes frames given to its send() member function and queues them for output. The frame buffer is NOT reclaimed after sending.

class Port {
public:

    unsigned id() const;

    virtual void* enablewait();

    void disablegiveBack(void *frame);

    void isEnabledsend(void *frame) const;

protected:

    Port(Channel*, unsigned id);
};

Implementation

General

Bit numbering: Bit zero is the least significant bit of a value (contrary to the convention in PowerPC documents).

Channel

The protected constructor takes a PPI definition object plus other information supplied by the factory code and the system startup code.

From the factory code:

• name
• instance
• versionSoft
• description

From the startup code:

• PPI definition
• id
• pipe

class Channel {
protected:

    Channel(
        struct PpiDef*,// Return null or a pointer to the Port that should
    // receive the data.
    virtual Port* deducePort(void *frameHeader) const;

protected:
    // Create a Channel which is initially in the disabled state.
    Channel(
        unsigned channelTableKeyid,
         RCE::config::Tables&,
	ChannelFactory *,
const char* name,
        unsigned instance,
     );

  virtual ~Channel();

};

Port

Remembers the Channel that created it.

Allows client code to wait for new frames.

Reclaims frame buffers that the client code has finished using (or has just
created).

 unsigned pipe,
	unsigned versionSoft,
        const char* description);

    virtual ~Channel();
};

Channel factory

The abstract base class for objects that manufacture Channel instances. Channel factories are created during system startup and last until system shutdown. Assignment or copying of instances is forbidden.

Derived classes contain or find the type name, software version number, recommended number of buffers and the description string, adding these to each class object created.

class PortChannelFactory {
public:

  Port(Channel *, unsigned portNumber);
  Channel* channel RCE::chantype::Number type() const;

    const char* name() const;

    unsigned portNumberversionSoft() const;

   virtual void*unsigned waitnumBuffersAdvised() const;

 //  Returns aconst pointer to the frame header when one becomes available.
  void claimBuffer(void *frameHeader) const;
};

Implementation

General

Bit numbering: Bit zero is the least significant bit of a value (contrary to the convention in PowerPC documents).

Channel factory

The abstract base class for objects that manufacture Channel instances. Channel factories are created during system startup and last until system shutdown. Assignment or copying of instances is forbidden.

Derived classes add the type name, the recommended number of buffers and the description string to the information gotten from the configuration tables.

class ChannelFactory {
public:

    RCE::chantype::Number typeNumber() constchar* description() const;

    virtual Channel* createChannel(
            struct PpiDef*,
	    unsigned id,
            unsigned pipe);
    ) = 0;

protected:

    ChannelFactory(
        unsigned type,
	const char* name,
	unsigned versionSoft,
	unsigned numBuffersAdvised,
	const char *description
    );

    const char* typeNamevirtual ~ChannelFactory() const;

    unsigned firmwareVersion() const;

    unsigned headerSize() const;

    unsigned maxPayloadSize() const;

    unsigned numBuffersAdvised() const;

    unsigned codeContainerName() const;

    const char* description() const;

    virtual Channel* createChannel(
        unsigned channelTableKey,
        RCE::config::Tables&
    ) = 0;

protected:

    ChannelFactory(
        unsigned factoryTableKey,
        const RCE::config};

Configuration table access

Implementation classes for Gen I and Gen II are derived from this base. PPI definition numbers range from zero to seven. Lane numbers range from zero to 31. Use of a number outside the valid range will cause an exception. An unused PPI definition will have a type number equal to EMPTY. An unused pipe table entry will be equal to EMPTY.

class Configuration {

public:

    Configuration();

    static unsigned EMPTY = 0xffffffff;

    void ppiDef(unsigned defNum, PpiDef&::Tables&,
	const char* typeName,
	unsigned numBuffersAdvised,
	const char *description
    );

    virtualunsigned ~ChannelFactorypipe(unsigned lane);
};

Gen I

Frames are still divided into header and payload sections. Each channel manages its own buffer pool.

Location and format of the configuration tables

A data container in the configuration flash contains tables of information about the hardware and firmware; this information can't be gotten directly from the Virtex-4 hardware and firmware.

The tables are called PpiDefs and Pipes. Each table is an array of plain-old-data structs. To make alignment easier we use 32-bit fields wherever possible, even for 16-bit quantities such as port numbers. All of the declarations for the configuration structs are in namespace RCE::config.

The PpiDefs table will come first, at the beginning of the container, followed by the Pipes table.

PpiDefs table

The PpiDefs table will have eight elements, one for each possible PPI in the system. An element is considered unused if the type member contains 0xffffffff (EMPTY) in which case all the other members of that element are zero.

struct PpiDef {
    unit32_t  type;
    unit32_t  lanes;
    uint32_t  version;
    uint32_t  maxPorts;
    uint32_t  pibs;
    uint32_t  pebs;
    uint32_t  ecbs;
    uint32_t  flbs;
    uint32_t  payloadSize;
    uint32_t  headerSize;
};

Member descriptions (when type != EMPTY):

• type. The type number of the resulting Channel object as given in the header ChannelTypes.hh.
• lanes. Bit N is set to indicate that lane N is used by this PPI.
• version. The version number for the PPI firmware.
• maxPorts. The size of the port number space.
• pibs, pebs, ecbs, and flbs. Bit N is set to indicate that the PPI uses the N'th PIB, PEB, ECB or FLB, respectively.
• payloadSize. The maximum number size in bytes of a frame payload.
• headerSize. The size of a frame header.

Pipes table

In order to discover which pipe PPI uses will use you take the lowest lane number used by the PpiDef entry and use it to index the Pipes table, which has 32 elements. A pipe number of 0xffffffff indicates an unused element, otherwise the ID of the pipe is given. A pipe ID will probably contain several subfields telling what kind of target the pipe connects to, etc.

struct Pipe {
    uint32_t  pipeId;
};

Class FrameBuffer

A FrameBuffer instance is placed at the beginning of a buffer that will also hold the frame proper. The instances are created at system startup and live until system shutdown.

A FrameBuffer contains regions of fixed size used in the management of the frame itself and its buffer:

• One or more links used to make singly-linked lists.
• The firmware's in-memory descriptor. The firmware TDE for the frame points to the descriptor.
• Status. The firmware writes operation completion status here.

Immediately following the end of the FrameBuffer buffer in we have first the frame's header and then its payload.

The links, descriptor and status areas have the same sizes for all channel types.

The descriptor must begin on a 64-byte boundary; for ease of layout the entire buffer also has that alignment. The buffer should also begin on a cache line boundary in order to reduce cache thrashing but since a cache line is only 32 bytes long this requirement is already met.

FrameBuffer instances may not be copied or assigned.

class FrameBuffer {
public:
    FrameBuffer(void *bufferAddress);
    FrameBuffer* link(unsigned linkNum) const;
    const Descriptor& descriptor() const;
    const OpStatus& status() const;
    char* header() const;
    char* payload() const;
private:
    FrameBuffer* link[N];
    Descriptor descr;
    OpStatus status;
    // The frame header starts here.
};

Use case: System startup

For Gen I we need to store the configuration tables and factory code modules in configuration flash. We'll use the core's boot-level flash-reading code to read them during this startup process. That way the configuration flash's channel object factory and channel object will be constructed in the usual way, so there must be a corresponding factory code module in flash. Once startup is complete the boot-level flash code will no longer be used.

1. Boot code:
   1. Loads and starts the system core.
2. System core
   1. Initializes the CPU.
   2. Initializes RTEMS.
   3. Initializes any extra C++ support.
   4. Sets up the MMU's TLB and enables the MMU.
   5. Creates the default instance of the dynamic linker.
   6. Calls the ChannelList constructorchannel list creator.
   7. Loads and links the application code using the default dynamic linker.
   8. Initializes the network.
      1. Initializes each ethernet channel.
      2. Initializes IP, UDP, TCP and BSD sockets.
      3. Gets a DHCP lease if required.
   9. Calls the application entry point.

ChannelList constructorChannel list creator:

1. Reads each possible PPI defn. and for each non-empty one:
   1. Reads from flash and links the factory code module based on PPI type (if not already done).
   2. Calls the entry point in the factory module that creates channel factories (if not already done).
   3. Uses the resulting factory to create a channel object.
   4. Adds the resulting channel to the list.
   5. Saves the recommended number of buffers for the channel.
2. Decides how many buffers of what size to allocate for each channel.
3. For each Channel on the list:
   1. Gives the assigned buffers to the channel and enables it.

Use case: Frame import

TBD.

Use case: Frame export

TBD.

Gen II

Location of the configuration tables

Probably somewhere in DCR space so we'll have to use mfdcr instructions to read them. The exact organization is TBD.

No PIC blocks

The PpiDefs and Pipes tables will be obtained directly from the hardware via DCR registers.

Protocol plug-ins.

Protocol plug-ins Protocol plugins will no longer be implemented using PIC blocks.

Frames

The old division of frames into header and payload will no longer be made. All input frame buffers for all channels will be held in a single free-list; when a buffer is needed the one best matching the required size is allocated.