typedef uint32_t StringOff; // Offset within the Strings table.

struct Channel {
    uint32_t  key;
    unit32_t  typeKeyofficialType;   // The official channel  // Ref. to Channel Types tabletype number.
    unit32_t  lanesUsed;      // Bit-mask of lanes used.
    uint32_t  blocksUsedpibsUsed;       // Bit-mask of Pending PICImport blocksBlocks used.
    uint32_t  numPortspebsUsed;       // SizeBit-mask of port-number spacePending Export Blocks used.
    StringOff description;
};

struct ChannelType {
    uint32_t  ecbsUsed;       // Bit-mask of Event Completion Blocks used.
    uint32_t  key;flbsUsed;       // Bit-mask of Free List Blocks used.
    uint32_t  officialNumber;
numPorts;        StringOff officialName;// Size of port-number space.
    StringOff description;
};

The Data Paths table:

The Buffers table:

Here we assume that the firmware delivers all ethernet framing bytes as well as the payload and that jumbo frames are not allowed. We assume that the formware delivers configuration flash data in units of pages and that a page is 2K bytes long.

How the tables appear in RAM

We copy the tables without alteration and set the members of the following structure:

struct Tables {
    unsigned numChannels;
    Channel *channel;      // Pointer to first element of table.
    unsigned numChannelTypesnumFactories;
    ChannelTypeFactory *channelType;
    unsigned numFactories;
    Factory *factoryfactory;
    unsigned numDataPaths;
    DataPath *dataPath;
    unsigned numBuffers;
    Buffer *buffer;
};

Use case: System startup

1. Boot code
   1. Loads and starts the system core.
2. System core
   1. Initializes the CPU.
   2. Initializes RTEMS.
   3. Initializes any extra C++ support.
   4. Sets up the MMU's TLB and enables the MMU.
   5. Registers the factories for configuration flash and catch-all.
   6. Creates the configuration flash channel.
   7. Allocates uncached RAM for config. flash pages.
   8. Enables the configuration flash channel.
   9. Copies the Channels table and related info from configuration flash into RAM.
   10. Creates the default instance of the dynamic linker.
   11. Loads and links the factory code marked as loadable in the Factories table.
   12. Calls all factory code entry points so that all factories are registered.
   13. Walks the Channels table and creates all Channel objects.
   14. Walks the Buffers table and allocates uncached memory for I/O buffers.
   15. Enables all Channel objects not already enabled.
   16. Loads and links the application code using the default dynamic linker.
   17. Initializes the network.
       1. Initializes each ethernet channel.
       2. Initializes IP, UDP, TCP and BSD sockets.
       3. Gets a DHCP lease if required.
   18. Calls the application entry point.
3. Factory code entry point.
   1. Registers with the core two functions that create the right kind of Channel and Factory objects.

Use case: Frame import

Prior to this the code wishing to use the port has associated a consumer task with it. That task is blocked or idling waiting for new frames.

1. Plug-in
   1. Writes the frame via DMA.
   2. Causes a frame-arrival interrupt.
2. ISR
   1. Reads the cause-of-interrupt register and passes its value to the I/O dispatcher task.
3. I/O dispatcher task
   1. Finds the Channel objects based on cause-of-interrupt
   2. Channel object
      1. Gets the descriptor for the frame from the right firmware queue.
      2. Builds a Frame object of the right kind using the descriptor.
   3. Frame object
      1. Examines the frame data to determine the right port.
   4. Port object
      1. Enqueues the frame.
      2. Wakes up the consumer task if necessary.
4. Consumer task
   1. Consumes at least one of the enqueued frames.
   2. Yields or blocks.

Running the Channel, Frame and Port object code in a dispatcher task rather than the ISR keeps the latter simple; it doesn't have to know about system data structures, just hardware. Simplicity should translate to speed of response. The dispatcher task can be normal C++ code that runs with the MMU on, which an ISR by default doesn't.

Sub-case: Full cooperative multitasking

We assign the same priority to the I/O dispatcher task and all the consumer tasks. All of these tasks remain in the ready queue; each puts itself at the back of ready queue (yields) when it finds that its input queue is empty or after it has completed a certain amount of work. The consumer tasks and the I/O dispatcher task don't need to synchronize their communication because no one of them can preempt another.

The communication between the ISR and the I/O dispatcher task needs some synchronization since the ISR can preempt the dispatcher task at any time. We need not resort to semaphores or other locks, though; there are relatively simple lock-free algorithms we can use. The dispatcher task, once it comes to the front of the ready queue, loops until it manages to read its input queue without detecting interference from the ISR, then either yields immediately if the queue was empty or after performing one or more dispatches. As an alternative the dispatcher can disable interrupts for the short time it takes to check its queue.

Sub-case: Preemptive multitasking

If the I/O dispatcher is given a higher priority than consumer threads then it must synchronize its communication with them. It also won't just be able to yield when it has nothing to do since it will go onto what amounts to a different ready queue from the consumer tasks, one that is examined first. The dispatcher task would always run, starving the consumers. The dispatcher task would have to actually block itself and the ISR would have to unblock it.

If the consumer tasks don't all have the same priority then they too will have to block themselves and be unblocked by the dispatcher task.

If we use time-slicing then we don't have to be so careful in deciding when each task should yield but then all inter-task communication will require synchronization. If priority assignment is non-uniform then we need to use explicit blocking and unblocking instead of yielding.

Use case: Frame export

Low-level API

Official RCE channel type numbers and names

These are given in a header file made available to both core and application code. The numbers are members of an enumeration and the names are given in a static array.

namespace RCE::channel {

    enum {
        ETHERNET,
        CONFIG_FLASH,
        CATCH_ALL,
        ...
    } Numbers;

    const char *names = {
        "eth",
        "config",
        "catch-all"
        ...
    };
}

TBD.

Use case: Frame export

TBD.

Low-level API

Official RCE channel type numbers and names

These are given in a header file made available to both core and application code. The numbers are members of an enumeration and the names are given in a static array.

namespace RCE::chantype {

    enum {
        ETHERNET,
        CONFIG_FLASH,
        CATCH_ALL,
        ...
    } Numbers;

    const char *name = {
        "eth",
        "config",
        "catch-all"
        ...
    };

    const char *description = {
        "ethernet",
        "configuration flash",
        "catch-all channel",
        ...
    };
}

Channel factory

The abstract base class for objects that manufacture Channel instances. Channel factories are created during system startup and last until system shutdown. Assignment or copying of instances is forbidden.

class ChannelFactory {
public:
    ChannelFactory();
    virtual Channel* create(
        unsigned ident,              // Assigned by init. code.
        unsigned channelsTableKey,
        RCE::config::Tables&) = 0;
    virtual ~ChannelFactory();
};

Channel type registry

An instance of this class will be exported by the core code using some design pattern such as Singleton or functional equivalent. This instance is created during system startup and lasts until system shutdown. Assignment or copying of instances is forbidden.

class ChanneTypeRegistry {

    ChannelTypeRegistry();

    // Register a Channel factory using its official RCE type number.
    void register(
        unsigned officialNumber,
        ChannelFactory &,
    );

    ChannelFactory& channelFactory(unsigned officialNumber);

    unsigned officialNumber(const char *officialName);
);

Channel

A channel object manages a set of I/O buffers. At the beginning of each buffer the channel will build a SuperFrame object. Each channel object is created at system startup, initially in a disabled state. Afterward the system startup code will allocate the buffers, feed them to the channel objects and then enable them. Channel objects live until system shutdown.

Each channel creates and destroys Port objects on demand. Each port is assigned a unique ID number. The client code may request a particular ID in the legal range for the type of channel, e.g., a well-known TCP port number. The client may also allow the channel to assign an ID not currently in use by any port.

A channel knows how to derive the ID number of the Port that will receive a frame from the frame header.

Channel and its derived classes do not allow copying or assignment of their instances.

class Channel {
public:
  // Set the system's channel ID anf the typer number. Leaves
  // the Channel in the disabled state.
  Channel(unsigned systemAssignedId, unsigned typeNum);

  unsigned id() const;

  unsigned typeNum() const;

  // Create a Port with a specified port number.
  // Returns 0 if the number is already taken.
  virtual Port* createPort(int portNum) = 0;

  // Create a Port with a number assigned by the channel.
  // Returns 0 if no port number is free.
  virtual Port* createPort() = 0;

  virtual void destroyPort(Port *) = 0

Channel and Frame factories

These are the abstract base classes.

class ChannelFactory {
public:
    ChannelFactory();
    virtual Channel* create(
        unsigned ident,   // Assigned by init. code.
        unsigned channelsTableKey,
        RCE::config::Tables&) = 0;
    virtual ~ChannelFactory();
};

Channel type registry

class ChanneTypeRegistry {

    ChannelTypeRegistry();

    // Register a Channel factory using its official RCE type number.
    void register(
        unsigned officialNumber,
        ChannelFactory &,
    );

    ChannelFactory& channelFactory(unsigned officialNumber);

    unsigned officialNumber(const char *officialName);
);

An instance of this class will be exported by the core code using some design pattern such as Singleton or functional equivalent.

Class Frame

• Instance responsibilities.

1. Ownership of a frame buffer. Once created a Frame instance must be kept alive as long as the frame buffer is in use by the client code. The destructor of the Frame subclass will return buffers to the I/O pool. For Virtex-4 each Channel instance has its own I/O pool; for Virtex-5,6 the pool is global.
2. Furnish on request the locations and sizes of the following sections of the frame buffer:
   1. Links.
   2. Descriptor.
   3. Status.
   4. Header.
   5. Payload.
3. Calculate a channel port number from the header contents.

Frame instances are created by Channel instances.

Channel

class Channel {
  // Class responsibilities
  // ----------------------
  // Maintain the following mappings:
  // 1) PIC block number to Channel (1-to-1).
  // 2) Global Channel ID to Channel (1-to-1).
  // 3) ChannelClass to Channel (1-to-many).
  //
  // Instance responsibilities
  // -------------------------
  // Create and destroy Ports. Allocate and free port numbers. Maintain
  // the following mappings:
  // 1) port number to Port (1-to-1).
  // Give out the minimum frame buffer size and the recommended number of
  // frames for the channel.
public:
  // Install a new channel of the given class and assign it the next available
  // ID number. Associated with the class are two DCR addresses, one for
  // imported frames and one for exported frames. Presumably these are the
  // DCRs used to dequeue and enqueue frame descriptors, respectively.
  // Using this constructor automatically makes the object findable
  // by any of the static member functions that perform searches.
  Channel(const ChannelClass &ctype,
          unsigned importDcr,
          unsigned exportDcr);

  // Return the ID assigned by the constructor.
  unsigned id() const;

  // Return the DCR address associated with importing frames.
  unsigned importDcr() const;

  // Return the DCR address associated with exporting frames.
  unsigned exportDcr() const;

  // Return the class of the channel.
  const ChannelClass &channelClass() const;

  // Create a new Port object for this Channel, assigning its
  // global and port nos.
  void createPort();

  // Destroy a Port. It is an error to give a predefined Port or
  // a Port not belonging to this Channel (throws std::logic_error).
  void destroyPort(Port *);

  // Return the Port associated with the port number, or return 0.
  Port *findPort(unsigned) const;

  // Examine the Frame header to see which of this Channel's Ports
  // the Frame should belong to. If such a Port is found then
  // enqueue the frame on it and return true. If no matching Port
  // can be found then return false.
  virtual bool enqueueFrame(Frame*) const;

  // The minimum Frame buffer size for this channel.
  virtual unsigned frameSize() const;

  // The recommended number of Frame buffers to allocate for this channel.
  virtual unsigned numberOfFrames() const;

  // Return 0 or the ChannelPort whichwith has the given ID, or return 0number.
  staticPort Channel* findChannelfindPort(unsigned channelIdportNum) const;

  // ReturnManage thea m'th Channel that belonging to the given class,
  // assuming that Channels are tested in ascending order by IDnew buffer.
  void addBuffer(void *);

  // numbers. The first matching Channel corresponds to m=0, etc. Return 0Reclaim a buffer that was given out by one of the Ports
  // ifcreated noby suchthis channel exists.
  static Channel* findChannel(const ChannelClass &ctype,
                              unsigned m);

void reclaimBuffer(void *);

  void enable();

  void isEnabled() const;

  // Return -1 or the number of the port to which the frame should go,
  // Returnassuming the frame Channelbelongs to withthis thechannel.
 given DCRvirtual address, or return 0.int getPortNumber(void *frameHeader) const;

  staticvirtual Channel*unsigned findChannelByDcrAddressframeHeaderSize(unsigned) const;

privateprotected:

  // Does nothing because Channels last until the next system shutdown.
  virtual ~Channel();

};


class Port {
  // Class responsibilities
  // ----------------------
  // Maintain a mapping of Port IDs to Ports (1-to-1).
  //
  // Instance responsibilities
  // -------------------------
  // Maintain a queue of Frames. Manage a consumer task, letting it
  // put itself to sleep if the queue is empty and waking it up if
  // needed when new Frames are queued.
public:

  // Return local number local to the Channel owning this Port.
  unsigned localPortNum() const;

  // Return the Channel that owns this Port.
  Channel *channel() const;

  // Put Frames on the queue maintained by this Port. Wake up
  // any consumer task if needed. The argument is the first
  // member of a list of Frames to enqueue.
  void enqueueFrames(Frame*);

  // Return a pointer to a Frame which is the first in a list
  // of Frames removed from the queue. If no Frames are available
  // then block set the consumer task ID and block until at least one
  // Frame is ready.
  // TDB: Needs a way to indicate that the Port has been
  // destroyed (a special Frame instance? Frame flags?)
  Frame *dequeueFrames();

  // Destroy this Port by calling upon the Channel that owns it.
  void destroy();

  // Return the Port with the given global ID, or return 0.
  Port *findPort(unsigned);

private:
  friend class Channel;

  // Create a port for the given Channel and port. The next available
  // global Port ID number is automatically assigned.
  Port(Channel *owner, unsigned localPortNum);

  ~Port();

};

 system shutdown.
  virtual ~Channel();

};

Class Port

Remembers the Channel that created it.

Allows client code to wait for new frames.

Reclaims frames that the client code has finished using AND that were provided by this particular port.

class Port {
public:
  Port(Channel &, unsigned portNum);
  Channel& channel() const;
  unsigned portNum() const;
  virtual void* wait() const; // Returns a pointer to the frame header.
  void reclaim(void *frameHeader) const;
};

API implementation classes

These classes are not exposed to the client but are usde internally by the API.

Class SuperFrame

A SuperFrame instance is placed at the beginning of a buffer that will also hold the frame proper immediately after the SuperFrame. The instances are created at system startup and live until system shutdown.

A SuperFrame contains regions of fixed size used in the management of the frame itself and its buffer:

• One or more links used to make singly-linked lists.
• The firmware's in-memory descriptor. The firmware TDE for the frame points to the descriptor.
• Status. The firmware writes operation completion status here.

Immediately following the end of the SuperFrame buffer in we have first the frame's header and then its payload.

The links, descriptor and status areas have the same sizes for all channel types.

The descriptor must begin on a 64-byte boundary; for ease of layout the entire buffer also has that alignment. The buffer should also begin on a cache line boundary in order to reduce cache thrashing but since a cache line is only 32 bytes long this requirement is already met.

SuperFrame is used internally by the API; no client code is ever given any of its instances.

class SuperFrame {
public:
    SuperFrame(void *bufferAddress);
    SuperFrame* link(unsigned linkNum) const;
    const Descriptor& descriptor() const;
    const OpStatus& status() const;
    char* header() const;
    char* payload() const;
private:
    SuperFrame* link[N];
    Descriptor descr;
    OpStatus status;
    // The frame header starts here.
};