Draft 3.
...
2
Table of Content Zone | ||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||||||||||
ConceptsRCE code is divided into three layers:
Each layer's code is stored independently somewhere on the RCE, e.g., in configuration flash. The low-level interface to an RCE's protocol plug-ins uses abstractions called ports, frames, frame buffers, channels, lanes, pipes and factories. A port is the RCE end of a two-way communications link similar in concept to a BSD socket. Ports are globally visible but not MT-safe; at a given time at most one task may be waiting for data from, receiving data from or delivering data to any given port. Ports receive and transfer data packaged in frames. The exact content of a frame depends on the protocol being used. One frame corresponds to one message on the I/O medium and is delivered in a single frame buffer. In other words all ports implement datagram rather than byte-stream protocols. It's up to higher-level software such as a TCP stack to provide any operations that cross frame boundaries. Each port contains two queues of frames; one for frames which have arrived and have not yet been consumed by the application, the other for frames generated by the application but which have not yet been transmitted. In addition to a frame a frame buffer contains administrative information that is private to the implementation of the plug-in interface; the user gives and takes only A channel represents a hardware I/O engine capable of DMA to and from system RAM, i.e., a protocol plug-in of a particular type. An RCE has at most eight channels (which limit derives from the limit on the number of plugins). Most channels will make use of one or more of the Multi-Gigabit Transceiver modules (lanes) available in firmware, though some may simply offer access to on-board resources such as DSPs. Each channel has its own port space where each port is identified by a 16-bit unsigned integer. Each port represents a different source and/or sink of data such as a UDP port or a section of Petacache memory. The size of the port number space on a channel depends on the channel type and may be as low as one. Every port object created is allocated a port number from its channel's space but no two ports of a channel may have the same port number. Each incoming frame must contain information from which a port number can be derived. If it doesn't, or if it specifies a port number that is invalid or not allocated to a port object, then the channel's lost-frame count is incremented and the frame is recycled. All the lanes (if any) of a given channel map to one pipe, which represents some on-board or off-board target. Each type of channel has both an official (unsigned) number and an official short name such as "eth" or "config". Channels that differ only in the number of lanes they use, e.g., 10 Gb/s ethernet (4) and a slower ethernet (1) have the same channel type, in this case "eth". PPI-handling code is held in relocatable modules recorded on the RCE. Each module contains an entry point that creates a factory object for a particular type of channel. Each module is loaded, relocated and bound to the system core before its first use. A factory object returns values of type Channel* which refer to instances of a derived class specific to the channel type. Both PPI hardware and the channel factory modules both have version numbers which will allow some measure of compatibility checking. Any incompatibility detected will will cause a fatal exception to be thrown during system startup. InterfaceThe C++ class declarations given here contain only those members intended for the user. Whether a method is virtual is not specified, nor are friend declarations shown; these are considered implementation details. NamespacesAll the public declarations for the interface are in namespace
Header filesAll the header files for the public interface are the top level of package
Note that in the repository both h2 Universal constants ( All RCEs whether of Gen I or Gen II each have the same limits on the number of plugin instances ( Channel type enumeration ( |
Code Block | ||||
---|---|---|---|---|
| ||||
enum { CONFIG_FLASH, ETHERNET, CONFIGPETACACHE_FLASH, etc. } REMOTE_REGISTER, etc., INVALID_CHANNELTYPE } ChannelType; |
Channel list
This class is a Borg-type singleton; the constructor makes a stateless object whose member functions access the true (shared) state defined elsewhere. The destructor destroys these stateless objects but does not touch the true state information. You can therefore just use the constructor whenever you need to access the One True List, e.g., ChannelList().head()
.
You can get a count of the number of channels or the first channel on the list (the list can't be empty). The dump()
member function will print informational messages in the system log which show the contents of the channel list.
Note |
---|
The location and form of the system log depends on how the system logging package was initialized at application startup. Client code making log entries is not aware of this initialization. |
A particular channel may be looked up in several different ways:
- By its global ID number, assigned in sequence starting from zero as channels are created.
- By its type number and the creation sequence number within the type, e.g., (ETHERNET,0), (ETHERNET,1), etc.
- By the number of the pipe the channel is connected to.
Lookup methods return a null pointer if the search fails.
class ChannelList {
public:
ChannelList() {}
~ChannelList() {}
int numChannels() const;
Channel* head() const;
Channel* lookup(unsigned type, unsigned instance) const;
Channel* lookup(unsigned id) const;
Channel* lookupByPipe(unsigned pipe) const;
void dump() const;
);
The header file also contains a specialization of the template RCE::service::EnumInfo
which allows one to use the function templates emin<>()
, emax<>()
, ecount<>()
, evalid<>()
, enext<>()
, eprev<>()
and estr<>()
:
Code Block | ||||
---|---|---|---|---|
| ||||
emin<ChannelType>() == CONFIG_FLASH
emax<ChannelType>() == ChannelType(INVALID_CHANNELTYPE - 1)
ecount<ChannelType>() == int(INVALID_CHANNELTYPE)
evalid(x) is true for all from emin() to emax() inclusive, else false
enext(emax()) == eprev(emin()) == INVALID_CHANNELTYPE
estr(CONFIG_FLASH) == "CONFIG_FLASH", etc.
estr(x) == "**INVALID**" if and only if evalid(x) is false
|
ecount<>()
can't be used as a dimesion for static arrays since the compiler considers it to be non-constant; in that case use EnumInfo<ChannelType>::count
.
Channel list
This class is a Borg-type singleton; the constructor makes a stateless object whose member functions access the true (shared) state defined elsewhere. The destructor destroys these stateless objects but does not touch the true state information. You can therefore just use the constructor whenever you need to access the One True List, e.g., ChannelList().head()
.
You can get a count of the number of channels or the first channel on the list (the list can't be empty). The report()
member function will print informational messages in the system log which show the contents of the channel list in brief form, one line per channel.
Note |
---|
The location and form of the system log depends on how the system logging package was initialized at application startup. Client code making log entries is not aware of this initialization. |
A particular channel may be looked up in several different ways:
- By its global ID number, assigned in sequence starting from zero as channels are created.
- By its type number and the creation sequence number within the type, e.g.,
(ETHERNET,0)
,(ETHERNET,1)
, etc. - By the number of the pipe the channel is connected to.
Lookup methods return the null pointer if the search fails.
Code Block | ||||
---|---|---|---|---|
| ||||
class ChannelList {
public:
ChannelList() {}
~ChannelList() {}
int numChannels |
Channel
A channel object represents a particular instance of a protocol plug-in. Each channel object is created at system startup. Channel objects live until system shutdown and may not be copied or assigned.
Each channel creates and destroys Port objects on demand. Each port is assigned one of the channel's legal port numbers not used by any other port. The client code may request a port number for the type of channel, e.g., a well-known TCP port number. The client may also allow the channel to assign a number not currently in use by any port.
Every channel object is a member of the linked list accessed though class ChannelList and may not be removed from the list. Use the next()
member function to iterate over the list.
A short name for the type and a short description of the channel are also provided.
A "lost" counter is provided which counts the number of incoming frames that were discarded, for whatever reason, instead of being queued in a port.
Other information provided:
- The ID of the pipe associated with the channel.
- The hardware (firmware) version number.
- The software version of the channel factory.
The dump()
member function produces detailed description of the channel in the system log.
Code Block | ||
---|---|---|
none | none | class Channel { public: Port* allocate(int portNum); Port* allocate(); void deallocate(Port *); unsigned lost() const; unsigned id() const; unsignedChannel* typehead() const; const char* name() const; Channel* lookup(unsigned type, unsigned instance() const; unsignedChannel* pipe() const; lookup(unsigned versionHard(id) const; unsigned versionSoft() const; Channel* nextlookupByPipe() const; const char* description(unsigned pipe) const; void dumpreport() const; }); |
Port
Channel
A channel object represents a particular instance of a protocol plug-in. Each channel object is created at system startup. Channel objects live until system shutdown and may not be copied or assigned.
Each channel creates and destroys Port objects on demand. Each port is assigned one of Each port is created by a channel and is assigned a unique ID in the channel's port number space.
A port queues frames directed to it by the channel that created it. These frames may be waited for and retrieved using the wait()
member function. Client code will normally keep a frame for a short time then give it back to the port they got it from using the port's giveBack()
member function. It's an error to try to give back a frame to a port which didn't produce it; doing so will produce undefined behavior.
A port takes frames given to its send()
member function and queues them for output. The frame buffer is NOT reclaimed after sending.
legal port numbers not used by any other port. The client code may request a port number for the type of channel, e.g., a well-known TCP port number. The client may also allow the channel to assign a number not currently in use by any port.
Every channel object is a member of the linked list accessed though class ChannelList and may not be removed from the list. Use the next()
member function to iterate over the list.
A short name for the type and a short description of the channel are also provided.
A "lost" counter is provided which counts the number of incoming frames that were discarded, for whatever reason, instead of being queued in a port.
Other information provided:
- The ID of the pipe associated with the channel.
- The hardware (firmware) version number.
- The software version of the channel factory.
The report()
member function produces detailed multi-line description of the channel in the system log, including all platform-specific information.
Code Block | ||||
---|---|---|---|---|
| ||||
class Channel | ||||
Code Block | ||||
none | none | class Port { public: unsignedPort* idallocate(int portNum) const; voidPort* waitallocate(); void giveBackdeallocate(voidPort *frame); voidunsigned send(void *frame); }; |
Implementation
General
Bit numbering: Bit zero is the least significant bit of a value (contrary to the convention in PowerPC documents).
Namespaces
The namespace RCE::ppi
contains implementation declarations but these don't appear in public header files. Declarations common to Gen I and II appear at the top level. Generation specific code is inside one of the generation specific nested namesapces.
Code Block | ||
---|---|---|
none | none | namespace RCE { namespace ppi { lost() const; unsigned id() const; unsigned type() const; const char* name() const; unsigned typeSequence() const; unsigned class Factorypipe() const; unsigned class ConfigurationversionHard() const; unsigned class PluginversionSoft() const; Channel* class Pipe; namespace gen1 { next() const; const char* description() const; void class Descriptor; class Buffer; class OpStatus; etc. } namespace gen2 { etc. } etc. } } |
Header files
The header files with implementation declarations is private to the ppi
project. The directory hierarchy mirrors that of the namespaces.
report() const;
};
|
Port
Each port is created by a channel and is assigned a unique ID in the channel's port number space.
A port queues frames directed to it by the channel that created it. These frames may be waited for and retrieved using the wait()
member function. Client code will normally keep a frame for a short time then give it back to the port they got it from using the port's giveBack()
member function. It's an error to try to give back a frame to a port which didn't produce it; doing so will produce undefined behavior.
A port takes frames given to its send()
member function and queues them for output. The frame buffer is NOT reclaimed after sending.
Code Block | ||||
---|---|---|---|---|
| ||||
release/ class rce/Port { public: unsigned ppi/id() const; void* wait(); void src/giveBack(void *frame); void send(void *frame); }; |
Implementation
General
Bit numbering: Bit zero is the least significant bit of a value (contrary to the convention in PowerPC documents).
Namespaces
The namespace RCE::ppi
also contains implementation declarations but these don't appear in public header files but in package private headers (see below). Some names are declared in the same way for both Gen I and Gen II, others look different depending on the platform and some may not appear at all on a given platform.
Header files
All implementation code (including inline function definitions) appears somwhere under the directory rce/ppi/src/
. Generic code appears directly in src/
while platform specific code will appear in subdirectories gen1/
, /gen2
, and possibly i86-linux/
. Unit test code is in rce/ppi/test/
which has the same substructure as src/
. Some implementation is partly generic and partly platform specific so a filename may appear in several places.
Code Block | ||||
---|---|---|---|---|
| ||||
release/
rce/
ppi/
src/
Channel-inl.hh
Channel.cc
ChannelList-inl.hh
ChannelList.cc
Configuration.hh
Factory.hh
Port-inl.hh
Port.cc
gen1/
ChannelList.cc
Configuration.cc
ConfigurationData.hh
Factory.cc
Pipe.hh
Plugin.hh
etc.
gen2/
etc.
etc.
test/
testPpi.cc
gen1/
testPpi.cc
etc.
|
Channel list
We export a static list-building member function void build()
whose implementation depends on the platform.. All the instances of ChannelList
created by the constructor are empty; the non-static member functions all use the static head-of-list pointer and never use this
.
Code Block | ||||
---|---|---|---|---|
| ||||
class ChannelList {
public:
static void build();
private:
static Channel* m_head;
static int m_numChannels;
};
|
Channel
The protected constructor takes a PPI definition object plus other information supplied by the factory code and the system startup code. There are also protected member functions to increment the "lost" count and to set the "next" pointer (even though ChannelList
is a friend we prefer to use an accessor for the "next" pointer).
From the factory code:
- name
- instance
- versionSoft
- description
From the startup code:
- An instance of Plugin
- versionHard
- id
- pipe
Code Block | ||||
---|---|---|---|---|
| ||||
class Plugin; class Channel { protected: friend class ChannelList; Channel( Factory.hh Configuration.hh PpiDef.hh PipeDef.hh gen1/ Descriptor.hh Buffer.hh OpStatus.hh etc. Plugin*, unsigned id, gen2/ ChannelType type, const char* name, etc. unsigned typeSequence, etc. |
Channel list
We export a static list-building member function. All the instances created by the default constructor are empty; the non-static member functions all use the static head-of-list pointer and never use this
.
Code Block | ||
---|---|---|
none | none | class ChannelList { public: unsigned pipe, unsigned versionHard, unsigned versionSoft, static void build( const char* description); private: virtual static Channel* head_~Channel() = 0; void ChannelListincLost(const ChannelList&); ChannelList& operator=(const ChannelList&); }; |
Channel
The protected constructor takes a PPI definition object plus other information supplied by the factory code and the system startup code.
From the factory code:
- name
- instance
- versionSoft
- description
From the startup code:
- An instance of Plugin.
- id
- pipe
class Plugin;
class Channel {
protected:
Channel(
Plugin*,
unsigned id,
const char* name,
unsigned instance,
unsigned pipe,
unsigned versionSoft,
const char* description);
virtual ~Channel();
private:
Channel(const Channel &);
Channel& operator=(const Channel &);
};
void next(Channel*);
private:
Channel(const Channel &);
Channel& operator=(const Channel &);
private:
Plugin* m_plugin;
unsigned m_lost;
Channel* m_next;
unsigned m_id;
ChannelType m_type;
const char* m_name;
unsigned m_typeSequence;
unsigned m_pipe;
unsigned m_versionHard;
unsigned m_versionSoft;
const char* m_description;
};
|
The name and description string pointers must refer to non-automatic storage since the object must remain valid until shutdown.
Port
The first private constructor stores the given Channel* and port number.
Code Block | ||||
---|---|---|---|---|
| ||||
class Port { private: friend class Channel; Port(Channel*, unsigned portNum); ~Port(); Port(const Port&); Port(const Port&)& operator=(const Port&); private: Channel* m_channel; Port& operator=(const Port&)unsigned m_id; }; |
Channel factory
The abstract base class for objects that manufacture Channel instances. Channel factories are created during system startup and last until system shutdown. Assignment or copying of instances is forbidden.
Derived classes contain or find the type name, software version number, recommended number of buffers per channel and the description string, adding these to each class object created.
Code Block | ||||
---|---|---|---|---|
| ||||
class Factory { public: ChannelType type() const; const char* name() const; unsigned versionSoft() const; unsigned numBuffersAdvised() const; const char* description() const; virtual Channel* createChannel( class Plugin*, unsigned id, unsigned pipe); ) = 0; protected: Factory( unsigned type, const char* name, unsigned versionSoft, unsigned numBuffersAdvised, const char *description ); virtual ~Factory(); }; |
Configuration table access
Implementation classes The same declaration is used for Gen I and Gen II are derived from this base. PPI definition numbers range from zero to seven. Lane numbers range from zero to 31. Use of a number outside the valid range causes an exception. An unused PPI definition has a type number equal to EMPTY. An unused pipe table entry has a pipe ID equal to EMPTYsince it uses only references to classes Pipe
and Plugin
. The implementations are quite different, however.
Code Block | ||||
---|---|---|---|---|
| ||||
static unsigned EMPTY = 0xffffffff; class Configuration { public: Configuration(); virtual ~Configuration(); static unsigned EMPTY = 0xffffffff; virtual void plugin(unsigned defNum, Plugin&) = 0; virtual void pipe(unsigned lane, Pipe&) = 0; }; |
Gen I
Frames are still divided into header and payload sections. Each channel manages its own buffer pool.
Location and format of the configuration tables
A data Configuration container zero in the configuration flash contains tables of information about the hardware and firmware; this information can't be gotten directly from the hardware and firmware.
To make alignment easier we use 32-bit fields wherever possible, even for 16-bit quantities such as port numbers.
The plugin table will come first, at the beginning of the container, immediately followed by the pipe table.
Plugin table
The plugin table will have eight array elements, one for each possible PPI in the system. An element is considered unused if the type
field contains 0xffffffff
(EMPTY
) in which case all the other members of that element are zero.
Field name | Type | Description |
---|---|---|
type | unit32_t | The type number of the resulting Channel object as given in the header ChannelTypes.hh |
lanes | unit32_t | Lane usage bitmask; bit N is set to indicate that lane N is used by this PPI |
version | unit32_t | The version number for the PPI firmware |
maxPorts | unit32_t | The size of the port number space |
pibs | unit32_t | PIB usage bitmask |
pebs | unit32_t | PEB usage bitmask |
ecbs | unit32_t | ECB usage bitmask |
flbs | unit32_t | FLB usage bitmask |
payloadSize | unit32_t | The maximum size in bytes of a frame payload |
headerSize | unit32_t | The size of a frame header |
Pipe table
In order to discover which pipe a PPI uses will you take the lowest lane number used by the PPI and use it to index the pipe table, which has 32 array elements. A pipe ID of 0xffffffff
indicates an unused element, otherwise the ID of the pipe is given. A pipe ID will probably contain several subfields telling what kind of target the pipe connects to, etc.
Field name | Type | Description |
---|---|---|
pipeId | unit32_t | Pipe ID number |
Class Buffer
A Buffer instance is placed at the beginning of a buffer that will also hold the frame proper. The instances are created at system startup and live until system shutdown.
A Buffer contains regions of fixed size used in the management of the frame itself and its buffer:
- One or more links used to make singly-linked lists.
- The firmware's in-memory descriptor. The firmware TDE for the frame points to the descriptor.
- Status. The firmware writes operation completion status here.
Immediately following the end of the Buffer we have first the frame's header and then its payload.
The links, descriptor and status areas have the same sizes for all channel types.
The descriptor must begin on a 64-byte boundary; for ease of layout the entire buffer also has that alignment. The buffer should also begin on a cache line boundary in order to reduce cache thrashing but since a cache line is only 32 bytes long this requirement is already met.
Buffer instances may not be copied or assigned.
Code Block | ||||
---|---|---|---|---|
| ||||
class Buffer { public: Buffer(void *bufferAddress); Buffer* link(unsigned linkNum) const; const Descriptor& descriptor() const; const OpStatus& status() const; char* header() const; char* payload() const; private: Buffer* link[N]; Descriptor descr; OpStatus status; // The frame header starts here. }; |
Use case: System startup
For Gen I we need to store the configuration tables and factory code modules in configuration flash. We'll use the core's boot-level flash-reading code to read them during this startup process. That way the configuration flash's channel object factory and channel object will be constructed in the usual way, so there must be a corresponding factory code module in flash. Once startup is complete the boot-level flash code will no longer be used.
- Boot code:
- Loads and starts the system core.
- System core
- Initializes the CPU.
- Initializes RTEMS.
- Initializes any extra C++ support.
- Sets up the MMU's TLB and enables the MMU.
- Creates the default instance of the dynamic linker.
- Calls ChannelList::build().
- Loads and links the application code using the default dynamic linker.
- Initializes the network.
- Initializes each ethernet channel.
- Initializes IP, UDP, TCP and BSD sockets.
- Gets a DHCP lease if required.
- Calls the application entry point.
ChannelList::build():
- Reads each possible PPI defn. and for each non-empty one:
- Reads from flash and links the factory code module based on PPI type (if not already done).
- Calls the entry point in the factory module that creates channel factories (if not already done).
- Uses the resulting factory to create a channel object.
- Adds the resulting channel to the list.
- Saves the recommended number of buffers for the channel.
- Decides how many buffers of what size to allocate for each channel.
- For each Channel on the list:
- Gives the assigned buffers to the channel and enables it.
Use case: Frame import
TBD.
Use case: Frame export
TBD.
Gen II
Location of the configuration tables
The PpiDefs and PipeDefs table entries will be obtained directly from the hardware via DCR registers.
Protocol plug-ins.
Protocol plug-ins will no longer be implemented using PIC blocks.
Frames
The old division of frames into header and payload will no longer be made. All input frame buffers for all channels will be held in a single free-list maintained by the firmware; when asked for a buffer of a certain size it allocates the one whose size is the best match.