ConceptsRCE code is divided into three layers: - The core is the same for all RCEs of a given generation. It contains low-level hardware management code, RTEMS, generic C/C++ support libraries, etc.
- The protocol plug-in (PPI) handling code.
- Application code which is entered only after the first two layers are fully initialized.
Each layer's code is stored independently somewhere on the RCE, e.g., in configuration flash. The low-level interface to an RCE's protocol plug-ins uses abstractions called ports, virtual channels, frames, frame buffers, channels, lanes, pipes conduits and factories. A port virtual channel is the RCE end of a two-way communications link similar in concept to a BSD socket. Ports They are globally visible but not MT-safe; at a given time at most one thread may be waiting for data from, receiving data from or delivering data to any given portvirtual channel. Ports Virtual channels receive and transfer data packaged in frames. The exact content of a frame depends on the protocol being used. One frame corresponds to one I/O message and is delivered in a single frame buffer. In other words all ports virtual channels implement datagram rather than byte-stream protocols. It's up to higher-level software such as a TCP stack to provide any operations that cross frame boundaries. Each port contains virtual channel uses two queues of frames; one for frames which have arrived and have not yet been consumed by the application, the other for frames generated by the application but which have not yet been transmitted. In addition to a frame, a frame buffer contains administrative information that is private to the implementation of the plug-in interface; the user gives and takes only void* pointers to the beginning of the frame proper. Information such as the size of the frame has to be fixed by the protocol or has to be included in the frame itself. A channel port represents a hardware I/O engine capable of DMA to and from system RAM, i.e., a protocol plug-in of a particular type. An RCE has at most eight channels ports (which limit derives from the limit on the number of plugins). Most channels will make use of one or more of the Multi-Gigabit Transceiver modules (lanes) available in firmwareEach port may connect to pins leading out of the FPGA, though some may simply offer access to on-board FPGA resources such as DSPs. Each channel has its own port No pin may be used by more than one port. Each port has its own virtual channel space where each port virtual channel is identified by a 16-bit unsigned integer. Each port virtual channel represents a different source and/or sink of data such as a UDP port or a section of Petacache memory. The size of the port virtual channel number space on a channel port depends on the channel port type and may be as low as one. Every port virtual channel object created is allocated a port virtual channel number from its channelport's space but no two ports virtual channels of a channel port may have the same port virtual channel number. Each incoming frame must contain information from which a port virtual channel number can be derived. If it doesn't, or if it specifies a port virtual channel number that is invalid or not allocated to a port virtual channel object, then the channelport's lost-frame count is incremented and the frame is recycled. All the lanes (if any) of a given channel map to one pipe, which represents some on-board or off-board target. The FPGA is connected to the larger system by data paths called conduits. Each conduit connects to one or more pins on the FPGA; no pin may belong to more than one conduit. At boot time each port that uses pins will be matched to the conduit that connects to exactly the same set of pins. If the resulting mapping is not one-to-one then the boot process fails. Each conduit has associated type and version numbers which the software for the matching port checks for validity at boot time. If the software rejects the conduit then again booting will fail. Each type of port has both an official (unsigned) number and Each type of channel has both an official (unsigned) number and an official short name such as "eth" or "config". Channels Ethernet ports that differ only in the number of lanes pins they use, e.g., 10 Gb/s ethernet (4) and a slower ethernet (1) have the same channel type, in this case "eth"port type and short name. PPI-handling code software is held in relocatable modules recorded on the RCE. A module for plugin type FOO holds: - An implementation of a class FooChannel FooPort derived from ChannelPort.
- An implementation of class FooFactory derived from Factory. A FooFactory creates FooChannel instances, returning ChannelPort* values.
- An entry point that creates an instance of FooFactory, returning a Factory* value.
Each module is loaded, relocated and bound to the system core before its first use. The core code knows only the abstract base classes Factory and ChannelPort, not the derived classes specific to plugin type. Both PPI hardware and the channel port factory modules both have version numbers which will allow some measure of compatibility checking at boot time. Any incompatibility detected will will cause a fatal exception to be thrown during system startupcauses the boot to fail. InterfaceThe C++ class declarations given here contain only those members intended for the user. Whether a method is virtual is not specified, nor are friend declarations shown; these are considered implementation details. NamespacesAll the public declarations for the interface are in namespace RCE::ppi , as shown in this pseudo-C++ code: Code Block |
---|
|
namespace RCE {
namespace ppi {
static const unsigned MAX_PLUGINS = 8;
static const unsigned MAX_LANES = 32;
class ChannelPort;
class ChannelListPortList;
enum ChannelTypePortType;
class PortVirtualChannel;
}
}
|
All the header files for the public interface are the top level of package ppi of project rce . Code Block |
---|
|
release/
rce/
ppi/
constants.hh
Conduit.hh
ChannelConduitType.hh
ChannelListPort.hh
ChannelTypePortList.hh
PortPortType.hh
VirtualChannel.hh
|
Note that in the repository Note that in the repository both rce and release appear at the top level. Universal constantsAll RCEs whether of Gen I or Gen II each have the same limits on the number of plugin instances (MAX_PLUGINS ) and on the number of lanes conduits (MAX_LANESCONDUITS ). Channel Port-type enumerationThe numbers are members of an enumeration assigned by the DAQ project. Code Block |
---|
|
enum {
CONFIG_FLASH,
ETHERNET,
PGP1,
PGP2,
etc.,
INVALID_CHANNELTYPEPORTTYPE
} ChannelTypePortType;
|
The header file also contains a specialization of the template RCE::service::EnumInfo which allows one to use the function templates emin<>() , emax<>() , ecount<>() , evalid<>() , enext<>() , eprev<>() and estr<>() : Code Block |
---|
|
emin<ChannelType>emin<PortType>() == CONFIG_FLASH
emax<ChannelType>emax<PortType>() == ChannelTypePortType(INVALID_CHANNELTYPEPORTTYPE - 1)
ecount<ChannelType>ecount<PortType>() == int(INVALID_CHANNELTYPEPORTTYPE)
evalid(x) is true for all from emin() to emax() inclusive, else false
enext(emax()) == eprev(emin()) == INVALID_CHANNELTYPEPORTTYPE
enext(CONFIG_FLASH) == ETHERNET, etc.
eprev(ETHERNET) == CONFIG_FLASH, etc.
estr(CONFIG_FLASH) == "CONFIG_FLASH", etc.
estr(x) == "**INVALID**" if and only if evalid(x) is false
|
ecount<>() can't be used as a dimension for static arrays since the compiler considers it to be non-constant; in that case use EnumInfo<ChannelType>EnumInfo<PortType>::count .
Conduit-type enumeration (TBD)When specified this enumeration will be set up like the port type. Channel Port listThis class is a Borg-type singleton; the constructor makes a stateless object whose member functions access the true (shared) state defined elsewhere. The destructor destroys these stateless objects but does not touch the true state information. You can therefore just use the constructor whenever you need to access the One True List, e.g., ChannelListPortList().head() . You can get a count of the number of channels ports or the first channel port on the list (the list can't be empty). The report() member function will print informational messages in the system log which show the contents of the channel port list in brief form, one line per channelport. Note |
---|
The location and form of the system log depends on how the system logging package was initialized at application startup. Client code making log entries is not aware of this initialization. |
A particular channel port may be looked up in several different ways: - By its global ID number, assigned in sequence starting from zero as channels ports are created.
- By its type number and the creation sequence number within the type, e.g.,
(ETHERNET,0) , (ETHERNET,1) , etc. - By the number of the pipe the channel port is connected to.
Lookup methods return the null pointer if the search fails. Code Block |
---|
|
class ChannelListPortList {
public:
ChannelListPortList() {}
~ChannelList~PortList() {}
int numChannels() const;
ChannelPort* head() const;
ChannelPort* lookup(ChannelTypePortType type, unsigned typeSeq) const;
ChannelPort* lookup(unsigned id) const;
ChannelPort* lookupByPipe(unsigned pipe) const;
void report() const;
);
| ChannelPortA channel port object represents a particular instance of a protocol plug-in. Each channel port object is created at system startup. Channel Port objects live until system shutdown and may not be copied or assigned. Each channel port creates and destroys Port VirtualChannel objects on demand. During its lifetime each Port VirtualChannel object has exclusive use of one of the channels port ports virtual channel numbers; the port virtual channel number becomes available again once the Port VirtualChannel object is deallocated. The client code may request a specific, unused port virtual channel number for the type of channelport, e.g., a well-known TCP port number. The client may also allow the channel port to assign a number not currently in use by any PortVirtualChannel. Every channel port object is a member of the linked list accessed though class ChannelList PortList and may not be removed from the list. Use the next() member function to iterate over the list. A short name for the type and a short description of the channel port are also provided. A "lost" counter is provided which counts the number of incoming frames that were discarded, for whatever reason, instead of being queued in a portvirtual channel. Other information provided: - The ID of the pipe associated with the channelport.
- The hardware version number of the associated plugin.
- The software version number of the associated plugin module.
The report() member function produces detailed multi-line description of the channel port in the system log, including all platform-specific information. Code Block |
---|
|
class ChannelPort {
public:
PortVirtualChannel* allocate(int portNum);
PortVirtualChannel* allocate();
void deallocate(PortVirtualChannel *);
unsigned lost() const;
unsigned id() const;
unsigned type() const;
const char* name() const;
unsigned typeSequence() const;
unsigned pipe() const;
unsigned versionHard() const;
unsigned versionSoft() const;
ChannelPort* next() const;
const char* description() const;
void report() const;
};
| PortVirtual channelEach Port VirtualChannel object is created by a channel port and is assigned a unique ID in the channelport's port virtual channel number space. Incoming frames on the associated channel port may be waited for and retrieved using the wait() member function. Client code will normally keep a frame for a short time then give it back to the port virtual channel they got it from using the portvirtual channel's giveBack() member function. It's an error to try to give back a frame to a port virtual channel which didn't produce it; the results of doing so will be unpredictable. A port virtual channel takes frames given to its send() member function and queues them for output. The frame buffer is NOT reclaimed after sending nor does the plugin interface tell you when it's safe to re-use the buffer; that's up to the message protocol. For example, the PGP1 protocol used with Petacache flash memory allows the client to put an arbitrary transaction ID into every message sent, and every message produces a response incorporating the original transaction ID. The client can then track pending transactions and the send buffers associated with them. Code Block |
---|
|
class PortVirtualChannel {
public:
unsigned id() const;
void* wait();
void giveBack(void *frame);
void send(void *frame);
};
|
ImplementationGeneralBit numbering: Bit zero is the least significant bit of a value (contrary to the convention in PowerPC documents). NamespacesThe namespace RCE::ppi also contains implementation declarations but these don't appear in public header files but in package private headers (see below). Some names are declared in the same way for both Gen I and Gen II, others look different depending on the platform and some may not appear at all on a given platform. All implementation code (including inline function definitions) appears somwhere somewhere under the directory rce/ppi/src/ . Generic code appears directly in src/ while platform specific code will appear in subdirectories gen1/ , gen2/ , and possibly i86-linux/ . Unit test code is in rce/ppi/test/ which has the same substructure as src/ . Some implementation is partly generic and partly platform specific so a filename may appear in several places. Code Block |
---|
|
release/
rce/
ppi/
src/
Conduit.hh
ChannelConduit-inl.hh
ChannelConduit.cc
ChannelList-inlConduitType.hh
ChannelListConduitList.cchh
ConfigurationConduitList-inl.hh
FactoryConduitList.hhcc
Port-inl.hh
Port.cc
gen1/PortList-inl.hh
ChannelListPortList.cc
Configuration.hh
Configuration.cc
ConfigurationSpace.hh
ConfigurationData.hh
Factory.hh
Factory.cc
VirtualChannel-inl.hh
PipeVirtualChannel.hhcc
gen1/
Plugin.hh
PortList.cc
etc.
gen2/Configuration.cc
etcConfigurationData.hh
etc.
Factory.cc
test/
testPpietc.cc
gen1gen2/
testPpietc.cc
etc.
|
Channel listWe export a static list-building member function void build() whose implementation depends on the platform.. All the instances of ChannelList created by the constructor are empty; the non-static member functions all use the static head-of-list pointer and never use this . Code Block |
---|
none | none |
class ChannelList {
public:
etc.
static void build();
private:
test/
testPpi.cc
gen1/
static Channel* m_head; testPpi.cc
static int m_numChannels;
};
|
ChannelThe protected constructor takes a PPI definition object plus other information supplied by the factory code and the system startup code. There are also protected member functions to increment the "lost" count and to set the "next" pointer (even though ChannelList is a friend we prefer to use an accessor for the "next" pointer). From the factory code: - name
- instance
- versionSoft
- description
From the startup code: - An instance of Plugin
- versionHard
- id
- pipe
Conduit listAll the instances of ConduitList created by the default constructor are empty; the non-static member functions all use the static head-of-list pointer and never use this . The constructor that takes a Configuration\& argument actually builds the list before returning the empty object. Code Block |
---|
|
class Plugin;
class ChannelConduitList {
protectedpublic:
friend class ChannelListConduitList();
explicit Channel(ConduitList(const Configuration&);
Conduit* head() const;
unsigned Plugin*,numConduits() const;
Conduit* lookup(unsigned long unsigned id,
ChannelType type,long pins);
private:
const char* name,static Conduit* m_head;
static int m_Conduits;
};
|
Conduit Code Block |
---|
|
class Conduit {
public:
unsigned id() const;
ConduitType type() const;
unsigned version() const;
unsigned long long pins() const;
Conduit* next() const;
protected:
friend class Configuration;
Conduit(unsigned id, ConduitType, unsigned version, unsigned long long pins);
~Conduit();
void next(Conduit*);
};
|
Port listAll the instances of PortList }} created by the default constructor are empty; the non-static member functions all use the static head-of-list pointer and never use this . The constructor that takes a Configuration\& argument actually builds the list before returning the empty object. Code Block |
---|
|
class PortList {
public:
PortList(const Configuration&);
private:
static Port* m_head;
static int m_ports;
};
|
PortThe protected constructor takes a PPI definition object plus other information supplied by the factory code and the system startup code. There are also protected member functions to increment the "lost" count and to set the "next" pointer (even though PortList is a friend we prefer to use an accessor for the "next" pointer). Code Block |
---|
|
class Plugin;
class Port {
public:
virtual unsigned lookup(void*);
protected:
friend class Configuration;
Port(
unsigned id,
PortType type,
unsigned long long pins,
unsigned versionHard);
virtual ~Port() = 0;
void incLost();
void next(Port*);
private:
Port(const Port &);
Port& operator=(const Port &) unsigned typeSequence,
unsigned pipe,
unsigned versionHard,
unsigned versionSoft,
const char* description);
virtual ~Channel() = 0;
void incLost();
void next(Channel*);
private:
Channel(const Channel &);
Channel& operator=(const Channel &);
private:
Plugin* m_plugin;
unsigned m_lost;
Channel* m_next;
unsigned m_id;
ChannelType m_type;
const char* m_name;
unsigned m_typeSequence;
unsigned m_pipe;
unsigned m_versionHard;
unsigned m_versionSoft;
const char* m_description;
};
|
The name and description string pointers must refer to non-automatic storage since the object must remain valid until shutdown. PortVirtual channelThe first private constructor stores the given ChannelPort* and port virtual channel number. Code Block |
---|
|
class PortVirtualChannel {
private:
friend class ChannelPort;
VirtualChannel(Port(Channel*, unsigned portNum);
~Port~VirtualChannel();
PortVirtualChannel(const PortVirtualChannel&);
PortVirtualChannel& operator=(const PortVirtualChannel&);
private:
ChannelPort* m_channelport;
unsigned m_id;
};
| Channel Port factoryThe abstract base class for objects that manufacture Channel Port instances. Channel Port factories are created during system startup and last until system shutdown. Assignment or copying of instances is forbidden. The derived classes use the protected constructor to save the channel port type name, software version number, recommended number of reception buffers per channel port and the channel port description string. Code Block |
---|
|
class Factory {
public:
ChannelTypePortType type() const;
const char* name() const;
unsigned versionSoft() const;
unsigned numBuffersAdvised() const;
const char* description() const;
virtual ChannelVirtualChannel* createChannel(
Plugin*,
unsigned channelId,
unsigned pipeId);
) = 0;
protected:
Factory(
unsigned type,
const char* name,
unsigned versionSoft,
unsigned numBuffersAdvised,
const char *description
);
virtual ~Factory() = 0;
};
|
Plugin software modulesEach module's entry point is named rce_appmain ; this symbol is recognized by the module build system which places its value in the transfer address slot of the module's ELF header. The prototype of the entry point function is Code Block |
---|
|
extern "C" Factory* rce_appmain();
|
The system init code uses placement new to construct an instance of PluginModule atop the image of the module then calls the run() member function to obtain the factory object. Code Block |
---|
|
class PluginModule: public RCE::ELF::Module {
public:
typedef Factory* (*EntryPoint)();
Factory* run();
};
|
The module's entry point is run directly without creating a new thread (otherwise we'd need synchronization in order to wait for the factory to be produced). Configuration table accessThe same declaration is used for both Gen I and Gen II since it uses only references to classes Pipe and Plugin . The implementations are quite different, however. Code Block |
---|
|
static unsigned EMPTY = 0xffffffff;
class Configuration {
public:
Configuration();
~Configuration();
void plugin(unsigned defNum, Plugin&);
void pipe(unsigned lane, Pipe&);
};
|
Gen IFrames are still divided into header and payload sections. Each channel port manages its own pool of reception (import) buffers. Configuration container zero in the configuration flash contains tables of information about the hardware and firmware; this information can't be gotten directly from the hardware and firmware. To make alignment easier we use 32-bit fields wherever possible, even for 16-bit quantities such as port virtual channel numbers. The plugin table will come first, at the beginning of the container, immediately followed by the pipe table and the network address table. Plugin tableThe plugin table will have eight array elements, one for each possible PPI in the system. An element is considered unused if the type field contains 0xffffffff (EMPTY ) in which case all the other members of that element are zero. Field name | Type | Description |
---|
type | uint32_t | The type number of the resulting Channel Port object as given in the header ChannelTypes.hh | lanes | uint32_t | Lane usage bitmask; bit N is set to indicate that lane N is used by this PPI | version | uint32_t | The version number for the PPI firmware | maxPorts | uint32_t | The size of the port virtual channel number space | pibs | uint32_t | PIB usage bitmask | pebs | uint32_t | PEB usage bitmask | ecbs | uint32_t | ECB usage bitmask | flbs | uint32_t | FLB usage bitmask | importPayloadSize | uint32_t | The maximum size in bytes of an import frame payload | importHeaderSize | uint32_t | The size of an import frame header | exportPayloadSize | uint32_t | The maximum size in bytes of an export frame payload | exportHeaderSize | uint32_t | The size of an export frame header |
Pipe tableIn order to discover which pipe a PPI uses will you take the lowest lane number used by the PPI and use it to index the pipe table, which has 32 array elements. A pipe ID of EMPTY indicates an unused element, otherwise the ID of the pipe is given. A pipe ID may contain subfields telling what the pipe is connected to or it may be a simple index into a further table that gives such information. Field name | Type | Description |
---|
pipeId | uint32_t | Pipe ID number |
Network address tableEthernet plugins for Gen I RCEs need to have their MAC and IP addresses supplied to them. The table has eight array elements with each unused element having a MAC address of 0xffffffff ffffffff . The information is read from configuration flash and stored in main memory from whence an API yet to be defined will retrieve it. Field name | Type | Description |
---|
mac | uint64_t | Ethernet MAC address, right justified | ip | uint32_t | IPv4 address in network byte order | pipeId | uint32_t | Pipe ID number |
Storage of plugin software modulesOnly a few different types of protocol plugins are found on Gen I systems so each type is assigned to a fixed Configuration container: Plugin type | Container name |
---|
ETHERNET | 1 | PGP1 | 2 | PGP2 | 3 | future expansion | 4-10 |
Later a container may be used for CONFIG_FLASH, if we ever manage to make a usable wrapper for the FCI package makes it look like another plugin. Class BufferFrames buffers for imports will be allocated by the system startup code; those for exports will be allocated by the client software. Before a frame buffer may be used for either import or export an instance of class Buffer must be created at the beginning of the buffer using placement new. A Buffer contains regions of fixed size used in the management of the frame itself and its buffer: - The firmware's in-memory descriptor. The firmware TDE for the frame points to the descriptor.
- Status. The firmware writes operation completion status here.
- One or more links used to make singly-linked lists.
Immediately following the end of the Buffer we have first the frame's header and then its payload. The links, descriptor and status areas have the same sizes for all channel port types. The descriptor must begin on a 64-byte boundary; for ease of layout the entire buffer also has that alignment. The buffer should also begin on a cache line boundary in order to reduce cache thrashing but since a cache line is only 32 bytes long this requirement is already met. Buffer instances may not be copied or assigned. Code Block |
---|
|
class Buffer {
public:
Buffer(void *bufferAddress);
Buffer* link(unsigned linkNum) const;
const Descriptor& descriptor() const;
const OpStatus& status() const;
char* header() const;
char* payload() const;
enum {NUM_LINKS = 4};
private:
Descriptor m_descr;
OpStatus m_status;
Buffer* m_link[NUM_LINKS];
};
|
Use case: System startup- Boot code:
- Loads and starts the system core.
- System core
- Initializes the CPU.
- Initializes RTEMS.
- Initializes any extra C++ support.
- Sets up the MMU's TLB and enables the MMU.
- Creates the default instance of the dynamic linker.
- Reads Configuration container zero.
- Saves the ethernet MAC table for later use.
- Reads, links and runs the required plugin modules, creating the Factory objects.
- Creates the import buffer pools for all plugins (using information supplied by the factories).
- Uses the Factories to create all required Channel Port objects (the Factories know how to find the buffer pools).
- Initializes the network (uses the network address table).
- Initializes each ethernet channelport.
- Initializes IP, UDP, TCP and BSD sockets.
- Gets any required DHCP leases.
- Loads and links the application code using the default dynamic linker.
- Calls the application entry point.
Use case: Frame importTBD. Use case: Frame exportTBD. Gen IILocation of the configuration tablesThe PpiDefs and PipeDefs table entries will be obtained directly from the hardware via DCR registers. Protocol plug-ins.Protocol plug-ins will no longer be implemented using PIC blocks. FramesThe old division of frames into header and payload will no longer be made. All input frame buffers for all channels will be held in a single free-list maintained by the firmware; when asked for a buffer of a certain size it allocates the one whose size is the best match. |