Draft 3.
...
4
Table of Content Zone | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ConceptsRCE code is divided into three layers:
Each layer's code is stored independently somewhere on the RCE, e.g., in configuration flash. The low-level interface to an RCE's protocol plug-ins uses abstractions called ports, frames, frame buffers, channels, lanes, pipes and factories. A port is the RCE end of a two-way communications link similar in concept to a BSD socket. Ports are globally visible but not MT-safe; at a given time at most one thread may be waiting for data from, receiving data from or delivering data to any given port. Ports receive and transfer data packaged in frames. The exact content of a frame depends on the protocol being used. One frame corresponds to one I/O message and is delivered in a single frame buffer. In other words all ports implement datagram rather than byte-stream protocols. It's up to higher-level software such as a TCP stack to provide any operations that cross frame boundaries. Each port contains two queues of frames; one for frames which have arrived and have not yet been consumed by the application, the other for frames generated by the application but which have not yet been transmitted. In addition to a frame, a frame buffer contains administrative information that is private to the implementation of the plug-in interface; the user gives and takes only A channel represents a hardware I/O engine capable of DMA to and from system RAM, i.e., a protocol plug-in of a particular type. An RCE has at most eight channels (which limit derives from the limit on the number of plugins). Most channels will make use of one or more of the Multi-Gigabit Transceiver modules (lanes) available in firmware, though some may simply offer access to on-board resources such as DSPs. Each channel has its own port space where each port is identified by a 16-bit unsigned integer. Each port represents a different source and/or sink of data such as a UDP port or a section of Petacache memory. The size of the port number space on a channel depends on the channel type and may be as low as one. Every port object created is allocated a port number from its channel's space but no two ports of a channel may have the same port number. Each incoming frame must contain information from which a port number can be derived. If it doesn't, or if it specifies a port number that is invalid or not allocated to a port object, then the channel's lost-frame count is incremented and the frame is recycled. All the lanes (if any) of a given channel map to one pipe, which represents some on-board or off-board target. Each type of channel has both an official (unsigned) number and an official short name such as "eth" or "config". Channels that differ only in the number of lanes they use, e.g., 10 Gb/s ethernet (4) and a slower ethernet (1) have the same channel type, in this case "eth". PPI-handling code is held in relocatable modules recorded on the RCE. A module for plugin type FOO holds:
Each module is loaded, relocated and bound to the system core before its first use. The core code knows only the abstract base classes Factory and Channel, not the derived classes specific to plugin type. Both PPI hardware and the channel factory modules both have version numbers which will allow some measure of compatibility checking. Any incompatibility detected will will cause a fatal exception to be thrown during system startup. InterfaceThe C++ class declarations given here contain only those members intended for the user. Whether a method is virtual is not specified, nor are friend declarations shown; these are considered implementation details. NamespacesAll the public declarations for the interface are in namespace
Header filesAll the header files for the public interface are the top level of package
Note that in the repository both Universal constantsAll RCEs whether of Gen I or Gen II each have the same limits on the number of plugin instances ( Channel type enumerationThe numbers are members of an enumeration assigned by the DAQ project.
The header file also contains a specialization of the template
Channel listThis class is a Borg-type singleton; the constructor makes a stateless object whose member functions access the true (shared) state defined elsewhere. The destructor destroys these stateless objects but does not touch the true state information. You can therefore just use the constructor whenever you need to access the One True List, e.g., You can get a count of the number of channels or the first channel on the list (the list can't be empty). The
A particular channel may be looked up in several different ways:
Lookup methods return the null pointer if the search fails.
ChannelA channel object represents a particular instance of a protocol plug-in. Each channel object is created at system startup. Channel objects live until system shutdown and may not be copied or assigned. Each channel creates and destroys Port objects on demand. During its lifetime each Port object has exclusive use of one of the channels port numbers; the port number becomes available again once the Port object is deallocated. The client code may request a specific, unused port number for the type of channel, e.g., a well-known TCP port number. The client may also allow the channel to assign a number not currently in use by any Port. Every channel object is a member of the linked list accessed though class ChannelList and may not be removed from the list. Use the A short name for the type and a short description of the channel are also provided. A "lost" counter is provided which counts the number of incoming frames that were discarded, for whatever reason, instead of being queued in a port. Other information provided:
The
PortEach Port object is created by a channel and is assigned a unique ID in the channel's port number space. Incoming frames on the associated channel may be waited for and retrieved using the A port takes frames given to its
ImplementationGeneralBit numbering: Bit zero is the least significant bit of a value (contrary to the convention in PowerPC documents). NamespacesThe namespace Header filesAll implementation code (including inline function definitions) appears somwhere under the directory
Channel listWe export a static list-building member function
ChannelThe protected constructor takes a PPI definition object plus other information supplied by the factory code and the system startup code. There are also protected member functions to increment the "lost" count and to set the "next" pointer (even though From the factory code:
From the startup code:
The name and description string pointers must refer to non-automatic storage since the object must remain valid until shutdown. PortThe first private constructor stores the given Channel* and port number.
Channel factoryThe abstract base class for objects that manufacture Channel instances. Channel factories are created during system startup and last until system shutdown. Assignment or copying of instances is forbidden. The derived classes use the protected constructor to save the channel type name, software version number, recommended number of reception buffers per channel and the channel description string.
Plugin software modulesEach module's entry point is named
The system init code uses placement new to construct an instance of PluginModule atop the image of the module then calls the
The module's entry point is run directly without creating a new thread Configuration table accessThe same declaration is used for both Gen I and Gen II since it uses only references to classes
Gen IFrames are still divided into header and payload sections. Each channel manages its own pool of reception (import) buffers. Location and format of the configuration tablesConfiguration container zero in the configuration flash contains tables of information about the hardware and firmware; this information can't be gotten directly from the hardware and firmware. To make alignment easier we use 32-bit fields wherever possible, even for 16-bit quantities such as port numbers. The plugin table will come first, at the beginning of the container, immediately followed by the pipe table and the ethernet MAC network address table. Plugin tableThe plugin table will have eight array elements, one for each possible PPI in the system. An element is considered unused if the
Pipe tableIn order to discover which pipe a PPI uses will you take the lowest lane number used by the PPI and use it to index the pipe table, which has 32 array elements. A pipe ID of
Network address tableEthernet plugins for Gen I RCEs need to have their MAC and IP addresses supplied to them. System init code also needs to know which ethernet interfaces need to used DHCP in order to obtain IP numbers. The table has eight array elements with each unused element having a MAC address of
Storage of plugin software modulesOnly a few different types of protocol plugins are found on Gen I systems so each type is assigned to a fixed Configuration container:
Later a container may be used for CONFIG_FLASH, if we ever manage to make a usable wrapper for the FCI package makes it look like another plugin. Class BufferFrames buffers for imports will be allocated by the system startup code; those for exports will be allocated by the client software. Before a frame buffer may be used for either import or export an instance of class Buffer must be created at the beginning of the buffer using placement new. A Buffer contains regions of fixed size used in the management of the frame itself and its buffer:
Immediately following the end of the Buffer we have first the frame's header and then its payload. The links, descriptor and status areas have the same sizes for all channel types. The descriptor must begin on a 64-byte boundary; for ease of layout the entire buffer also has that alignment. The buffer should also begin on a cache line boundary in order to reduce cache thrashing but since a cache line is only 32 bytes long this requirement is already met. Buffer instances may not be copied or assigned.
Use case: System startup
Use case: Frame importTBD. Use case: Frame exportTBD. Gen IILocation of the configuration tablesThe PpiDefs and PipeDefs table entries will be obtained directly from the hardware via DCR registers. Protocol plug-ins.Protocol plug-ins will no longer be implemented using PIC blocks. FramesThe old division of frames into header and payload will no longer be made. All input frame buffers for all channels will be held in a single free-list maintained by the firmware; when asked for a buffer of a certain size it allocates the one whose size is the best match. |