Concepts

RCE code is divided into components:

1. The core is the same for all RCEs of a given generation. It contains low-level processor management code, RTEMS, generic C/C++ support libraries, etc.
2. The protocol plug-in (PPI) software modules.
3. Application code which is entered only after the first two components are fully initialized.

Each component's code is stored independently somewhere on the RCE, e.g., in configuration flash.

The core and the PPI software modules together offer the use of protocol plugins at different levels of abstraction:

1. Low-level for boot code and plugin software modules.
2. Mid-level for raw device users.
3. High-level for most applications.

This document describes the first two levels of PPI service interface which uses abstractions called ports, virtual channels, payloads (data), conduits and factories.

Mid-level

A port represents a hardware I/O engine capable of DMA to and from system RAM, i.e., a protocol plug-in of a particular type. An RCE has at most eight ports (which limit derives from the limit on the number of plugins). Each type of port has both an official type number and an official short name such as "eth" or "config". Ethernet ports that differ only in speed, e.g., 10 Gb/s ethernet and a 100 Mb/s ethernet have the same port type and short name. Each port has a certain number of virtual channels which may be allocated and deallocated upon request from the user; some types of ports may have as few as one virtual channel while others may have thousands.

A virtual channel is one end of a two-way communications link similar in concept to a BSD socket. They are globally visible but not MT-safe; at a given time at most one thread may be making use of a given virtual channel. Each virtual channel represents a different source and/or sink of data such as a UDP port or a section of Petacache memory. Each has a number that uniquely identifies it amongst all the virtual channels belonging to the same port. When asking for a virtual channel to be allocated the user may specify a specific number that may have meaning at a higher level, e.g., it may map to a UDP port number. Even when asking for a specific virtual channel number no channel may be allocated more than once. The other method of allocation just picks some virtual channel that has not yet been allocated. When finished with a virtual channel the user passes it back to the owning port for deallocation.

Virtual channels supply and are supplied with data in the form of Payload objects. The content of a Payload depends on the protocol being used. At this level each Payload represents a single inbound or outbound message where the boundaries between messages are respected; in other words the virtual channel implements datagram rather than byte-stream protocols. It's up to high level software such as a TCP stack to provide any operations that cross message boundaries.

To transmit an outgoing message the user requests an empty Payload from the virtual channel, fills its buffer with data then passes the Payload back to the virtual channel for transmission. To receive an incoming message the user makes a request that blocks until a message is available, uses the data in the Payload eventually supplied then returns the Payload to the virtual channel that supplied it.

Low-level

Each message coming in on a port must contain information from which a destination virtual channel can be inferred. If it doesn't, or if it specifies a virtual channel that is invalid or not allocated, then the port's lost-message count is incremented and the Payload object is recycled.

Each port may connect to pins leading out of the FPGA, though some may simply offer access to FPGA resources such as DSPs. No pin may be used by more than one port.

The FPGA is connected to the larger system by data paths called conduits. Each conduit connects to one or more pins on the FPGA; no pin may belong to more than one conduit. At boot time each port that uses pins will be matched to the conduit that connects to exactly the same set of pins. If the resulting mapping is not one-to-one then the boot process fails. Each conduit has associated type and version numbers which the software for the matching port checks for validity at boot time. If the software rejects the conduit then again booting will fail.

PPI-handling software is held in relocatable modules recorded on the RCE. A module for plugin type FOO holds:

• An implementation of a class FooPort derived from Port.
• An implementation of class FooFactory derived from Factory. A FooFactory creates FooPort instances returned as Port* values.
• An entry point that creates an instance of FooFactory returned as a Factory* value.

Each module is loaded, relocated and bound to the system core before its first use. The core code knows only the abstract base classes Factory and Port, not the derived classes specific to plugin type.

Both PPI hardware and the port factory modules have version numbers which will allow some measure of compatibility checking at boot time. Any incompatibility detected causes the boot to fail.

RCE boot code discovers the set of protocol plugins and conduits available using a set of configuration registers. These registers have their own address space, the "configuration space". For each plugin the configuration space registers yield the plugin type, the plugin version number and the set of FPGA pins connected to it. For each conduit they yield the conduit type, the conduit version number and its set of FPGA pins.

Access to configuration registers is via an object which given an abstract register address reads or writes the contents of the corresponding register. Another object uses the abstract register layer to provide all the configuration info for a plugin or a conduit given its index number (or supplies an indication that the given entity doesn't exist).

Differences between Gen I and Gen II

Gen II

The configuration space registers are hardware memory locations filled with information by the IPMI Controller (IPMC).

Message buffers, once created, are managed entirely by firmware. Payload and headers are managed independently of each other. The firmware manages buffers directly using the buffer addresses.

Gen I

There is no IPMC and there are no hardware configuration registers. The low-level configuration information is burned into a configuration flash container. At boot time the container is read and its contents fed into an object which simulates the Gen II configuration register space. Above that layer the handling of the information is just like that in Gen II.

Protocol plugins are assembled from Protocol Interface Core (PIC) blocks which have no counterpart in Gen II. We want to make the configuration information handling like that in Gen II, so rather than extending it with PIC block assignments we sweep those under the rug by embedding them in the plugin software modules.

Control of I/O buffers passes between firmware and application software; a Transfer Descriptor Entry (TDE) is used to pass buffer references in either direction. Frame headers are fully exposed to software which must know for a given plugin the header size, the maximum payload size and the maximum number of buffers for a given plugin. This information can also be embedded in the plugin-handling module at the cost of having to tailor the module to the RCE application.

Mid-level API

The class declarations given in this section contain only those members intended for use after booting is complete and all plugins are on-line. Whether a method is virtual is not specified, nor are friend declarations shown; these are considered implementation details.

Classes and their responsibilities

Universal constants (constants.hh)

All RCEs whether of Gen I or Gen II each have the same limits on the number of plugin instances (MAX_PLUGINS).

static const unsigned MAX_PLUGINS  =  8;

Port-type enumeration (PortTypes.hh)

The numbers are members of an enumeration assigned by the Data Acquisition Tools project.

enum PortType {
    CONFIG_FLASH,
    ETHERNET,
    PGP1,
    PGP2,
    etc.,
    INVALID_PORT_TYPE
};

The header file also contains a specialization of the template RCE::service::EnumInfo which allows one to use the function templates emin<>(), emax<>(), ecount<>(), evalid<>(), enext<>(), eprev<>() and estr<>():

emin<PortType>() == CONFIG_FLASH
emax<PortType>() == PortType(INVALID_PORTTYPE - 1)
ecount<PortType>() == int(INVALID_PORTTYPE)
evalid(PortType x) is true for all from emin() to emax() inclusive, else false
evalid(int) and evalid(unsigned) make similar tests on ints and unsigneds.
enext(emax()) == eprev(emin()) == INVALID_PORTTYPE
enext(CONFIG_FLASH) == ETHERNET, etc.
eprev(ETHERNET) == CONFIG_FLASH, etc.
estr(CONFIG_FLASH) == "CONFIG_FLASH", etc.
estr(x) == "**INVALID**" if and only if evalid(x) is false

ecount<>() can't be used as a dimension for static arrays since the compiler considers it to be non-constant; in that case use EnumInfo<PortType>::count.

Port list (PortList.hh)

This class is a Borg-type singleton; the constructor makes a stateless object whose member functions access the true (shared) state defined elsewhere. The destructor destroys these stateless objects but does not touch the true state information. You can therefore just use the constructor whenever you need to access the One True List, e.g., PortList().head().

You can get a count of the number of ports or the first port on the list (the list can't be empty). The report() member function will print informational messages in the system log which show the contents of the port list in brief form, one line per port.

A particular port may be looked up in several different ways:

• By its global index number, assigned in sequence starting from zero as ports are created.
• By its type number and the index number within the type. The first ethernet port would be (ETHERNET,0), the second (ETHERNET,1), etc.
• By the number of the conduit the port is connected to.

Lookup methods return the null pointer if the search fails.

class PortList {
public:
    PortList() {}
    ~PortList() {}
    int numPorts() const;
    Port* head() const;
    Port* lookup(PortType type, unsigned typeIndex) const;
    Port* lookup(unsigned index) const;
    Port* lookupByConduit(unsigned conduit) const;
    void report() const;
);

Port (Port.hh)

A port object represents a particular instance of a protocol plug-in. Each port object is created at boot time. Port objects live until system shutdown and may not be copied or assigned.

Each port allocates and deallocates VirtualChannel objects on demand. During its lifetime each VirtualChannel object has exclusive use of one of the port's virtual channel numbers; the virtual channel number becomes available again once the VirtualChannel object is deallocated. The application code may request a specific, unused virtual channel number for the type of port, e.g., a well-known TCP port number. The application may also allow the port to assign a number not currently in use by any VirtualChannel.

Every port object is a member of the linked list accessed though class PortList and may not be removed from the list. Use the next() member function to iterate over the list.

A short name for the type and a short description of the port are also provided.

A "lost" counter is provided which counts the number of inbound messages that were discarded, for whatever reason.

Other information provided:

• The index number of the conduit associated with the port.
• The hardware version number of the associated plugin.
• The software version number of the associated plugin module.
• A bitmask giving the FPGA pins (if any) used by the plugin.
• The size of the VC-number space.

The report() member function produces detailed multi-line description of the port in the system log, including all platform-specific information.

High-level and mid-level code isn't allowed to create or destroy instances; only low-level code is allowed to do that.

class Port {
public:
    VirtualChannel* allocate(int vcNum);
    VirtualChannel* allocate();
    void deallocate(VirtualChannel *);
    unsigned lost() const;
    unsigned index() const;
    unsigned type() const;
    const char* name() const;
    unsigned typeIndex() const;
    uunsigned conduit() const;
    unsigned versionHard() const;
    unsigned versionSoft() const;
    Port* next() const;
    const char* description() const;
    unsigned maxVcs() const;
    void report() const;
};

Virtual channel (VirtualChannel.hh)

Each VirtualChannel object is allocated by a Port and is assigned a unique ID in the Port's virtual channel number space.

Messages inbound on the associated port may be waited for and retrieved using the receive() member function. Client code will normally keep a Payload for a short time then give it back to the virtual channel they got it from using the virtual channel's deallocate() member function. It's an error to request a virtual channel to deallocate a Payload it didn't produce; the result of doing so will be unpredictable.

A virtual channel takes Payloads given to its transmit() member function and queues them for output. Any such Payload object must have been allocated using the allocate() member function of the same virtual channel (or its receive()); breaking this rule results in unpredictable behavior. Once the data are transmitted the Payload is automatically deallocated, so the user should not try to use it after calling transmit().

class VirtualChannel {
public:
    unsigned vcNum() const;

    Payload* receive();         // To receive: first this ...
    void deallocate(Payload*);  // ... then this.

    Payload* allocate();        // To transmit: first this ...
    void transmit(Payload*);    // ... then this.
};

Payload Payload.hh

A Payload object p owns size() bytes of message data in a buffer of capacity p.capacity() bytes located at p.data().

class Payload {
public:
    void *data() const;
    size_t size() const;
    size_t capacity() const;
};

Low level API

In this section we describe the code used to manage configuration information and construct the global PortList. Some of the classes already introduced above will have new members described here; other classes will be completely new.

New members of old classes

The PortList class has a static member function build() whose main purpose is to produce the list of Port instances. To do so it will have to read configuration information about plugins and conduits, load and activate plugin software and match conduits to ports. There is also a member function add() which places a new Port on the end of the list.

class PortList {
  public:
  static void build();
  private:
  void add(Port*);
};

The Port class' constructor is used by PortList::build(). It's constructor and destructor are used by derived classes. Also provided are the means to increment the count of lost import frames, to set the next entry in the PortList and to find out the set of FPGA pins used by the port.

class Port {
protected:
  Port(unsigned index,
       PortType type,
       const char *name,
       unsigned typeIndex,
       unsigned conduit,
       unsigned versionHard,
       unsigned versionSoft,
       const char* description,
       unsigned long long pins,
       unsigned maxVcs);
   virtual ~Port() = 0;
   unsigned long long pins() const;
   void incLost();
   void next(Port*);
};

Instances of VirtualChannel are created and destroyed only inside Ports. There is an access member added which gives the owning Port instance.

class VirtualChannel {
private:
  VirtualChannel(Port*, unsigned vcNum);
  ~VirtualChannel();
  Port* port() const;
};

New classes and their responsibilities

ConduitConfig (ConduitConfig.hh)

This is a Plain Old Data (POD) class describing a single conduit. The default constructor sets all members to zero.

The default constructor creates an invalid instance, one that represents a conduit that doesn't exist. A member function tests whether the instance is a valid one.

class ConduitConfig {
public:
  unsigned index;
  ConduitType type;
  unsigned version;
  unsigned long long pins;
  ConduitConfig();
  ConduitConfig(unsigned ind, ConduitType, unsigned ver, unsigned long long);
  bool isValid() const;
};

ConduitType (ConduitType.hh)

These types are not well defined yet so for now we just define a generic type code. The header also provides a specialization of rce::service::EnumInfo<> similar to that provided for PortType.

enum ConduitType {
  CONDUIT,
  INVALID_CONDUIT_TYPE
};

ConfigReader (ConfigReader.hh)

One member function returns instances of PluginConfig, the other returns instances of ConduitConfig. Both take an argument that is the index of the object whose configuration you want to look up; plugins and conduits are numbered separately staring from zero. The lookups return invalid config instances if the requested entities don't exist.

Instances have no data of their own but get what they need from ConfigSpace; you can generate and throw away instances as often as you want.

class ConfigReader {
  public:
  void lookupConduit(unsigned index, ConduitConfig&) const;
  void lookupPlugin(unsigned index, PluginConfig&) const;
};

ConfigSpace (ConfigSpace.hh)

Each instance implements an abstract space of configuration registers. How the abstract registers are used to collect configuration information is an implementation decision which will however be the same for both Gen I and II. Register addresses start at zero; an attempt to read or write a register at an invalid address, or to write to a read-only register, will throw std::logic_error. Use the implements() member function to determine a virtual register is implemented as a given address.

Instances have no data of their own but get what they need from some central source on the RCE; exactly where differs between Gen I and Gen II. You can create and destroy instances at will.

class ConfigSpace {
  public:
  ConfigSpace();
  bool implements(unsigned address) const;
  unsigned read(unsigned address) const;
  void write(unsigned address, unsigned value);
};

PluginConfig (PluginConfig.hh)

This is a Plain Old Data (POD) class describing a single plugin instance. The default constructor sets all members to zero.

The default constructor creates an invalid instance, one that represents a conduit that doesn't exist. A member function tests whether the instance is a valid one.

struct PluginConfig {
  unsigned index;
  PortType type;
  unsigned version;
  unsigned long long pins;
  PluginConfig();
  PluginConfig(unsigned ind, PortType, unsigned ver, unsigned long long);
  bool isValid() const;
};

PortFactory (PortFactory.hh)

This is an abstract base class. Once the boot code knows the types of the available plugins it will load the plugin software module for each type. It will call the entry point of each plugin software module once to obtain an instance of a class derived from PortFactory.

Once it has matched a PluginConfig instance with a ConduitConfig instance, or determines that the plugin needs no conduit, the boot code uses the factory object to create Port instances for the given type of plugin. If the plugin and conduit versions are incompatible the factory member function will throw std::logic_error. It will do the same if no ConduitConfig is supplied when one is required.

The report function will log full details of the factory, including any platform-dependent information.

class PortFactory {
public:
  PortType type() const;
  const char* name() const;
  unsigned version() const;
  const char* description() const;
  Port* makePort(const PluginConfig&);
  Port* makePort(const PluginConfig&, const ConduitConfig&);
  PortFactory* next() const;
  void next(PortFactory*);
  void report() const = 0;
protected:
  PortFactory(PortType, const char* name, unsigned version, const char* description);
  ~PortFactory();
};

PortFactoryList (PortFactoryList.hh)

Another Borg singleton, very similar in concept to PortList; the underlying list of factories is built at about the same time as the list of ports. There is at most one factory per port type. The lookup function returns a null pointer if no matching factory is on the list. The report function logs a one-line summary per factory. New PortFactory instances are added to the end of the list.

class PortFactoryList {
public:
  PortFactoryList();
  PortFactory* head() const;
  unsigned numFactories() const;
  PortFactory* lookup(PortType) const;
  void report() const;
  void add(PortFactory*);
};

Plugin software module interface PluginModule.hh

Each module's entry point is named rce_appmain; this symbol is recognized by the module build system which places its value in the transfer address slot of the module's ELF header. The prototype of the entry point function is

extern "C" PortFactory* rce_appmain();

The boot code uses class PluginModule to find then run or write the plugin software module. The first constructor fetches the module from some internal RCE storage while the second one reads it from a file. In either case one may then write the module to internal storage or run it to obtain the factory object. Once the module has been run any attempt to write it will throw std::logic_error because the module code will no longer be relocatable.

class PluginModule {
public:
  typedef PortFactory* (*EntryPoint)();
  explicit PluginModule(PortType);
  PluginModule(PortType, const char* filename);
  PortFactory* run();
  void write() const;
};

The module's entry point is run directly without creating a new thread (otherwise we'd need synchronization in order to wait for the factory to be produced).

Gen I-specific initialization

On Gen I RCEs the application software is left with the job of allocating I/O buffers with their corresponding Payload objects and it can't do that without knowing for each plugin the header sizes, max payload sizes and max number of frame buffers for import and export. That information is available from the port factories but it means downcasting the PortFactory* values gotten from the PortFactoryList. TDEs for the inbound buffers must be pushed into one or more FLBs before any data may be received. The interface described here lets the application do the needed initialization without exposing the innards of the plugin-handling system.

Ethernet (TBD)

PGP version 1

SetupPgp (SetupPgp.hh)

Creating an instance of this class allocates buffers and Payload objects in uncached memory. Buffers for inbound messages are shared amongst all PGP ports while each port has its own pool of buffers for outbound messages. The default is to allocate the maximum number of each kind of buffer and to use all PGP ports; only if you need to use fewer than the maximum do you need to specify the exact number. If you try to use more PGP ports than exist then std::logic_error is thrown. If you try to use more buffers than the maximum then the maximum is used.

The PIC blocks for the requested PGP ports will all be reset and (where needed) re-initialized. TDEs for all import buffers will be queued in the FLB that all PGP ports share.

The constructor will make a few log entries describing how PGP has been set up.

All buffers and Payload objects created by the constructor will be destroyed by the destructor.

class SetupPgp {
public:
  static const int ALL = -1;
  explicit SetupPgp(int numPorts=ALL, int numInboundBuffers=ALL, int numOutboundBuffersPerPort=ALL);
  ~SetupPgp();

  unsigned numPorts() const;
  unsigned numInboundBuffers() const;
  unsigned numOutboundBuffers(unsigned portNum) const;
};

Gen I-specific low-level API

Payload (Payload.hh)

On Gen I hardware the plugin firmware makes use of descriptors that reside in main memory. The allocation and preparation of message headers is the responsibility of the application; the firmware doesn't manage them. For the simplicity the descriptors, header, payload and links to other Payloads for a given buffer will all be welded together into a single Payload object in non-cached memory. Each part will have a fixed offset which is hard-coded into the Payload object, so that we don't have to refer to non-cached memory just to find out their addresses. For that reason we allocate a fixed number of bytes for the header no matter the type of plugin; the largest header is 32 bytes for the Petacache PGP non-register messages so we'll allocate twice that.

For details about the descriptors see chapter 4 of the Cluster Element Module document at http://www.slac.stanford.edu/exp/npa/design/CEM.pdf

struct Payload {

private:

  static const unsigned COMPLETION_DESCRIPTOR_SIZE = 2 * CACHE_LINE_SIZE;

  struct Completion {
    unsigned parameter: 24;
    unsigned wasBlockError: 1;
    unsigned reason: 6;
    unsigned wasError: 1;
    uint32_t  transferCount;
    uint8_t pad[COMPLETION_DESCRIPTOR_SIZE - offsetof(Completion,pad)];
  };

  struct Transaction {
    uint32_t  payloadLen;
    void*  headerPtr;
    void*  payloadPtr;
    Completion* completionPtr;
    Transaction(headerPtr, Completion *comp);
  };

  static const unsigned MAX_HEADER_SIZE = 64;
  static const unsigned TRANSACTION_DESCRIPTOR_ALIGNMENT = 64;
  static const unsigned COMPLETION_DESCRIPTOR_ALIGNMENT = CACHE_LINE_SIZE;
  static const unsigned DMA_ALIGNMENT = CACHE_LINE_SIZE;

  Transaction m_transaction              __attribute__((aligned(TRANSACTION_DESCRIPTOR_ALIGNMENT));
  Payload*    m_next;
  Payload*    m_prev;
  Completion  m_completion               __attribute__((aligned(COMPLETION_DESCRIPTOR_ALIGNMENT));
  uint8_t     m_header[MAX_HEADER_SIZE]  __attribute__((aligned(DMA_ALIGNMENT)));
  uint32_t    m_payload;                 __attribute__((aligned(DMA_ALIGNMENT)));

  Transaction* transaction() const {return &m_transaction;}
  Payload*     next()        const {return m_next;}
  Payload*     prev()        const {return m_prev;}
  Completion*  completion()  const {return &m_completion;}
  void*        header()      const {return &m_header;}
  void*        payload()     const {return &m_payload;}
};

Ethernet (TBD)

PGP version 1

Petacache

Plugin software module (PetaPgpPlugin.cc)

Class PetaPortFactory adds accessors to Gen-I specific information not found in the base class. The application interface class SetupPgp will use it.

class PetaPortFactory: public PortFactory {
public:
  unsigned importHeaderSize() const;
  unsigned exportHeaderSize() const;
  unsigned maxImportPayloadSize() const;
  unsigned maxExportPayloadSize() const;
  unsigned maxImportBuffers() const;
  unsigned maxExportBuffers() const;
  unsigned pebBlock(unsigned index) const;
  unsigned ecbBlock() const;
  unsigned flbBlock() const;
  unsigned pibBlock(unsigned index) const;
};

Gen I PGP1 plugins all share the same ECB and FLB but each is assigned its own PEB and PIB. The index you pass to the accessor function is the index within plugin type PGP1, the same number you would obtain from Port::typeIndex().

The Petacache PGP port factory makes instances of class PetaPort which at present only implements the accessors and the report function.

Gen I initialization of ConfigSpace

Configuration container zero in the configuration flash contains tables of information about the hardware and firmware; this information can't be gotten directly from the hardware and firmware.

Not knowing the actual layout of the RCE's circuit board(s) I've arbitrarily assigned conduit 0 to the 10 Gb ethernet and conduits 1, 2, 3 and (for petacache) 4 to PGP1. The four MGT's assigned to the ethernet I assign to pins 0-3 while the ones for PGP I assign to pins 4, 5, 6 and (peta) 7.

The container contents consist of eight instances of PluginConfig followed immediatly by eight instances of ConduitConfig, except that the "index" members are not recorded. For a Petacache RCE board (PGP only) the tables look like this: (substitute 0xffffffff for "EMPTY" and ):

The EMPTY value must be one that is is not valid for conversion to either PortType or ConduitType; 0xffffffff will do. Replace "PGP1" and "CONDUIT" with the numerical values of the corresponding enumerators.

Gen I storage of plugin software modules

Only a few different types of protocol plugins are found on Gen I systems so each type is assigned to a fixed Configuration container:

Later a container may be used for CONFIG_FLASH, if we ever manage to make a usable wrapper for the FCI package that makes it look like another plugin.

Use case: Gen I booting

1. Boot code:
   1. Loads and starts the system core.
2. System core
   1. Initializes the CPU.
   2. Initializes RTEMS.
   3. Initializes any extra C++ support.
   4. Sets up the MMU's TLB and enables the MMU.
   5. Creates the default instance of the dynamic linker.
   6. Reads ConfigSpace.
      1. The first read triggers the reading of Configuration container zero.
      2. Builds the Port and PortFactory lists, reading, linking and running plugin software modules as required.
   7. Performs other initialization, e.g., ethernet, BSD stack.
   8. Loads and links the application code using the default dynamic linker.
   9. Calls the application entry point.

Source code organization

All the classes, enums and other declarations will appear in the namespace RCE::ppi.

Platform-neutral header files for the application interface are immediately below the directory rce/ppi/.

Platform-neutral header files for the boot code and plugin software module interfaces are directly below rce/ppi/src/.

Platform-specific header files are in subdirectories gen1/, gen2/ and i86-linux/ below src/.

Platform-neutral compilation units are in src/ while those that depend on the platform are in src/gen1/, etc.

Inline definitions are broken out into their own header files in src/ or its subdirectories unless they are few and trivial for a given class. Example: Foo-inl.hh.

Unit test code goes in rce/ppi/test/ whose subdirectory structure mirrors that of src/.