Page History
Include Page | ||||
---|---|---|---|---|
|
Info |
---|
Draft, in work. |
Table of Contents |
---|
Include Page | ||||
---|---|---|---|---|
|
Major Xtc Library Classes
The most important data containers and associated classes in the Xtc library are listed below. (Inhereitance and collaboration trees are shown at right.)
Container Classes
- XtcData:Dgram, which is the root container, essentially an envelope, for other containers. By inheritance, Dgrams store information about LCLS machine state transitions and their associated pulseIDs and timestamps.
- XtcData:Xtc, the parent data container class for detector data and descriptive metadata. Each Dgram has one top-level Xtc container, which in turn contains other structures, including other Xtc/Xtc-derived containers.
Important Xtc Derived Classes and Utility Classes
- XtcData:TypeId is used to identify which flavor of Xtc container listed below is being worked with when recording or parsing.XtcData:Names, derived Xtc container for storing named metadata about data types and data dimensionality. Identified with an integer ID.
- XtcData:ShapesData, a derived convenience Xtc container with methods to access detector data.
- XtcData:Shapes, a derived Xtc container containing metadata about the structure of the raw detector data.
- XtcData:Data, a derived Xtc container for raw detector data itself.
Card | ||||||
---|---|---|---|---|---|---|
| ||||||
Card | ||||
---|---|---|---|---|
| ||||
The DAQ Side: Generating Xtc
For tutorial purposes, the following streamlined example shows XXX. For an example involving more detectors, formats and algorithms, see xtcdata/xtcdata/app/xtcwriter.cc.
A Notional DAQ Harness
Let's motivate the example. Assume we're developing a an Xtc writer class to output HSD data. This class will get plugged in to a (notional) DAQ framework.
Code Block | ||||
---|---|---|---|---|
| ||||
// %sample:intro:cpp%
// Assume some totally made-up data acquisition callback framework named DAQFramework
// Developers of the psdaq package are welcome to come in and change this toy example
// to match the real DAQ API.
HSDXtcWriter hsdwriter()
DAQFramework.registerHandler("configure", hsdwriter.setup)
DAQFramework.registerHandler("runstart", std::bind(hsdwriter.openXtcFile, filename))
// Assume we get a "readout" message per event
DAQFramework.registerHandler("readout", hsdwriter.writeFeature)
DAQFramework.registerHandler("runend", hsdwriter.closeXtcFile)
//%endsample% |
"setup" Method: Define Data Signature and Create Names Structure
Deck of Cards | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
|
"writeFeature" Method: Respond to Readout Events and Add Readout Data
id | writefeature |
---|
default | true |
---|---|
label | Coding Steps |
(Introduce XtcData:CreateData and its set_value (assign values explicitly) and allocate (set aside structured memory for the data then fill in the structure with values) methods here. With FexDef, we're positing that the data coming out has a well-defined shape, i.e. definable as a record, so per-event, you will update and emit this data record into the output stream)
Another list of steps/big ideas this time about appending data.
- For the array data
- CreateData.allocate() returns an array Template of whatever datatype you define, arrayT, which you then populate with data.
- CreateData.allocate() calls set_array_shape() internally.
- XtcData:NamesId, utility class that generates a composite ID, somewhat like a relational database composite key, combining the integer nodeID and Xtc:Names integer ID.
- XtcData:NamesVec, utility class for storing a set of XtcData:Names.
- XtcData:TypeId is used to identify which flavor of Xtc container (Parent, Names, ShapesData, Shapes, Data, etc.) is being worked with when recording or parsing.
Deck of Cards | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
|
The DAQ Side: Generating Xtc
For tutorial purposes, the following streamlined example shows how to use the Xtc library to develop a data writer and data reader for a fictitious but real-world HSD outputting feature extraction data. For an example involving more detectors, formats and algorithms, see xtcdata/xtcdata/app/xtcwriter.cc.
A Notional DAQ Harness
Let's motivate the example. Assume we're developing a an Xtc writer class to output HSD data. This class will get plugged in to a (notional) DAQ framework.
Code Block | ||||
---|---|---|---|---|
| ||||
// %sample:intro:cpp%
// Assume some totally made-up data acquisition callback framework named DAQFramework
// Developers of the psdaq package are welcome to come in and change this toy example
// to match the real DAQ API.
HSDXtcWriter hsdwriter()
DAQFramework.registerHandler("configure", std::bind(hsdwriter.openXtcFile, filename))
DAQFramework.registerHandler("runstart", hsdwriter.setup)
// Assume we get a "readout" message per event
DAQFramework.registerHandler("readout", hsdwriter.writeFeature)
DAQFramework.registerHandler("runend", hsdwriter.closeXtcFile)
//%endsample% |
"setup" Method: Define Data Signature and Create Names Structure
Deck of Cards | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
|
"writeFeature" Method: Respond to Readout Events and Add Readout Data
Deck of Cards | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
|
The Pipeline Side: Parsing Whole Xtc Files, Using Small Data Files
xtcreader.cc and XtcIterator.hh
A Notional Offline Pipeline Harness
Notes for Real-World Code
Realistic Xtc Writers
(Bullet lists of gotchas and recommendations)
- Thread safety issue #1
- Thread safety issue #2
- (You'll likely need to sub-class from XYZ in order to ABC)
Realistic Xtc Readers
(Bullet lists of gotchas and recommendations)
- Thread safety issue #1
- Thread safety issue #2
- (You'll likely need to sub-class from XYZ in order to ABC)
Full "Hello Xtc" Code Listings
HSDXtcWriter Example
Code Block | ||||
---|---|---|---|---|
| ||||
// %sample:codesample1:cpp%
#include "xtcdata/xtc/ShapesData.hh"
#include "xtcdata/xtc/DescData.hh"
#include "xtcdata/xtc/Dgram.hh"
#include "xtcdata/xtc/TypeId.hh"
#include "xtcdata/xtc/XtcIterator.hh"
#include "xtcdata/xtc/VarDef.hh"
// various std library includes omitted...
using namespace XtcData;
#define BUFSIZE 0x4000000
HSDXtcWriter::HSDXtcWriter()
{
// Define the "signature" of the data in
// our feature extraction data set
class FexDef:public VarDef
{
public:
enum index
{
floatFex,
arrayFex,
intFex
};
FexDef()
{
NameVec.push_back({"floatFex",Name::DOUBLE});
// "arrayFex" has rank 2; 0 is the default, for scalar data
NameVec.push_back({"arrayFex",Name::FLOAT,2});
NameVec.push_back({"intFex",Name::INT64});
}
};
// Define some instance variables.
FILE *this->xtcoutfile;
// -------- Set up the metadata ------------
// Declare an Xtc::Dgram to store metadata about the output
Dgram& this->metadataDgram;
// Declare a NamesVec to store Names info
NamesVec this->namesVec;
// Declare a NamesId object to act as composite lookup key
NamesId this->namesId;
// Store the node ID of this node
unsigned this->nodeid = 1;
// -------- Set up the readout data ---------
// Declare an Xtc:Dgram to store readout data
Dgram& this->readoutDgram;
// Create one of our previously-defined FexDefs
this->FexDef = new FexDef();
// Now initialize various things
_initializeMetadataDgram();
}
void HSDXtcWriter::_initializeMetadataDgram()
{
TypeId tid(TypeId::Parent, 0);
void* configbuf = malloc(BUFSIZE);
this->metadataDgram = *(Dgram*)configbuf;
this->metadataDgram.xtc.contains = tid;
this->metadataDgram.xtc.damage = 0;
this->metadataDgram.xtc.extent = sizeof(Xtc |
Card | ||||
---|---|---|---|---|
| ||||
|
The Pipeline Side: Parsing Whole Xtc Files, Using Small Data Files
xtcreader.cc and XtcIterator.hh
A Notional Offline Pipeline Harness
Notes for Real-World Code
Realistic Xtc Writers
(Bullet lists of gotchas and recommendations)
- Thread safety issue #1
- Thread safety issue #2
- (You'll likely need to sub-class from XYZ in order to ABC)
Realistic Xtc Readers
(Bullet lists of gotchas and recommendations)
- Thread safety issue #1
- Thread safety issue #2
- (You'll likely need to sub-class from XYZ in order to ABC)
Full "Hello Xtc" Code Listings
HSDXtcWriter Example
Code Block | ||||
---|---|---|---|---|
| ||||
// %sample:codesample1:cpp% #include "xtcdata/xtc/ShapesData.hh" #include "xtcdata/xtc/DescData.hh" #include "xtcdata/xtc/Dgram.hh" #include "xtcdata/xtc/TypeId.hh" #include "xtcdata/xtc/XtcIterator.hh" #include "xtcdata/xtc/VarDef.hh" using namespace XtcData; #define BUFSIZE 0x4000000 HSDXtcWriter::HSDXtcWriter() { // Define the "signature" of the data in // our feature extraction data set class FexDef:public VarDef { public: enum index { floatFex, arrayFex, intFex }; FexDef() { NameVec.push_back({"floatFex",Name::DOUBLE}); // "arrayFex" has rank 2 NameVec.push_back({"arrayFex",Name::FLOAT,2}); NameVec.push_back({"intFex",Name::INT64}); } }; // Define some instance variables. FILE *this->xtcoutfile; // Declare an Xtc::Dgram to store metadata about the output Dgram& this->metadataDgram; // Declare an Xtc:Dgram to store readout data Dgram& this->readoutDgram; std::vector<NameIndex>& this->namesVec; this->FexDef = new FexDef(); // Now initialize various things _initializeMetadataDgram(); } void HSDXtcWriter::_initializeMetadataDgraminitializeReadoutDataDgram() { TypeId tid(TypeId::Parent, 0); void* configbufbuf = malloc(BUFSIZE); this->metadataDgram>readoutDgram = *(Dgram*)configbufbuf; this->metadataDgram>readoutDgram.xtc.contains = tid; this->metadataDgram>readoutDgram.xtc.damage = 0; this->metadataDgram>readoutDgram.xtc.extent = sizeof(Xtc); } void HSDXtcWriter::_initializeReadoutDataDgramaddNames() { TypeId tid(TypeId::Parent, 0); void* buf = malloc(BUFSIZE); this->readoutDgram = *(Dgram*)buf; this->readoutDgram.xtc.contains = tid; this->readoutDgram.xtc.damage = 0; this->readoutDgram.xtc.extent = sizeof(Xtc); } void HSDXtcWriter::_addNames(Xtc& parent, std::vector<NameIndex>& aNamesVec) { // Instantiate an Alg to define metadata for // a feature extraction ("fex") algorithm, // version 4.5.6 Alg hsdFexAlg("fex",4,5,6); // Create a Names structure that identifies// Instantiate an Alg to define metadata for // a feature extraction ("fex") algorithm, // version 4.5.6 Alg hsdFexAlg("fex",4,5,6); // Create a NamesId composite lookup key and // then a Names structure (ID'd with that key) // that identifies the location, algorithm, // detector type and detectorID. this->namesId.NamesId(this->nodeId,1); // nodeId=1, namesId = 1 Names& fexNames = *new(xtc) Names("xpphsd", hsdFexAlg, "hsd", "detnum1234", this->namesId) // And now we add the Names structure to the // parent Xtc metadata container along with // the location, algorithm, detector type // and detectorID. Names& fexNames = *new(parent) Names("xpphsd", hsdFexAlg, "hsd", "detnum1234") // And now we add the Names structure to the // parent Xtc metadata container along with // the FexDef data signature. fexNames.add(parent, this->FexDef); // add this Names structure to the vector aNamesVec.push_back(NameIndex(this->fexNames)); } void HSDXtcWriter::setup() { // Add a Names structure to the configuration Dgram _addNames(this->metadataDgram.xtc, this->namesVec);FexDef data signature. fexNames.add(this->metadataDgram.xtc, this->FexDef); // add this Names structure to the NamesVec this->namesVec[namesId1] = NameIndex(this->fexNames); } void HSDXtcWriter::openXtcFile(xtcoutfilename) { this->xtcoutfile = fopen(xtcoutfilename, "w"); } void HSDXtcWriter::setup() { // Behind this public method, call internal to // set up the metadata this->_addNames(); // Omitting code to confirm successful file operation.... fwrite(&this->metadataDgram, sizeof(this->metadataDgram) + this->metadataDgram.xtc.sizeofPayload(), 1, this->xtcoutfile) } void HSDXtcWriter::openXtcFilewriteFeature(xtcoutfilename) { this->xtcoutfile = fopen(xtcoutfilename, "w"); } void HSDXtcWriter::writeFeature() { // Initialize the Dgram for readout data this->initializeReadoutDataDgram(); // Need some makebelieve code here to show how this // callback method unpacks data from DAQ // to cram into these Xtc structures... // Talk to Chris. Is this event by event? Would you want // to destruct this CreateData after each event? CreateData fex(parentthis->readoutDgram.xtc, NamesVecthis->namesVec, nameIdthis->namesId); // Need stub code here to represent getting values from DAQ // and using set_value and allocate with something real-ish // set_value() lets you set scalar values in the data record. // The 'floatFex' ID is defined in our custom FexDef class. fex.set_value(FexDef::floatFex, (double)41.0); // Use allocate() to set up data that's better represented as // vectors or matrices. // The 'arrayFex' ID is defined in our custom FexDef class. unsigned shape[MaxRank] = {2,3}; //MaxRank is an Xtc library //global upper limit on data complexity. Array<float> arrayT = fex.allocate<float>(this->FexDef::arrayFex, shape); for(unsigned i=0; i<shape[0]; i++){ for (unsigned j=0; j<shape[1]; j++) { arrayT(i,j) = 142.0+i*shape[1]+j; } }; // Another scalar value setter. // The 'intFex' ID is defined in our custom FexDef class. fex.set_value(FexDef::intFex, (int64_t) 42); } void HSDXtcWriter::closeXtcFile() { fclose(this->xtcoutfile); } // ... // %endsample% |
HSDXtcReader Example
Code Block | ||||
---|---|---|---|---|
| ||||
// %sample:codesample2:cpp% cout<<"Hi world?" // %endsample% |