Page History
Include Page | ||||
---|---|---|---|---|
|
Info |
---|
Draft, in work. |
Table of Contents |
---|
Include Page | ||||
---|---|---|---|---|
|
Major Xtc Library Classes
The most important data containers and associated classes in the Xtc library are listed below. (The list is indented to show a hierarchy of function, not a class hierarchy. Inheritance trees and collaboration. Inhereitance and collaboration trees are shown at right.)
Container Classes
- XtcData:Dgram, which is the root container, essentially an envelope, for other containers. By inheritance, Dgrams store information about LCLS machine state transitions and their associated pulseIDs and timestamps.
- XtcData:Xtc, the parent data container class for detector data and descriptive metadata. Each Dgram has one top-level Xtc container, which in turn contains other structures, including other Xtc/Xtc-derived containers.
Important Xtc Derived Classes and Utility Classes
- XtcData:Names, an Xtc container.
- XtcData:TypeId is used to identify which flavor of Xtc container listed below is being worked with when recording or parsing. XtcData:Names, derived Xtc container for storing named metadata about data types and data dimensionality. Identified with an integer ID.
- XtcData:ShapesData, a derived convenience Xtc container with methods to access detector data.
- XtcData:Shapes, a derived Xtc container containing metadata about the structure of the raw detector data.
- XtcData:Data, a derived Xtc container for raw detector data itself.
id | setup |
---|
Card | ||||
---|---|---|---|---|
| ||||
The DAQ Side: Generating Xtc
For tutorial purposes, the following streamlined example shows XXX. For an example involving more detectors, formats and algorithms, see xtcdata/xtcdata/app/xtcwriter.cc.
A Notional DAQ Harness
Let's motivate the example. Assume we're developing a an Xtc writer class to output HSD data. This class will get plugged in to a (notional) DAQ framework.
Code Block | ||||
---|---|---|---|---|
| ||||
// %sample:intro:cpp%
// Assume some totally made-up data acquisition callback framework named DAQFramework
// Developers of the psdaq package are welcome to come in and change this toy example
// to match the real DAQ API.
HSDXtcWriter hsdwriter()
DAQFramework.registerHandler("configure", hsdwriter.setup)
DAQFramework.registerHandler("runstart", std::bind(hsdwriter.openXtcFile, filename))
// Assume we get a "readout" message per event
DAQFramework.registerHandler("readout", hsdwriter.writeFeature)
DAQFramework.registerHandler("runend", hsdwriter.closeXtcFile)
//%endsample% |
"setup" Method: Set Up The Names Structure
Deck of Cards | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
|
"writeFeature" Method: Respond to Readout Events and Add Readout ShapesData and Data
id | writefeature |
---|
- XtcData:NamesId, utility class that generates a composite ID, somewhat like a relational database composite key, combining the integer nodeID and Xtc:Names integer ID.
- XtcData:NamesVec, utility class for storing a set of XtcData:Names.
- XtcData:TypeId is used to identify which flavor of Xtc container (Parent, Names, ShapesData, Shapes, Data, etc.) is being worked with when recording or parsing.
Deck of Cards | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
|
The DAQ Side: Generating Xtc
For tutorial purposes, the following streamlined example shows how to use the Xtc library to develop a data writer and data reader for a fictitious but real-world HSD outputting feature extraction data. For an example involving more detectors, formats and algorithms, see xtcdata/xtcdata/app/xtcwriter.cc.
A Notional DAQ Harness
Let's motivate the example. Assume we're developing a an Xtc writer class to output HSD data. This class will get plugged in to a (notional) DAQ framework.
Code Block | ||||
---|---|---|---|---|
| ||||
// %sample:intro:cpp%
// Assume some totally made-up data acquisition callback framework named DAQFramework
// Developers of the psdaq package are welcome to come in and change this toy example
// to match the real DAQ API.
HSDXtcWriter hsdwriter()
DAQFramework.registerHandler("configure", std::bind(hsdwriter.openXtcFile, filename))
DAQFramework.registerHandler("runstart", hsdwriter.setup)
// Assume we get a "readout" message per event
DAQFramework.registerHandler("readout", hsdwriter.writeFeature)
DAQFramework.registerHandler("runend", hsdwriter.closeXtcFile)
//%endsample% |
"setup" Method: Define Data Signature and Create Names Structure
Deck of Cards | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
|
"writeFeature" Method: Respond to Readout Events and Add Readout Data
Deck of Cards | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||
|
The Pipeline Side: Parsing Whole Xtc Files, Using Small Data Files
xtcreader.cc and XtcIterator.hh
A Notional Offline Pipeline Harness
Notes for Real-World Code
Realistic Xtc Writers
(Bullet lists of gotchas and recommendations)
- Thread safety issue #1
- Thread safety issue #2
- (You'll likely need to sub-class from XYZ in order to ABC)
Realistic Xtc Readers
(Bullet lists of gotchas and recommendations)
- Thread safety issue #1
- Thread safety issue #2
- (You'll likely need to sub-class from XYZ in order to ABC)
Full "Hello Xtc" Code Listings
HSDXtcWriter Example
Code Block | ||||
---|---|---|---|---|
| ||||
// %sample:codesample1:cpp%
#include "xtcdata/xtc/ShapesData.hh"
#include "xtcdata/xtc/DescData.hh"
#include "xtcdata/xtc/Dgram.hh"
#include "xtcdata/xtc/TypeId.hh"
#include "xtcdata/xtc/XtcIterator.hh"
#include "xtcdata/xtc/VarDef.hh"
// various std library includes omitted...
using namespace XtcData;
#define BUFSIZE 0x4000000
HSDXtcWriter::HSDXtcWriter()
{
// Define the "signature" of the data in
// our feature extraction data set
class FexDef:public VarDef
{
public:
enum index
{
floatFex,
arrayFex,
intFex
};
FexDef()
{
NameVec.push_back({"floatFex",Name::DOUBLE});
// "arrayFex" has rank 2; 0 is the default, for scalar data
NameVec.push_back({"arrayFex",Name::FLOAT,2});
NameVec.push_back({"intFex",Name::INT64});
}
};
// Define some instance variables.
FILE *this->xtcoutfile;
// -------- Set up the metadata ------------
// Declare an Xtc::Dgram to store metadata about the output
Dgram& this->metadataDgram;
// Declare a NamesVec to store Names info
NamesVec this->namesVec;
// Declare a NamesId object to act as composite lookup key
NamesId this->namesId;
// Store the node ID of this node
unsigned this->nodeid = 1;
// -------- Set up the readout data ---------
// Declare an Xtc:Dgram to store readout data
Dgram& this->readoutDgram;
// Create one of our previously-defined FexDefs
this->FexDef = new FexDef();
// Now initialize various things
_initializeMetadataDgram();
}
void HSDXtcWriter::_initializeMetadataDgram()
{
TypeId tid(TypeId::Parent, 0);
void* configbuf = malloc(BUFSIZE);
this->metadataDgram = *(Dgram*)configbuf;
this->metadataDgram.xtc.contains = tid;
this->metadataDgram.xtc.damage = 0;
this->metadataDgram.xtc.extent = sizeof(Xtc |
default | true |
---|---|
label | Coding Steps |
(Introduce CreateData and its set_value (assign values explicitly) and allocate (set aside structured memory for the data then full in the structure with values) methods here. What about set_array_shape? With FexDef, we're positing that the data coming out has a well-defined shape, i.e. definable as a record, so per-event, you will update and emit this data record into the output stream)
Another list of steps/big ideas this time about appending data.
- .
- A chunk of code summary.
Card | ||||
---|---|---|---|---|
| ||||
|
The Pipeline Side: Parsing Whole Xtc Files, Using Small Data Files
xtcreader.cc and XtcIterator.hh
A Notional Offline Pipeline Harness
Notes for Real-World Code
Realistic Xtc Writers
(Bullet lists of gotchas and recommendations)
- Thread safety issue #1
- Thread safety issue #2
- (You'll likely need to sub-class from XYZ in order to ABC)
Realistic Xtc Readers
(Bullet lists of gotchas and recommendations)
- Thread safety issue #1
- Thread safety issue #2
- (You'll likely need to sub-class from XYZ in order to ABC)
Full "Hello Xtc" Code Listings
HSDXtcWriter Example
Code Block | ||||
---|---|---|---|---|
| ||||
// %sample:codesample1:cpp% #include "xtcdata/xtc/ShapesData.hh" #include "xtcdata/xtc/DescData.hh" #include "xtcdata/xtc/Dgram.hh" #include "xtcdata/xtc/TypeId.hh" #include "xtcdata/xtc/XtcIterator.hh" #include "xtcdata/xtc/VarDef.hh" using namespace XtcData; #define BUFSIZE 0x4000000 HSDXtcWriter::HSDXtcWriter() { // Define the "signature" of the data in // our feature extraction data set class FexDef:public VarDef { public: enum index { floatFex, arrayFex, intFex }; FexDef() { NameVec.push_back({"floatFex",Name::DOUBLE}); NameVec.push_back({"arrayFex",Name::FLOAT,2}); NameVec.push_back({"intFex",Name::INT64}); } }; // Define some instance variables. FILE *this->xtcoutfile; // Declare an Xtc::Dgram to store metadata about the output Dgram& this->metadataDgram; // Declare an Xtc:Dgram to store readout data Dgram& this->readoutDgram; std::vector<NameIndex>& this->namesVec; this->FexDef = new FexDef(); // Now initialize various things _initializeMetadataDgram(); } void HSDXtcWriter::_initializeMetadataDgraminitializeReadoutDataDgram() { TypeId tid(TypeId::Parent, 0); void* configbufbuf = malloc(BUFSIZE); this->metadataDgram>readoutDgram = *(Dgram*)configbufbuf; this->metadataDgram>readoutDgram.xtc.contains = tid; this->metadataDgram>readoutDgram.xtc.damage = 0; this->metadataDgram>readoutDgram.xtc.extent = sizeof(Xtc); } void HSDXtcWriter::_initializeReadoutDataDgramaddNames() { TypeId tid(TypeId::Parent, 0); void* buf = malloc(BUFSIZE); this->readoutDgram = *(Dgram*)buf; this->readoutDgram.xtc.contains = tid; this->readoutDgram.xtc.damage = 0; this->readoutDgram.xtc.extent = sizeof(Xtc); } void HSDXtcWriter::_addNames(Xtc& parent, std::vector<NameIndex>& aNamesVec) { // Instantiate an Alg to define metadata for // a feature extraction ("fex") algorithm Alg hsdFexAlg("fex",4,5,6); // Create a Names structure that identifies// Instantiate an Alg to define metadata for // a feature extraction ("fex") algorithm, // version 4.5.6 Alg hsdFexAlg("fex",4,5,6); // Create a NamesId composite lookup key and // then a Names structure (ID'd with that key) // that identifies the location, algorithm, // detector type and detectorID. this->namesId.NamesId(this->nodeId,1); // nodeId=1, namesId = 1 Names& fexNames = *new(xtc) Names("xpphsd", hsdFexAlg, "hsd", "detnum1234", this->namesId) // And now we add the Names structure to the // parent Xtc metadata container along with // the location, algorithm, detector type // and detectorID. Names& fexNames = *new(parent) Names("xpphsd", hsdFexAlg, "hsd", "detnum1234") // And now we add the Names structure to the // parent Xtc metadata container along with // the FexDef data signature. fexNames.add(parent, this->FexDef); aNamesVec.push_back(NameIndex(this->fexNames)); } void HSDXtcWriter::setup() { // Add a Names structure to the configuration Dgram _addNames(this->metadataDgram.xtc, this->namesVec); } void HSDXtcWriter::openXtcFile(xtcoutfilename) { this->xtcoutfile = fopen(xtcoutfilename, "w");FexDef data signature. fexNames.add(this->metadataDgram.xtc, this->FexDef); // add this Names structure to the NamesVec this->namesVec[namesId1] = NameIndex(this->fexNames); } void HSDXtcWriter::openXtcFile(xtcoutfilename) { this->xtcoutfile = fopen(xtcoutfilename, "w"); } void HSDXtcWriter::setup() { // Behind this public method, call internal to // set up the metadata this->_addNames(); // Omitting code to confirm successful file operation.... fwrite(&this->metadataDgram, sizeof(this->metadataDgram) + this->metadataDgram.xtc.sizeofPayload(), 1, this->xtcoutfile) } void HSDXtcWriter::writeFeature() { // Initialize the Dgram for readout data this->initializeReadoutDataDgram(); // Need some makebelieve code here to show how this // callback method unpacks data from DAQ // to cram into these Xtc structures... // Talk to Chris. Is this event by event? Would you want // to destruct this CreateData after each event? CreateData fex(parentthis->readoutDgram.xtc, NamesVecthis->namesVec, nameIdthis->namesId); // Need stub code here to represent getting values from DAQ // and using set_value and allocate with something real-ish // set_value() lets you set scalar values in the data record. // The 'floatFex' ID is defined in our custom FexDef class. fex.set_value(FexDef::floatFex, (double)41.0); // Use allocate() to set up data that's better represented as // vectors or matrices. // The 'arrayFex' ID is defined in our custom FexDef class. unsigned shape[MaxRank] = {2,3}; //MaxRank is an Xtc library //global upper limit on data complexity. Array<float> arrayT = fex.allocate<float>(this->FexDef::arrayFex, shape); for(unsigned i=0; i<shape[0]; i++){ for (unsigned j=0; j<shape[1]; j++) { arrayT(i,j) = 142.0+i*shape[1]+j; } }; // Another scalar value setter. // The 'intFex' ID is defined in our custom FexDef class. fex.set_value(FexDef::intFex, (int64_t) 42); } void HSDXtcWriter::closeXtcFile() { fclose(this->xtcoutfile); } // ... // %endsample% |
HSDXtcReader Example
Code Block | ||||
---|---|---|---|---|
| ||||
// %sample:codesample2:cpp% cout<<"Hi world?" // %endsample% |