You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 439 Next »

Draft, in work.

Topics

Go back to the Top of this tree.

Major Xtc Library Classes

The most important data containers and associated classes in the Xtc library are listed below. (Inhereitance and collaboration trees are shown at right.)

Container Classes

  • XtcData:Dgram, which is the root container, essentially an envelope, for other containers. By inheritance, Dgrams store information about LCLS machine state transitions and their associated pulseIDs and timestamps.
  • XtcData:Xtc, the parent data container class for detector data and descriptive metadata. Each Dgram has one top-level Xtc container, which in turn contains other structures, including other Xtc/Xtc-derived containers.

Important Xtc Derived Classes

  • XtcData:TypeId is used to identify which flavor of Xtc container listed below is being worked with when recording or parsing.
  • XtcData:Names, derived Xtc container for storing named metadata about data types and data dimensionality.
  • XtcData:ShapesData, a derived convenience Xtc container with methods to access detector data.
    • XtcData:Shapes, a derived Xtc container containing metadata about the structure of the raw detector data.
    • XtcData:Data, a derived Xtc container for raw detector data itself.

    The DAQ Side: Generating Xtc

    For tutorial purposes, the following streamlined example shows XXX. For an example involving more detectors, formats and algorithms, see xtcdata/xtcdata/app/xtcwriter.cc.

    A Notional DAQ Harness

    Let's motivate the example. Assume we're developing a an Xtc writer class to output HSD data. This class will get plugged in to a (notional) DAQ framework.

    // %sample:intro:cpp%
    // Assume some totally made-up data acquisition callback framework named DAQFramework
    // Developers of the psdaq package are welcome to come in and change this toy example
    // to match the real DAQ API.
    
    HSDXtcWriter hsdwriter()
    
    DAQFramework.registerHandler("configure", hsdwriter.setup)
    DAQFramework.registerHandler("runstart", std::bind(hsdwriter.openXtcFile, filename))
    // Assume we get a "readout" message per event
    DAQFramework.registerHandler("readout", hsdwriter.writeFeature)
    DAQFramework.registerHandler("runend", hsdwriter.closeXtcFile)
    
    //%endsample%

    "setup" Method: Specify the Data Signature and Set Up The Names Structure

      To set things up, we create a vector of Xtc:Names, in which we register the algorithms and "shape" of the data associated with detector elements so the processing pipeline knows what to do with it.

      1. Set up two Xtc:Dgrams, the first to store metadata, the second to be used and re-used as readout events occur to store data:
        1. For metadata:
          1. Declare an XtcData:Dgram as the root container for metadata (see %codesample1:14,15,38-39,47:setupexamples%). 
          2. And call a utility method to actually create and set up this metadata Dgram (see %codesample1:14,15,38,39,44,45,46-58:setupexamples%).
        2. And analagously for readout data: 
          1. Declare an XtcData:Dgram as the container for readout data (see %codesample1:14,15,40-41,47:setupexamples%). 
          2. And create the Dgram (see %codesample1:14,15,40-41,59-69:setupexamples%).
      2. Subclass from Xtc:VarDef to define the signature of the feature extraction (fex) data we're going to record (see %codesample1:14-15,18-35,43,47:setupexamples%).
        1. "FexData" consists of three categories of data, a float (double) value, and array (to be defined later), and an integer (64 bit) value.
      3. Then some plumbing for Xtc:Names.:
        1. Set aside a vector to store some XtcData:Names we use to annotate this data (see %codesample1:14,15,42,47:setupexamples%).
        2. In the setup method itself, call an internal method to actually populate thie Xtc:Names structure. See %codesample1:89-94,71-87:setupexamples%.  
          1. We create an Xtc:Alg to record a specific feature extraction algorithm name, "fex", and version, 4.5.6, needed to process this data.
      • XtcData::Dgram is your root container for metadata (Names and Shapes) as well as the data itself
      • Sub-class from XtcData::VarDef to define data shapes
        • VarDef gives you a NameVec with push_back() method to xxxx

       

       

      "writeFeature" Method: Respond to Readout Events and Add Readout ShapesData and Data

        (Introduce CreateData and its set_value (assign values explicitly) and allocate (set aside structured memory for the data then full in the structure with values) methods here. What about set_array_shape? With FexDef, we're positing that the data coming out has a well-defined shape, i.e. definable as a record, so per-event, you will update and emit this data record into the output stream)

        Another list of steps/big ideas this time about appending data.

        1. .
        2. A chunk of code summary.

         

        • Use CreateData for XYZ applications:
          • set_value() for scalar data
          • allocate() for data vectors or matrices
        • Use DescribedData for XYZ applications

        The Pipeline Side: Parsing Whole Xtc Files, Using Small Data Files

        xtcreader.cc and XtcIterator.hh

        A Notional Offline Pipeline Harness


        Notes for Real-World Code

        Realistic Xtc Writers

        (Bullet lists of gotchas and recommendations)

        • Thread safety issue #1
        • Thread safety issue #2
        • (You'll likely need to sub-class from XYZ in order to ABC)

        Realistic Xtc Readers

        (Bullet lists of gotchas and recommendations)

        • Thread safety issue #1
        • Thread safety issue #2
        • (You'll likely need to sub-class from XYZ in order to ABC)

        Full "Hello Xtc" Code Listings

        HSDXtcWriter Example

        // %sample:codesample1:cpp%
        
        #include "xtcdata/xtc/ShapesData.hh"
        #include "xtcdata/xtc/DescData.hh"
        #include "xtcdata/xtc/Dgram.hh"
        #include "xtcdata/xtc/TypeId.hh"
        #include "xtcdata/xtc/XtcIterator.hh"
        #include "xtcdata/xtc/VarDef.hh"
        
        using namespace XtcData;
        
        #define BUFSIZE 0x4000000
        
        HSDXtcWriter::HSDXtcWriter()
        {
        	// Define the "signature" of the data in
            // our feature extraction data set
        	class FexDef:public VarDef
        	{
        		public:
        		  enum index
        		  {
        			  floatFex,
        		      arrayFex,
        		      intFex
        	      };
        
        		FexDef()
           		{
               		NameVec.push_back({"floatFex",Name::DOUBLE});
        			NameVec.push_back({"arrayFex",Name::FLOAT,2});
        			NameVec.push_back({"intFex",Name::INT64});
        		}
        	};
        
        	// Define some instance variables.
        	FILE *this->xtcoutfile;
        	// Declare an Xtc::Dgram to store metadata about the output
        	Dgram& this->metadataDgram;
        	// Declare an Xtc:Dgram to store readout data
        	Dgram& this->readoutDgram;
        	std::vector<NameIndex>& this->namesVec;
        	this->FexDef = new FexDef();
        
        	// Now initialize various things
        	_initializeMetadataDgram();
        }
        
        void HSDXtcWriter::_initializeMetadataDgram()
        {
        	TypeId tid(TypeId::Parent, 0);
        	void* configbuf = malloc(BUFSIZE);
        
            this->metadataDgram = *(Dgram*)configbuf;
        	this->metadataDgram.xtc.contains = tid;
            this->metadataDgram.xtc.damage = 0;
            this->metadataDgram.xtc.extent = sizeof(Xtc);
        }
        
        void HSDXtcWriter::_initializeReadoutDataDgram()
        {
        	TypeId tid(TypeId::Parent, 0);
        	void* buf = malloc(BUFSIZE);
        
            this->readoutDgram = *(Dgram*)buf;
        	this->readoutDgram.xtc.contains = tid;
            this->readoutDgram.xtc.damage = 0;
            this->readoutDgram.xtc.extent = sizeof(Xtc);
        }
        
        void HSDXtcWriter::_addNames(Xtc& parent, std::vector<NameIndex>& aNamesVec)
        {
        	// Instantiate an Alg to define metadata for
            // a feature extraction ("fex") algorithm,
        	// version 4.5.6
        	Alg hsdFexAlg("fex",4,5,6);
        	// Create a Names structure that identifies
            // the location, algorithm, detector type
            // and detectorID.
        	Names& fexNames = *new(parent) Names("xpphsd", hsdFexAlg, "hsd", "detnum1234")
        	// And now we add the Names structure to the
            // parent Xtc metadata container along with
            // the FexDef data signature.
        	fexNames.add(parent, this->FexDef);
        	// add this Names structure to the vector
        	aNamesVec.push_back(NameIndex(this->fexNames));
        }
        
        void HSDXtcWriter::setup()
        {
        	// Add a Names structure to the configuration Dgram
        	_addNames(this->metadataDgram.xtc, this->namesVec);
        }
        
        void HSDXtcWriter::openXtcFile(xtcoutfilename)
        {
        	this->xtcoutfile = fopen(xtcoutfilename, "w");
        }
        
        void HSDXtcWriter::writeFeature()
        {
        
        	// Need some makebelieve code here to show how this
            // callback method unpacks data from DAQ
        	// to cram into these Xtc structures...
        
        	// Talk to Chris. Is this event by event? Would you want
            // to destruct this CreateData after each event?
         	CreateData fex(parent, NamesVec, nameId);
        
        	// Need stub code here to represent getting values from DAQ
        	// and using set_value and allocate with something real-ish
        
        	// set_value() lets you set scalar values in the data record.
            // The 'floatFex' ID is defined in our custom FexDef class.
        	fex.set_value(FexDef::floatFex, (double)41.0);
        
        	// Use allocate() to set up data that's better represented as
            // vectors or matrices.
            // The 'arrayFex' ID is defined in our custom FexDef class.
        	unsigned shape[MaxRank] = {2,3}; //MaxRank is an Xtc library
                                             //global upper limit on data complexity.
            Array<float> arrayT = fex.allocate<float>(this->FexDef::arrayFex, shape);
            for(unsigned i=0; i<shape[0]; i++){
                for (unsigned j=0; j<shape[1]; j++) {
                    arrayT(i,j) = 142.0+i*shape[1]+j;
                }
            };
        	
        	// Another scalar value setter.
            // The 'intFex' ID is defined in our custom FexDef class.
        	fex.set_value(FexDef::intFex, (int64_t) 42);
        
        }
        
        void HSDXtcWriter::closeXtcFile()
        {
        	fclose(this->xtcoutfile);
        }
        
        // ...
        // %endsample%

        HSDXtcReader Example

        // %sample:codesample2:cpp%
        	cout<<"Hi world?"
        // %endsample% 

         

        • No labels