Presently there are two xtc to hdf5 translators, o2o-translate and psana-translate. o2o-translate is the original translator. It is being phased out of use and replaced by psana-translate. Translation is primarily carried out by automatic hdf5 translation that users can execute from the web portal. Documentation on o2o-translate, which discusses some history with regards to selecting hdf5 for a scientific data format for general use can be found

Translator
User Interface to Translator

Automatic translation of xtc to hdf5 is managed by the Interface Controlller.

A discussion of the hdf5 output format for translator can be found here:

The contents of HDF5 files - the output format. Users that will work with hdf5 output should review this document. Some key points are:

• Datasets need not be aligned.  That is the 5th image in a detector dataset may come from a different event than the 5th record in a gas detector dataset. One can match up records from different datasets by use the time datasets.
• One should use the _mask datasets to identify valid data. A _mask dataset record is 1 when the corresponding record of the data dataset if valid, 0 if it is not. When the _mask record is 0, the data record will be all zeros and should not be processed. The mask is 0 when the xtc data is damaged. The type damaged data can then be found in the _damage dataset. The main reason to record damaged data is to keep datasets as aligned as possible.
• The hdf5 group hierarchy has the following levels: run, calib cycle. type, source  - regular event data is organized into datasets that live at the source level. Epics is special, rather then the two groups type and source, there are three groups for epics: type, source and epics pv name. Epics aliases live alongside epics pv names in this group hierarchy. Finally configuration data (that usually arrives once) is found in subgroups to the configure groups at the top of the hdf5 hierarchy.

psana-translate

The rest of this document covers psana-translate. psana-translate runs as a psana module. As such, we have been able to develop several new features that will be discussed below. However the main technical reason for phasing out o2o-translate is to use a Data Description Language (DDL) to generate code that handles the many data types that different detectors produce. This use of DDL is part of psana-translate.

Running psana-translate

You can run psana-translate as you would any other psana module. Either through psana command line options or by writing a psana configuration file.  The module is Translator.H5Output.

When using the psana command line interface to run the module, the only option that is required to give to the Translator is the name of the output file.  This must be a fully qualified filename, with the output directory.  For example:

psana -m Translator.H5Output -o Translator.H5Output.output_file=exp-run001.h5 /reg/d/psdm/instrument/experiment/xtc/exp-run001*.xtc

would invoke the translator.  It will translate all the xtc files in run 001. This runs with default values for all the translator options. These are the recommended option values to use for translation.  The options include gzip compression at level 1 and no filtering on events or data. The Translator does not overwrite an existing h5 output file by default (set the option overwrite=true to overwrite the output).

The easiest way to try different translator options to write a psana.cfg file.  Copy the file default_psana.cfg that is included below or from the Translator package directory and modify option values that you wish.  The file default_psana.cfg includes extensive documentation on all the translator options.

New Features

With psana-translate, you can

• filter out whole events from translation
• filter out certain data, by data type, or by data source, or key string
• write ndarray's that other modules add to the event store
• write std::string's that other C++ modules add to the event store
• advanced: have a C++ module register a new type for translation

New Features Subject to Change

Three aspects of these new features are subject to change. These are highlighted in warning boxes below. In brief, these are

• How event key strings are incorporated into hdf5 paths
• How new C++ types are registered for translation.

Any future changes for these items will not affect automatic translation.

Important Changes between o2o-translate and psana-translate

The translation that psana-translate produces is most always backward compatible with what o2o-translate produced. The only difference likely to affect users is where the CsPad calibration constants are found, this is discussed in the The XTC-to-HDF5 Translator section below. There are also a number of minor differences which should make no difference to user code written to process o2o-translate hdf5 files. These are documented in the section Difference's with o2o-translate. hdf5 files created by o2o-translate or psana-translate contain an attribute defining the schema number. Below we document important changes introduced with Schema 4 as implemented in V00-01-00 and above for psana-translate. o2o-translate implemented schema versions 1,2 and 3. These important changes are the use of CalibStore for calibration constants, and dropping PNCCD::FullFrames from translation.

Calibrated Data

o2o-translate knows how to calibrate CsPad data. If o2o-translate was told where a calib-dir was (which it is for automatic translation) and calibration constants have been recorded in this directory (typically carried out by the calibration management tool by processing a dark run) then o2o-translate calibrates cspad data and write the calibrated data instead of the raw xtc data. It writes the calibrated data in the same place where the raw xtc would have gone. It would also write the calibration constants used (such as pedestals and pixel status) in a special group. Finally, if the common mode calibration was done (this depends on what files are deployed to the calib-dir) which is a correction calculated for each event, the source group containing all the event data will include a common_mode dataset with the common mode values. This allows users to recover the raw data from the calibrated data.

With psana-translate calibration is handled by external psana modules. These modules will produce calibrated data and psana-translate will find it and translate it to the hdf5 file. Understanding this flow of data is not necessary for automatic translation, however if users want to customize calibration, some understanding of how psana modules pass data through the event store, and are configured through config files is necessary.  The calibrated data will be distinguished from uncalibrated data with the use of a key in the event store. The key defaults to the value 'calibrated' but this is configurable through the psana.cfg file, in the section for the calibration module used. psana-translate provides special treatment for the calibration key. For psana-translate, the default value for the calibration key is 'calibrated' as well, but again, this is configurable through the psana.cfg file, in the section for Translator.H5Output. If psana-translate sees data with the key calibrated - it defaults to only translate data with the calibrated key and not the raw data. In the hdf5 file, one will find calibrated data where one would have otherwise found uncalibrated data. This is consistent with how o2o-translate translated calibrated cspad data. The calibrated key is not present in the hdf5 path names. This is different than what one finds for keys with ndarrays. For ndarrays the key is part of the h5 path name (see below).  The psana-translate option skip_calibrated can be set to true to get the uncalibrated data instead of calibrated data.

Calibration makes use of calibration constants - such as pedestals and pixel status. A key difference between psana-translate and o2o-translate is where these calibration constants are found, and the datatypes used to store them. For psana-translate they are found in the group CalibStore to the current configure group. For example, if we translate the first event in a run of the cxi tutorial data where we add the cspad calibration module before the psana-translate module:

psana -n 1 -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.output_file=calib.h5 exp=xpptut13:run=71

Then we will see

h5ls -r calib.h5 | grep -i "calibstore\|cspad"  # this command will include the following output

/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0/common_mode Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0/data Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0/element Dataset {1/Inf}
...
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.1/common_mode Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.1/data Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.1/element Dataset {1/Inf}
...
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PedestalsV1/XppGon.0:Cspad2x2.0/pedestals Dataset {185, 388, 2}
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PedestalsV1/XppGon.0:Cspad2x2.1/pedestals Dataset {185, 388, 2}
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PixelStatusV1/XppGon.0:Cspad2x2.0/status Dataset {185, 388, 2}
/Configure:0000/CalibStore/pdscalibdata::CsPadCommonModeSubV1/XppGon.0:Cspad2x2.0/data Dataset {SCALAR}
...

Things to note:

• There are common_mode datasets included with the data
• Both cspad sources have a pedestal dataset in CalibStore
• Only XppGon.0:Cspad2x2.0 has a common mode dataset in the calibstore.

When one looks at the common_mode dataset for XppGon.0:Cspad2x2.1, one sees the values are -10000, indicating no common mode calibration was done. The module loaded in this example to do calibration, cspad_mod.CsPadCalib, is equivalent to what o2o-translate would do. More information on o2o-translate calibration can be found at CsPad calibration in translator

An issue users may run into is understanding what calibration was done and recovering the raw data just from examining the hdf5 output. In the case of cspad, an understanding of the CsPadCalib module along with the what is in the hdf5 file does allow one to recover the uncalibrated data. This may not be possible with other calibration modules and detectors, in particular if nonlinear calibration algorithms are applied, such as applying a threshold.

PNCCD::FullFrame

This data is no longer translated. FullFrame is a copy of Frames with a more convenient interface for psana users, it is not considered to be as useful for hdf5 files. User's interested in having FullFrame written into their hdf5 files rather than the original Frames data should make a feature request.

Filtering Events

Since psana-translate runs as a psana module, it is possible to filter translated events through psana options and other modules. psana options allow you to start at a certain event, and process a certain number of events.  Moreover a user module that is loaded before the Translator module can tell psana that it should not pass this event on to any other modules, hence the Translator.H5Output module will never see the event and it will not get translated.

A downside of having modules loaded before the translator skip events is that updates to epics pv's, or configuration data will not get recorded. One may also wish to record the reason for filtering the event in the hdf5 file, as well as the event id's for the filtered events.  psana-translate provides an interface for doing these things. To use this mechanism, a module must put an object in the eventStore with a key that starts with

do_not_translate

For example, if a C++ module implements the event method to do the following:

  virtual void event(Event& evt, Env& env) {
	boost::shared_ptr<int> message = boost::make_shared<int>();        
    evt.put(message,"do_not_translate");
  }

Then none of the event data will get translated in any of the calib cycles.  The Translator will do the following

• For each calib cycle, it will make a filtered group
  • For instance, if the file has the group /Configure:0000/Run:0000/CalibCycle:0000, then it will also have:
    the group: /Configure:0000/Run:0000/Filtered:0000
• within each Filtered group, a time dataset that holds event id's of the filtered events.  

Suppose a user module has made some measurements that indicate this event should be filtered (for instance the beam energy is wrong). These measurements can be recorded in the hdf5 file by adding data to the event store that the Translator knows how to write.  As discussed below, the Translator can write ndarrays and strings as well as simple new types that user modules register. If a user module implements event to do the following:

// define user Module,

  virtual void event(Event& evt, Env& env) {
    boost::shared_ptr<std::string> message = boost::make_shared<std::string>("The beam energy is bad");        
    evt.put(message,"do_not_translate:message");
    const unsigned shape[1] = {4};
    boost::shared_ptr< ndarray<float,1> measurements = boost::make_shared< ndarray<float,1> >(shape);
    float *data = measurements->data();
    data[0] = 0.4;  data[1] = 1.3; data[2] = 2.2; data[3] = 3.1;
    evt.put(measurements,"do_not_translate:measurements");
  }

Note the use of the key "do_not_translate:xxx" the :xxx is not necessary, but it helps to uniquely quality the event data, and it will become a part of the 'src' group name where the do_not_translate event data is written to.  Since both std::string and a ndarray<float,1> are types that the Translator knows how to write, it will create the following groups and datasets in the hdf5 file:

• /Configure:0000/Run:0000/Filtered:0000/time  - this is as discussed above, the event id's for all filtered events
• /Configure:0000/Run:0000/Filtered:0000/std::string/noSrc__message/data  - this will be a dataset of variable length strings, each entry will be the string "The beam energy is bad"
• /Configure:0000/Run:0000/Filtered:0000/std::string/noSrc__message/time  - this will be a dataset of eventId's for the data above (there need not have been a std::string in all the filtered events).
• /Configure:0000/Run:0000/Filtered:0000/ndarray_float32_1/noSrc__measurements/data - this will be a dataset where each entry is a 1D array of 4 floats, with the values 0.4, 1.3, 2.2, 3.1
• /Configure:0000/Run:0000/Filtered:0000/ndarray_float32_1/noSrc__measurements/time - likewise the event ids for the ndarrays of the filtered events.

Note the src level group names: noSrc__mesage and noSrc__measurements. Since no source was specified with the calls to evt.put, the Translator starts with the string noSrc in the group name. Two underscores, __, separate the source from the keystring.

Note the fully qualified type information about the ndarray's written. This allows translation of different ndarrays in the event store that differ only by this type information (i.e: they have the same key and source).

Filtering from Python Modules

A Python module can use standard psana features to skip events as discussed above. It can also add any Python object into the event store that has the key "do_not_translate". This will create the Filtered:0000/time dataset as above. However to use the Translator filtering features that record user data, the Python module will have to add data that psana knows how to convert for C++ modules. Presently the only types that a Python module can add to the event store which will be seen by C++ modules are a number of ndarrays. A Python module will need to add one of these ndarray types to filter events, the data of the ndarray will be recorded in the hdf5 file.

Filtering Types

The psana.cfg file accepts a number of parameters that will filter out sets of psana types.  For example setting

EBeam = exclude

would cause any of the types Psana::Bld::BldDataEBeamV0, Psana::Bld::BldDataEBeamV1, Psana::Bld::BldDataEBeamV2, Psana::Bld::BldDataEBeamV3 or Psana::Bld::BldDataEBeamV4 to be excluded from translation.

All types are translated by default. To exclude a few types, you can add lines like EBeam = exclude to the psana.cfg file. You can also list them with the type_filter parameter:

type_filter exclude EBeam Andor

The type_filter parameter is useful for including a few types:

type_filter include CsPad Frame

A shortcut is available to turn off translation of all the Xtc data:

type_filter exclude psana

One would use this to only translate user module data, such as ndarrays, strings and newly registered types.

Src Filtering

Specific src's can be filtered by providing a list such as

src_filter = exclude NoDetector.0:Evr.2  CxiDs1.0:Cspad.0  CxiSc2.0:Cspad2x2.1  EBeam  FEEGasDetEnergy  CxiDg2_Pim

the syntax for a src in the filter list is what is supported by the Psana::Source class. This is a flexible syntax allowing for several ways to specify a src. It will match any detector or device number if this is not specified. See the section Psana Configuration File and all Options below for more details. If DAQ src aliases are present in the xtc file, these can be used for src filtering as well.  For example if the alias

acq01 -> SxrEndstation.0:Acqiris.0

is present, one can do

src_filter = exclude acq01

to exclude all data from the SxrEndstation.0:Acqiris.0 src.

Writing User Data

The translator will write NDarrays, C++ std::strings, and C++ types that the user registers.  Presently, registering new types is an advanced feature that requires familiarity with hdf5 programming.

NDArrays and Strings

ndarrays (up to dimension 4 of the standard integral types, floats and doubles) as well as std::string's that are written into the event store will be written to the hdf5 by default.  ndarrays can be passed to the Translator by Python modules as well as C++ modules. These events can be filtered as well.  The example in The XTC-to-HDF5 Translator above illustrates the group names used for ndarrays and strings. Note, the type group name for ndarrays is fully qualified by the template arguments, some examples of type names are

ndarray_int8_1           # a one dimensional array of 8 bit signed integers    (the C type char)
ndarray_uint8_2          # a two dimensional array of 8 bit unsigned integers
ndarray_int32_1          # a one dimensional array of 32 bit signed integers   (the C type int)
ndarray_uint64_3         # a 3D array of 64 bit unsigned integers   
ndarray_float32_2        # a 2D array of 32 bit floats  (the C type float)
ndarray_float64_1        # a 3D array of 64 bit floats (the C type double)

These names agree with what users find in the Python interface to psana.

Less common are the names used to store an ndarray of const data. An example name for such data is

ndarray_const_float32_2 

Fixed Dimensions vs. Variable Dimensions

The Translator defaults to using a fixed set of dimensions for all the ndarrays that go into the same dataset. The array received for the first data of the dataset determine these dimensions. For example, if from python one did

event.put(numpy.zeros((3,4),"mykey")

during the first event, but then

event.put(numpy.zeros((5,4),"mykey")

during the second event, the Translator would throw an error. Both of these arrays are supposed to go into an hdf5 path that ends with

/ndarray_float32_2/noSrc__mykey

but the underlying hdf5 type for this dataset has been set to a 2D array with dimensions (3,4). At present, one can start a new dataset during the second event

event.put(numpy.zeros((5,4),"mykey_larger")

to resolve this. In the near future, the Translator will support translation of ndarrays that vary only in the slow dimension to the same dataset. This feature will be activated with a modified key. One would prepend 'translate_vlen:' to the start of the keys. For example:

event.put(numpy.zeros((3,4),"translate_vlen:mykey")        # event one
event.put(numpy.zeros((5,4),"translate_vlen:mykey")        # event two

Now both ndarrays go to the same dataset, and the underlying hdf5 type changes to a vlen type of 1D arrays with dimension 4. The type name in the hdf5 path changes to indicate vlen, it will be

/ndarray_float32_2_vlen/noSrc__mykey

Registering New Types

C++ modules can register new types. Note, this is an advanced feature that requires familiarity with the Hdf5 programming in C.  Presently this feature is only suitable for simple types. An example is found in the file Translator/src/TestModuleNewWriter.cpp. We go through the example here. First a module will define the data type that they want to store. This type is a simple C struct of native types in the C language:

struct MyData {
  int32_t eventCounter;
  float energy;
};

Next, the module must define functions that create the hdf5 type for MyData, and fill a buffer to be written to the hdf5 file.  These functions must satisfy a particular signature:

  typedef hid_t (*CreateHDF5Type)(const void *userDataType);
  typedef const void * (*FillHdf5WriteBuffer)(const void *userDataType);

Here is what these functions might look like for MyData:

#include "hdf5/hdf5.h"
#include "MsgLogger/MsgLogger.h"

hid_t createMyDataHdf5Type(const void *) {
  static bool firstCall = true;
  static hid_t h5type = -1;
  if (not firstCall) return h5type;
  firstCall = false;
  h5type = H5Tcreate(H5T_COMPOUND, sizeof(MyData));
  
  herr_t status1 = H5Tinsert(h5type, "eventCounter", 
                             offsetof(MyData,eventCounter), 
                             H5T_NATIVE_UINT32);
  herr_t status2 = H5Tinsert(h5type, "energy", 
                             offsetof(MyData,energy), 
                             H5T_NATIVE_FLOAT);
  if ((h5type < 0) or (status1 < 0) or (status2<0)) {
    MsgLog("mydata",fatal,"unable to create MyData compound type");
  }
  MsgLog("mydata",trace,"Created hdf5 type for MyData  " << h5type);  
  return h5type;
}

const void * fillMyDataWriteBuffer(const void *data) {
  return data;
}

The function createMyDataHdf5Type must return an hdf5 type for MyData.  The void * that it is being passed will point to an actual instance of the MyData struct that was found in the eventStore.  Because MyData is so simple, the function createMyDataHdf5Type does not need to use this argument.  However a more complex type may include arrays of different sizes, and so the exact hdf5 type that describes the data cannot be determined without looking at the object.

The function fillMyDataWriteBuffer receives a void pointer to an instance of MyData that was found in the eventStore.  The function must then return a void pointer to a memory buffer that holds the data to be written into the hdf5 file.  Since MyData is so simply, the memory layout of the C++ object coincides with that of the hdf5 type, so we can simply return the original pointer to MyData. For more complex types, this will not be the case and fillMyDataWriteBuffer will have to manage a buffer of memory that persists after the function is called. It would then transfer the data in the complex C++ object into this memory buffer.

To register this new type for writing in the system, the user module must, in the beginJob() function, put a special object in the eventStore.  The Translator module will look for these special objects when it handles the beginJob() function. Then the user module can add MyData into the eventStore during the event() function:

#include "Translator/HdfWriterNew.h"

...
class TestNewHdfWriter : public Module {
public:
  TestNewHdfWriter(std::string moduleName) : Module(moduleName) {}
  virtual void beginJob(Event& evt, Env& env) {
    boost::shared_ptr<Translator::HdfWriterNew> newWriter = 
      boost::make_shared<Translator::HdfWriterNew>(&typeid(MyData), 
                                                   "data", 
                                                   createMyDataHdf5Type, 
                                                   fillMyDataWriteBuffer);
    evt.put(newWriter,"MyDataWriter");
  }
  
  virtual void event(Event& evt, Env& env) {
    boost::shared_ptr<MyData> myData = boost::make_shared<MyData>();
    myData->eventCounter = 11;
    myData->energy = 23.239;
    evt.put(myData,"example");
  }
};

The special type, HdfWriterNew, that is part of the Translator namespace, has the following arguments:

• the C++ std::type_info pointer for the new type being registered (&typeid(MyData) in the example)
• the name of the dataset ("data")
• the function that creates the hdf5 type (which we discussed above)
• the function that returns the memory buffer for writing (which we discussed above)

HdfWriterNew also takes an optional fifth argument that users can use to clean up resources.  Since MyData is so simple, there is no need to use this part of the API.  We will create the hdf5 type once, and not worry about closing it. 

The key "MyDataWriter" added when putting the newWriter in the event store is not important.  Giving it a distinct name can help debug problems that may arise in the Translator.

The translator, in each calib cycle, will make the following groups (for example in calib cycle 0):

• /Configure:0000/Run:0000/CalibCycle:0000/MyData/example
  • Note how the C++ type name, MyData, shows up in the path. 
  • Next the 'src' level group is based on the key "example" passed when putting myData in the event store.
• The dataset: /Configure:0000/Run:0000/CalibCycle:0000/MyData/example/data
  • The name "data" comes from the 2nd parameter to the HdfWriterNew object.
  • The dataset will be a 1D array of the hdf5 compound type with the fields
    • "eventCount"  uint32
    • "energy"  float

Psana Configuration File and all Options

When running the translator as a psana module, if is often convenient to create a psana.cfg file.  The Translator package include the file default_psana.cfg which is a psana configuration file that describes all the options possible, with extensive documentation as to what they mean.  Below we include this file for reference. To use this file, one could it and modify it. However it is not necessary to take the whole file - every value set is set to the default value. One could simply use this as a reference for those options values that one wants to change.

######################################################################
[psana]

# MODULES: any modules that produce data to be translated need be loaded 
# **BEFORE** Translator.H5Output (such as calibrated data or NDArray's)
# event data added by modules listed after Translator.H5Output is not translated.
modules = Translator.H5Output

files = **TODO: SPECIFY INPUT FILES OR DATA SOURCE HERE**

######################################################################
[Translator.H5Output]

# The only option you need to set for the Translator.H5Output module is
# output_file. All other options have default values (explained below).

# TODO: enter the full h5 output file name, including the output directory
output_file = output_directory/h5output.h5

# By default, the Translator will not overwrite the h5 file if it already exists
overwrite = false

# # # # # # # # # # # # # # # # # # # # #
# EPICS FILTERING
# The Translator can store epics pv's in one of two ways, or not at all.
# set store_epics below, to one of the following:
#
# updates_only   stores an epic pv when it has changed. The pv is stored 
#                in the current calib cycle.  For mutli calib cycle experiments, 
#                users may have to look back through several calib cycle's to 
#                find the latest value of a pv.
#
# calib_repeat   each calib cycle will include the latest value of all the epics 
#                pv's.  This can make it easier to find pv's for a calib cycle. 
#                For experiments with many short calib cycles, it produces 
#                many more datasets than neccessary.
#
# no             epics pv's will not be stored. You may also want to set Epics=exclude
#                (see below) if you do not want the epics configuration data stored

# The default is 'calib_repeat'

store_epics = calib_repeat

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# FILTERING
# 
# By default, all xtc data is Translated and many ndarrays that user modules (if any) 
# add are translated. Filtering can occur in either the code of user modules, or
# through options in the psana.cfg file. Here in the config file, different groups of 
# data can be filtered. There are four options for filtering data: 
#
#    type filtering   -  for example, exclude all cspad, regardless of the detector source
#    source filtering -  for example, exclude any data from a given detector source
#    key filtering    -  for example, include only ndarrays with a given key string
#    calibration      -  do not translate original xtc if a calibrated version is found
#
# Type filtering is based on sets of Psana data types. If you know what detectors or 
# devices to filter, leave type filtering alone and go to src_filter. 
#
# Type filtering has the highest precedence, then key filtering, then source 
# filtering, and lastly calibration filtering. When the Translator sees new data, 
# it first checks the type filter. If it is a filtered type (or unknown type) no further 
# translation occurs with the data - regardless of src or key. For data that gets 
# past the type filter, the Translator looks at the src and key. If the key 
# string is empty, it checks the source filter. Data with non empty key strings are 
# handled via the key filter. If the src is filtered, but the key is not, then the
# data will be translated. Data with the special calibration key string are handled 
# via the calibration filtering. 
#
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# TYPE FILTERING 
#
# One can include or exclude a class of Psana types with the following 
# options. Only the strings include or exclude are valid for these 
# type filtering options. 
# 
# Note - Epics in the list below refers only to the epicsConfig data
# which is the epics alias list, not the epics pv's. To filter the epics pv's
# see the 'store_epics' option above.

AcqTdc = include               # Psana::Acqiris::TdcConfigV1, Psana::Acqiris::TdcDataV1
AcqWaveform = include          # Psana::Acqiris::ConfigV1, Psana::Acqiris::DataDescV1
Alias = include                # Psana::Alias::ConfigV1
Andor = include                # Psana::Andor::ConfigV1, Psana::Andor::FrameV1
Arraychar = include            # Psana::Arraychar::DataV1
Control = include              # Psana::ControlData::ConfigV1, Psana::ControlData::ConfigV2, Psana::ControlData::ConfigV3
Cspad = include                # Psana::CsPad::ConfigV1, Psana::CsPad::ConfigV2, Psana::CsPad::ConfigV3, Psana::CsPad::ConfigV4, Psana::CsPad::ConfigV5, Psana::CsPad::DataV1, Psana::CsPad::DataV2
Cspad2x2 = include             # Psana::CsPad2x2::ConfigV1, Psana::CsPad2x2::ConfigV2, Psana::CsPad2x2::ElementV1
DiodeFex = include             # Psana::Lusi::DiodeFexConfigV1, Psana::Lusi::DiodeFexConfigV2, Psana::Lusi::DiodeFexV1
EBeam = include                # Psana::Bld::BldDataEBeamV0, Psana::Bld::BldDataEBeamV1, Psana::Bld::BldDataEBeamV2, Psana::Bld::BldDataEBeamV3, Psana::Bld::BldDataEBeamV4, Psana::Bld::BldDataEBeamV5
Encoder = include              # Psana::Encoder::ConfigV1, Psana::Encoder::ConfigV2, Psana::Encoder::DataV1, Psana::Encoder::DataV2
Epics = include                # Psana::Epics::ConfigV1
Epix = include                 # Psana::Epix::ConfigV1, Psana::Epix::ElementV1
EpixSampler = include          # Psana::EpixSampler::ConfigV1, Psana::EpixSampler::ElementV1
Evr = include                  # Psana::EvrData::ConfigV1, Psana::EvrData::ConfigV2, Psana::EvrData::ConfigV3, Psana::EvrData::ConfigV4, Psana::EvrData::ConfigV5, Psana::EvrData::ConfigV6, Psana::EvrData::ConfigV7, Psana::EvrData::DataV3
EvrIO = include                # Psana::EvrData::IOConfigV1
Evs = include                  # Psana::EvrData::SrcConfigV1
FEEGasDetEnergy = include      # Psana::Bld::BldDataFEEGasDetEnergy
Fccd = include                 # Psana::FCCD::FccdConfigV1, Psana::FCCD::FccdConfigV2
Fli = include                  # Psana::Fli::ConfigV1, Psana::Fli::FrameV1
Frame = include                # Psana::Camera::FrameV1
FrameFccd = include            # Psana::Camera::FrameFccdConfigV1
FrameFex = include             # Psana::Camera::FrameFexConfigV1
GMD = include                  # Psana::Bld::BldDataGMDV0, Psana::Bld::BldDataGMDV1
Gsc16ai = include              # Psana::Gsc16ai::ConfigV1, Psana::Gsc16ai::DataV1
Imp = include                  # Psana::Imp::ConfigV1, Psana::Imp::ElementV1
Ipimb = include                # Psana::Ipimb::ConfigV1, Psana::Ipimb::ConfigV2, Psana::Ipimb::DataV1, Psana::Ipimb::DataV2
IpmFex = include               # Psana::Lusi::IpmFexConfigV1, Psana::Lusi::IpmFexConfigV2, Psana::Lusi::IpmFexV1
L3T = include                  # Psana::L3T::ConfigV1, Psana::L3T::DataV1
OceanOptics = include          # Psana::OceanOptics::ConfigV1, Psana::OceanOptics::ConfigV2, Psana::OceanOptics::DataV1, Psana::OceanOptics::DataV2
Opal1k = include               # Psana::Opal1k::ConfigV1
Orca = include                 # Psana::Orca::ConfigV1
Partition = include            # Psana::Partition::ConfigV1
PhaseCavity = include          # Psana::Bld::BldDataPhaseCavity
PimImage = include             # Psana::Lusi::PimImageConfigV1
Pimax = include                # Psana::Pimax::ConfigV1, Psana::Pimax::FrameV1
Princeton = include            # Psana::Princeton::ConfigV1, Psana::Princeton::ConfigV2, Psana::Princeton::ConfigV3, Psana::Princeton::ConfigV4, Psana::Princeton::ConfigV5, Psana::Princeton::FrameV1, Psana::Princeton::FrameV2
PrincetonInfo = include        # Psana::Princeton::InfoV1
Quartz = include               # Psana::Quartz::ConfigV1
Rayonix = include              # Psana::Rayonix::ConfigV1, Psana::Rayonix::ConfigV2
SharedAcqADC = include         # Psana::Bld::BldDataAcqADCV1
SharedIpimb = include          # Psana::Bld::BldDataIpimbV0, Psana::Bld::BldDataIpimbV1
SharedPim = include            # Psana::Bld::BldDataPimV1
Spectrometer = include         # Psana::Bld::BldDataSpectrometerV0
TM6740 = include               # Psana::Pulnix::TM6740ConfigV1, Psana::Pulnix::TM6740ConfigV2
Timepix = include              # Psana::Timepix::ConfigV1, Psana::Timepix::ConfigV2, Psana::Timepix::ConfigV3, Psana::Timepix::DataV1, Psana::Timepix::DataV2
TwoDGaussian = include         # Psana::Camera::TwoDGaussianV1
UsdUsb = include               # Psana::UsdUsb::ConfigV1, Psana::UsdUsb::DataV1
pnCCD = include                # Psana::PNCCD::ConfigV1, Psana::PNCCD::ConfigV2, Psana::PNCCD::FramesV1

# user types to translate from the event store
ndarray_types = include        # ndarray<int8_t,1>, ndarray<int8_t,2>, ndarray<int8_t,3>, ndarray<int8_t,4>, ndarray<int16_t,1>, ndarray<int16_t,2>, ndarray<int16_t,3>, ndarray<int16_t,4>, ndarray<int32_t,1>, ndarray<int32_t,2>, ndarray<int32_t,3>, ndarray<int32_t,4>, ndarray<int64_t,1>, ndarray<int64_t,2>, ndarray<int64_t,3>, ndarray<int64_t,4>, ndarray<uint8_t,1>, ndarray<uint8_t,2>, ndarray<uint8_t,3>, ndarray<uint8_t,4>, ndarray<uint16_t,1>, ndarray<uint16_t,2>, ndarray<uint16_t,3>, ndarray<uint16_t,4>, ndarray<uint32_t,1>, ndarray<uint32_t,2>, ndarray<uint32_t,3>, ndarray<uint32_t,4>, ndarray<uint64_t,1>, ndarray<uint64_t,2>, ndarray<uint64_t,3>, ndarray<uint64_t,4>, ndarray<float,1>, ndarray<float,2>, ndarray<float,3>, ndarray<float,4>, ndarray<double,1>, ndarray<double,2>, ndarray<double,3>, ndarray<double,4>, ndarray<const int8_t,1>, ndarray<const int8_t,2>, ndarray<const int8_t,3>, ndarray<const int8_t,4>, ndarray<const int16_t,1>, ndarray<const int16_t,2>, ndarray<const int16_t,3>, ndarray<const int16_t,4>, ndarray<const int32_t,1>, ndarray<const int32_t,2>, ndarray<const int32_t,3>, ndarray<const int32_t,4>, ndarray<const int64_t,1>, ndarray<const int64_t,2>, ndarray<const int64_t,3>, ndarray<const int64_t,4>, ndarray<const uint8_t,1>, ndarray<const uint8_t,2>, ndarray<const uint8_t,3>, ndarray<const uint8_t,4>, ndarray<const uint16_t,1>, ndarray<const uint16_t,2>, ndarray<const uint16_t,3>, ndarray<const uint16_t,4>, ndarray<const uint32_t,1>, ndarray<const uint32_t,2>, ndarray<const uint32_t,3>, ndarray<const uint32_t,4>, ndarray<const uint64_t,1>, ndarray<const uint64_t,2>, ndarray<const uint64_t,3>, ndarray<const uint64_t,4>, ndarray<const float,1>, ndarray<const float,2>, ndarray<const float,3>, ndarray<const float,4>, ndarray<const double,1>, ndarray<const double,2>, ndarray<const double,3>, ndarray<const double,4>
std_string = include           # std::string


# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# TYPE FILTER SHORTCUT
#
# In addition to filtering Psana types by the options above, one can use
# the type_filter option below. For example:
#
# type_filter include cspad       # will only translate cspad types. Will not translate
#                                 # ndarrays or strings
# type_filter exclude Andor evr   # translate all except the Andor or Evr types
# 
# If you do not want to translate what is in the xtc file, use the psana shortcut:
#
# type_filter exclude psana       # This will only translate ndarray's and strings 
#
# Likewise doing:
#
# type_filter include psana       # will translate all xtc data, but skip any ndarray's or strings
#
# The default is to include all

type_filter include all

# note - if type_filter is anything other than 'include all' it takes precedence
# over the classes of type filter options above, like Cspad=include.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# SOURCE FILTERING
#
# The default for the src_filter option is "include all"
# If you want to include a subset of the sources, do
#
# src_filter include srcname1 srcname2  
#
#  or if you want to exclude a subset of sources, do
#
# src_filter exclude srcname1 srcname2
#
# The syntax for specifying a srcname follows that of the Psana Source (discussed in 
# the Psana Users Guide). The Psana Source recognizes DAQ alias names (if present
# in the xtc files), several styles for specifying a Pds Src, as well as detector matches 
# where the detector number, or device number is not known.
# 
# Specifically, format of the match string can be:
#
#       DetInfo(det.detId:dev.devId) - fully or partially specified DetInfo
#       det.detId:dev.devId - same as above
#       DetInfo(det-detId|dev.devId) - same as above
#       det-detId|dev.devId - same as above
#       BldInfo(type) - fully or partially specified BldInfo
#       type - same as above
#       ProcInfo(ipAddr) - fully or partially specified ProcInfo
#
# For example
#        DetInfo(AmoETOF.0.Acqiris.0)  
#        DetInfo(AmoETOF.0.Acqiris)  
#        DetInfo(AmoETOF:Acqiris)
#        AmoETOF:Acqiris
#        AmoETOF|Acqiris
#
# will all match the same data, AmoETOF.0.Acqiris.0. The later ones will match
# additional data (such as detector 1, 2, etc.) if it is present.
#
# A simple way to set up src filtering is to take a look at the sources in the 
# xtc input using the psana EventKeys module.  For example
#
# psana -n 5 -m EventKeys exp=cxitut13:run=22 
#
# Will print the EventKeys in the first 5 events.  If the output includes
#
#   EventKey(type=Psana::EvrData::DataV3, src=DetInfo(NoDetector.0:Evr.2))
#   EventKey(type=Psana::CsPad::DataV2, src=DetInfo(CxiDs1.0:Cspad.0))
#   EventKey(type=Psana::CsPad2x2::ElementV1, src=DetInfo(CxiSc2.0:Cspad2x2.1))
#   EventKey(type=Psana::Bld::BldDataEBeamV3, src=BldInfo(EBeam))
#   EventKey(type=Psana::Bld::BldDataFEEGasDetEnergy, src=BldInfo(FEEGasDetEnergy))
#   EventKey(type=Psana::Camera::FrameV1, src=BldInfo(CxiDg2_Pim))
#
# Then one can filter on these six srcname's:
#
#  NoDetector.0:Evr.2  CxiDs1.0:Cspad.0  CxiSc2.0:Cspad2x2.1  EBeam  FEEGasDetEnergy  CxiDg2_Pim
#

src_filter = include all

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# CALIBRATION FILTERING
#
# Psana calibration modules can produce calibrated versions of different 
# data types. Depending on the module used, you may get an NDArray, an 
# image, or the same data type as was in the xtc but with calibrated data.
#
# If you are doing the latter, the module output will be data of the same type 
# and src as the uncalibrated data, with an additional key, such as 'calibrated'.
# If these modules are configured to use a different key, set calibration_key
# below accordingly:

calibration_key = calibrated

# The Translator defaults to writing calibrated data in place of uncalibrated
# data. If you do not want the calibrated data, set skip_calibrated to true.

skip_calibrated = false

# note, setting skip_calibrated to true will force sets exclude_calibstore 
# (below) to be true as well.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# CALIBSTORE FILTERING
#
# Calibration modules may publish the data they used to produce the calibrated
# event objects. Examples of data would be pedestal values, pixel status (what
# pixels are hot) and common mode algorithm parameters. This data will be published
# in what is called the Psana calibStore. When the Translator sees calibrated 
# event data, it will look for the corresponsinding calibStore data as well.
# If you do not want it to translate calibStore data, set the following to true.

exclude_calibstore = false

# otherwise, the Translator will create a group CalibStore that holds the
# calibstore data. Note, the Translator looks for all calibStore data associated 
# with the calibration modules. If a calibration module was configured to not do 
# certain calibrations (such as gain) but the module still put gain values
# in the config store (even though it did not use them) the Translator 
# would still translate those gain values.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# KEY FILTERING
#
# Psana modules loaded before the translator may put a variety of objects in the event 
# store. Be default, the Translator will translate any new data that it knows about.
# In addition to the psana types, it knows about NDArrays, C++ strings, and has a C++ interface 
# for registering new simple types. NDarray's up to 4 dimensions of 10 basic types 
# (8, 16, 32 and 64 bit signed and unsigned int, float and double) as well as the const 
# versions of these types are translated.
#
# Generally Psana modules will attach keys to these objects (the keys are simply strings).
# To filter the set of keys that are translated, modify the parameter below:

key_filter = include all

# The default is to not look at the key but rather translate all data that the translator
# knows about. An example of including only data with the key finalanswer would be
#
# key_filter = include finalanswer
#
# To exclude a few keys, one can do
#
# key_filter = exclude arrayA arrayB
#
# Note, key filtering does not affect translation of data without keys. For instance
# setting key_filter = include keyA does not turn off translation of data without keys.
# Of all the data with keys, only those where the key is keyA will be translated.
#
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
# COMPRESSION 
#
# The following options control compression for most all datasets.
# Shuffling improves compression for certain datasets. Valid values for
# deflate (gzip compression level) are 0-9. Setting deflate = -1 turns off
# compression.

shuffle = true
deflate = 1

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# TECHNICAL, ADVANCED CONFIGURATION
# 
# ---------------------------------------
# CHUNKING
# The commented options below give the default chunking options.
# Objects per chunk are selected from the target chunk size (16 MB) and 
# adjusted based on min/max objects per chunk, and the max bytes per chunk.
# It is important that the chunkCache (created on a per dataset basis) be 
# large enough to hold at least one chunk, ideally all chunks we need to have
# open at one time when writing to the dataset (usually one, unless we repair
# split events):
 
# chunkSizeTargetInBytes = 1703936 (16MB)
# chunkSizeTargetObjects = 0 (0 means select objects per chunk from chunkSizeInBytes)
# maxChunkSizeInBytes = 10649600  (100MB)
# minObjectsPerChunk = 50              
# maxObjectsPerChunk = 2048
# chunkCacheSizeTargetInChunks = 3
# maxChunkCacheSizeInBytes = 10649600  (100MB)

# ---------------------------------------
# REFINED DATASET CONTROL
#
# There are six classes of datasets for which individual options for shuffle,
# deflate, chunkSizeTargetInBytes and chunkSizeTargetObjects can be specified:
#
# regular (most everything, all psana types)
# epics (all the epics pv's)
# damage (accompanies all regular data from event store)
# ndarrays (new data from other modules)
# string's (new data from other modules)
# eventId (the time dataset that also accompanies all regular data, epics pvs, ndarrays and strings)
#
# The options for regular datasets have been discussed above. The other five datasets 
# get their default values for shuffle, deflate, chunkSizeInBytes and chunkSizeInObjects
# from the regular dataset options except in the cases below:
 
# damageShuffle = false
# stringShuffle = false
# epicsPvShuffle = false
# stringDeflate = -1
# eventIdChunkSizeTargetInBytes = 16384
# epicsPvChunkSizeTargetInBytes = 16384

# The rest of the shuffle, deflate and chunk size options for the other five datasets are:
#
# eventIdShuffle = true
# eventIdDeflate = 1
# damageDeflate = 1
# epicsPvDeflate = 1
# ndarrayShuffle = true
# ndarrayDeflate = 1
# eventIdChunkSizeTargetObjects = 0
# damageChunkSizeTargetInBytes = 1703936
# damageChunkSizeTargetObjects = 0
# stringChunkSizeTargetInBytes = 1703936
# stringChunkSizeTargetObjects = 0
# ndarrayChunkSizeTargetInBytes = 1703936
# ndarrayChunkSizeTargetObjects = 0
# epicsPvChunkSizeTargetObjects = 0

# ---------------------------------------
# SPLIT EVENTS
# When the Translator encounters a split event, it checks a cache to see
# if it has already seen it.  If it has, it fills in any blanks that it can.
# To prevent this cache from growing to large, set the maximum number of
# split events to look back through here (default is 3000):

max_saved_split_events = 3000

# ---------------------------------------
# HDF5 FILE PROPERTIES
#
# split large files, presently we only support NoSplit. Future options may be: Family and SplitScan
# for future splitting, splitSize defaults to 10 GB
split = NoSplit
splitSize = 10737418240

Translation and Damage

psana has a specific damage policy that tells it what damaged data is acceptable for psana modules and what data is not. The default behavior is

• configStore - only undamaged data is stored in the configStore
• EventStore - undamaged data, and EBeam data with user damage is stored in the event, all other damage is not stored
  
  

psana-translate records event ids and damage for any xtc data that passes psana's damage policy. So by default, damaged config objects, and damaged events (other then user damaged EBeam data) are not translated. This deviates slightly from what o2o-translate would translate.  o2o-translate would also store out of order damaged event data.  There is a psana option that can be added to the [psana] section of the .cfg file to recover this behavior.  Below we document some special options that control what damaged data psana stores:

• store-out-of-order-damage  - defaults to false, set to true if you want to translate out of order damaged data
• store-user-ebeam-damage  - defaults to true, set to false if you do not want to translate EBeam data that only has user damage
• store-damaged-config - defaults to false, set to true if you want to store damaged config data

Difference's with o2o-translate

Here we cover differences with o2o-translate that we expect will be minor and not affect user code.

Feature's Dropped from o2o-translate

hdf file creation parameters
Only NoSplit is implemented - no family or split drivers.

In general a number of o2o-translate options are no longer supported.  In particular:
-G (long names like CalibCycle:0000 instead of CalibCycle) is always on.

Speed

psana-translate runs about 10% slower than o2o-translate does.

Performance testing was done during November/December of 2013.  Both o2o-translate and psana-translate worked through a 92 GB xtc file using compression=1 on the rhat6 machine psdev105.  They read and wrote the data from /u1. They both used the non-parallel compression library.  o2o-translate produced a 68GB file in 65 minutes and psana-translate produced a 65GB file in 70 minutes.  (Speeds of about 22MB/sec).  Production runs will use the parallel compression library and are expected to run at faster speeds (about 50MB/sec).

Technical Difference's with o2o-translate

Below is a list of technical differences between psana-translate and o2o-translate. These differences should not affect users.

• File attributes runNumber, runType and experiment not stored, instead expNum, experiment, instrument and jobName are stored (from the psana Env object)
• The attribute :schema:timestamp-format is always "full", there is no option for "short"
• The output file must be explicitly specificed in the psana cfg file. It is not inferred from the input.
• The File attribute origin is now psana-translator as opposed to translator
• The end sec and nanoseconds are not written into the Configure group at the end of the job as there is no EventId in the Event at the end.
• integer size changes - a number of fields have changed size, a few examples are below.  In one quirky case, this caused translation to be different.  The reason was that the data was uninitialized, and the new 32 bit value was different than the old 16 bit value. Data produced from 2014 onward will not include unitialized data in the translation, users will not have to worry about.  Unitialized data is very rare in pre 2014 data and, due to its location, not likely to be used in analysis.
• A few Examples of field size changes:
  • EvrData::ConfigV7/seq_config - sync_source - enum was uint16, now uint32
  • EvrData::ConfigV7/seq_config - beam_source - enum was uint16, now uint32
  • Ipimb::DataV2 - source_id was uint16, now uint8
  • Ipimb::DataV2 - conn_id was uint16 now uint8
  • Ipimb::DataV2 - module was uint16, now uint8

Some types have their field names rearranged. For example with ControlData::ConfigV2 one has:

ControlData::ConfigV2:
  o2o: uses_duration uses_events duration events npvControls npvMonitors npvLabels
psana: events uses_duration uses_events duration npvControls npvMonitors npvLabels

EvrData::ConfigV7:
  o2o: code isReadout isCommand isLatch reportDelay reportWidth releaseCode maskTrigger maskSet maskClear desc readoutGroup
psana: code isReadout isCommand isLatch reportDelay reportWidth maskTrigger maskSet maskClear desc readoutGroup releaseCode

Epics Ctrl datasets (in the configure group as opposed to the calib group) are not chunked.  They are stored as fixed size datasets depending on the number of pv's.

Only one epics pv is stored per name (of course, one epics pv may have any number of elements within it). This is fine as the epic pv name is supposed to uniquely identify the pv.  However in xtc files, you can see several epics pv's with the same pvname, but different pvid's. This scenario should only arise when the same pv is coming from different sources, and replicates the same data.  Psana only stores one epics pv per name (the last one it sees in a datagram). This is the one that psana-translate will pick up and store.

All Epics pv's are stored in the source folder EpicsArch.0:NoDevice.0.  With o2o-translate, some could be split off into other folders (such as AmoVMI.0:Opal1000.0). As epics pv names uniquely identify the data, the source information should not be needed.

Some DAQ config objects include space for a maximum number of entries.  o2o-translate would only write entries for those used, not the maximum entries.  psana-translate does not.  For example:

• The Acqiris::ConfigV1 vert dataset now always prints the max of 20 channels, even if the user will only be using 3.
  • Note, in this case the Acqiris data will still only include the 3 channels being used. o2o-translate was making an adjustment to the config data being written.

psana-translate will write an emtpy output_lookup_table for the Opal1k::ConfigV1 dataset named output_lookup_table, even if output_lookup_table() is enabled.  o2o-translate would not.

psana-translate does not produce the _dim_fix_flag_20103 datasets that o2o-translate did.

Bld::BldDataGMDV  the field fSpare1 has been dropped from this type.

With psana-translate, if all the xtc's coming from a particular source are damaged, you will not see a 'data' dataset in the hdf5 file. You will see the time, _damage and _mask datasets that tell the damage and events where the omitted data lives. o2o-translate may have created a 'data' dataset filled with blanks.

As discussed above, OutOfOrder Damage is not translated by default. o2o-translate translated out of order damage, however psana-translate does not.  psana can be told to include this kind of damaged data by setting store-out-of-order-damage=true in the [psana] section of your .cfg file.

When the number of events is recorded in the control data, o2o-translate would set the chunk size based on this value.  psana-translate does the same.  However o2o-translate also looked at the actual number of events and used this as well to set chunk sizes in future calib cycles.  psana-translate does not do this latter part.