Introduction

Translation from the LCLS data format of XTC to the more general scientific format  HDF5 is carried out by the XTC to HDF5 Translator. We refer to the software that carries out the translation as psana-translate. This distinguishes it from the earlier version of the software called o2o-translate (which has been deprecated). Translation for experiments should generally be carried out by using the automatic hdf5 translation feature of the Web Portal of Experiments. Below features are discussed that allow for customization of the translation - filtering unwanted events or detectors as well as adding processed data. The web portal provides a mechanism to customize the translation using these features. Experiment POC's should be able to help with using these features. Documentation on o2o-translate, which discusses some history with regards to selecting hdf5 for a scientific data format for general use can be found

Translator
User Interface to Translator

Automatic translation of xtc to hdf5 is managed by the Interface Controlller.

A discussion of the hdf5 output format for translator can be found here:

The contents of HDF5 files - the output format. Users that will work with hdf5 output should review this document. Some key points are:

• Datasets need not be aligned.  That is the 5th image in a detector dataset may come from a different event than the 5th record in a gas detector dataset. One can match up records from different datasets by use the time datasets.
• One should use the _mask datasets to identify valid data. A _mask dataset record is 1 when the corresponding record of the data dataset if valid, 0 if it is not. When the _mask record is 0, the data record will be all zeros and should not be processed. The mask is 0 when the xtc data is damaged. The type damaged data can then be found in the _damage dataset. The main reason to record damaged data is to keep datasets as aligned as possible.
• The hdf5 group hierarchy has the following levels: run, calib cycle. type, source  - regular event data is organized into datasets that live at the source level. Epics is special, rather then the two groups type and source, there are three groups for epics: type, source and epics pv name. Epics aliases live alongside epics pv names in this group hierarchy. Finally configuration data (that usually arrives once) is found in subgroups to the configure groups at the top of the hdf5 hierarchy.

psana-translate

The rest of this document covers psana-translate. psana-translate runs as a psana module. As such, we have been able to develop several new features that will be discussed below. However the main technical reason for phasing out o2o-translate is to use a Data Description Language (DDL) to generate code that handles the many data types that different detectors produce. This use of DDL is part of psana-translate.

Running psana-translate

When translating entire runs of an experiment, users should use the web portal of experiments. Translation can produce large amounts of data that needs to be managed by LCLS. The automatic system launches jobs through the batch system using the appropriate queues. The queue used can affect I/O performance. In addition, a faster parallel compression library is used.

You can also run psana-translate by hand. You run it as you would any other psana module. Either through psana command line options or by writing a psana configuration file.  The module is Translator.H5Output. When using the psana command line interface to run the module, the only option that is required to give to the Translator is the name of the output file.  This must be a fully qualified filename, with the output directory.  For example:

psana -m Translator.H5Output -o Translator.H5Output.output_file=exp-run001.h5 exp=xpptut13:run=71

would invoke the translator.  It will translate all the xtc files in run 71 of the xpptut13 dataset. This runs with default values for all the translator options. These are the recommended option values to use for translation.  The options include gzip compression at level 1 and no filtering on events or data. The Translator does not overwrite an existing h5 output file by default (set the option overwrite=true to overwrite the output). There are many different options you can set for translation that are discussed below. The easiest way to try different translator options is to write a psana.cfg file.  Copy the file default_psana.cfg that is included below or from the Translator package directory and modify option values that you wish.  The file default_psana.cfg includes extensive documentation on all the translator options.

When running the Translator directly, you can make use of the same parallel compression library that the automatic translation system uses by setting the following environment variables (below is for bash):

export PAZLIB_MAX_THREADS=8
export LD_PRELOAD=libpazlib.so

before running. Automatic translation also calibrates all cspad data for which calibration constants have been deployed. To achieve this by hand, one can load the module cspad_mod.CsPadCalib as well (it must be loaded before the Translator). When running this module, it is useful to get some of its diagnostic messages to verify that it has found calibration files. After setting PAZLIB_MAX_THREADS and LD_PRELOAD, one could additionally set

export MSGLOGCONFIG=CalibDataProxy=trace

before running

psana -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.output_file=exp-run001.h5 exp=xpptut13:run=71

to translate in the same manner as the automatic translation system. Better I/O is achieved by writing output to the scratch folder of the experiment rather than a users home directory (on the LCSL system) and running translation through the batch system using the appropriate queue.

Split Scan Mode

The Translator supports split scan mode. In this mode, calib cycles are written to separate hdf5 files. A master file will have external links to these separate hdf5 files. Users need only work with the master file. The master file uses the same schema as one finds without split scan mode, with one exception discussed below (the logging of filtered events). Otherwise, little modification to user code is required when working with the master file. What is required, is following external links (see below for tips on this). Not all experiments use more than one calib cycle. For experiments that use one calib cycle per run, split scan mode provides no benefit. Two reasons to use split scan mode is first, that the resulting hdf5 file from normal translation is too large, and second, to make translation faster by translating the data in parallel.

Two versions of split scan mode have been implemented. The most recent (soon to be available in ana-0.13.2 and later) is an MPI (Message Passing Interface) based version. This is the recommended way to run split scan mode. However the previous version is still available and is documented below.

MPI Split Scan

Here is an example command for launching the mpi based splitscan Translator:

bsub -q psanaq -a mympi -n 9 -R "span[ptile=1]" -o translate_%J.out h5-mpi-translate -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.printenv=True -o Translator.H5Output.output_file=mydir/split.h5 exp=xpptut13:run=71

More study is needed to see if this is the optimal way to launch a job. The first few arguments for bsub set up the batch job. The batch job uses 9 processes that are distributed (the ptile=1) onto distinct nodes in the mpi cluster. We use distinct nodes since each translator job will use up to 9 threads with the parallel compression library. As you read through the above command line, you will see that h5-mpi-translate, rather than psana, is the driver program for mpi based split scan translation. An additional option, printenv=True, has been added. This option is only picked up by h5-mpi-translate and can be useful for debugging, but is not necessary (it prints all the environment variables the Translator sees). One node (with the master process) will read through the data, identify the start of calib cycles, and communicate via MPI with the other 8 nodes. The other 8 nodes will do all the translation - telling the master when they are done and ready for a new job. The master will add links to the master h5 output file only after the separate calib files are done.

Generally, there will be one calib cycle file for each calib cycle. However to prevent too many calib cycle files from being produced for experiments that have only a few events per calib cycle, small calib cycles are grouped into one file. They are grouped until the total number of events in the file exceeds a threshold. The threshold defaults to 100 but is configurable.

For the above command, assuming there are multiple calib cycles in run 71 of xpptut13 that each have at least 100 events, and that the batch job was 39283, you will produce the following files:

translate_39283.out
mydir/split.h5
mydir/split_cc0000.h5
mydir/split_cc0001.h5
...

When moving the hdf5 files, make sure they all reside in the same directory. The links from the master to the calib cycle files assume they are in the same directory.

The job output file (translate_39283.out) will record timing information for the master and workers. Identifying the optimal number of nodes for h5-mpi-translate requires further analysis, however you can identify idle workers or a poor distribution of the work by looking at the report in the batch job output file.

Non-MPI Split Scan

To run the Translator in split scan mode without MPI, three options must be set. These options do the following:

• Tell the Translator to split (option split=SplitScan)

• Tell the Translator how many jobs to run to parallelize the work (option jobTotal=N)

• Tell the Translator which job it is (option jobNumber=K)

Next, one must run N different Translator jobs where the jobNumber parameter varies. An example of using two jobs for translation is:

psana -m Translator.H5Output -o Translator.H5Output.output_file=mydir/split.h5 -o Translator.H5Output.split=SplitScan -o Translator.H5Output.jobNumber=0 -o Translator.H5Output.jobTotal=2 exp=xpp123:run=10
psana -m Translator.H5Output -o Translator.H5Output.output_file=mydir/split.h5 -o Translator.H5Output.split=SplitScan -o Translator.H5Output.jobNumber=1 -o Translator.H5Output.jobTotal=2 exp=xpp123:run=10

(This mode always writes one calib cycle per file).

In this mode, each Translator job reads through the entire set of xtc files. As more and more Translator jobs are run simultaneously, the overall speed of translation diminishes while the load on the network steadily increases. It is recommended that users run no more than 5 Translator jobs. Testing has shown 5 jobs provide up to a three times speed up in translation for the non-mpi mode. (greater speedups have been seen with the MPI version).

Reading While Translating

HDF5 presently has little support for reading a file that is being created, and in general does not recommend this. However the master file is written in a way to support this as well as possible. When using the MPI split scan translator, links are not added to the master file after the calib files are done. Thus it is always safe to traverse those links. With the non-MPI split scan translator this is true except for the last N links, where N is the number of jobs running. These links may be written before the calib cycle files they link to are finished. To see updates in the master file, users may need to shut down programs like Matlab and h5py and restart them. It is not sufficient to close and reopen the master file within a Python or Matlab session.

Translation differences with split scan mode

The only difference users should see is if they provide modules that use the special key 'do_not_translate' to drop events from Translation. Ordinarily, as discussed below, in addition to dropping the event from translation, a event id for the dropped event is recorded in a hdf5 group such as /Configure:0000/Run:0000/Filtered:0000. These groups are not created in split scan mode (the event will still be be dropped).

When working with the master file, it is necessary to follow external links. For instance, to get a recursive listing of all the groups in the output file using h5ls, one must do

h5ls -r -E master.h5

as opposed to h5ls -r file.h5. The -E option instructs h5ls to follow external links. Similar functionality should exist in h5py, Matlab, and other software that works with HDF5.

New Features

With psana-translate, you can

• filter out whole events from translation
• filter out certain data, by data type, or by data source, or key string
• write ndarray's that other modules add to the event store
• write std::string's that other C++ modules add to the event store
• advanced: have a C++ module register a new type for translation

New Features Subject to Change

Three aspects of these new features are subject to change. These are highlighted in warning boxes below. In brief, these are

• hdf5 group names for ndarray types
• How event key strings are incorporated into hdf5 paths
• How new C++ types are registered for translation.

Important Changes between o2o-translate and psana-translate

The translation that psana-translate produces is most always backward compatible with what o2o-translate produced. The only difference likely to affect users is where the CsPad calibration constants are found, this is discussed in the The XTC-to-HDF5 Translator section below. There are also a number of minor differences which should make no difference to user code written to process o2o-translate hdf5 files. These are documented in the section Difference's with o2o-translate. hdf5 files created by o2o-translate or psana-translate contain an attribute defining the schema number. Below we document important changes introduced with Schema 4 as implemented in V00-01-00 and above for psana-translate. o2o-translate implemented schema versions 1,2 and 3. These important changes are the use of CalibStore for calibration constants, and dropping PNCCD::FullFrames from translation.

Calibrated Data

o2o-translate knows how to calibrate CsPad data. If o2o-translate was told where a calib-dir was (which it is for automatic translation) and calibration constants have been recorded in this directory (typically carried out by the calibration management tool by processing a dark run) then o2o-translate calibrates cspad data and write the calibrated data instead of the raw xtc data. It writes the calibrated data in the same place where the raw xtc would have gone. It would also write the calibration constants used (such as pedestals and pixel status) in a special group. Finally, if the common mode calibration was done (this depends on what files are deployed to the calib-dir) which is a correction calculated for each event, the source group containing all the event data will include a common_mode dataset with the common mode values. This allows users to recover the raw data from the calibrated data.

With psana-translate calibration is handled by external psana modules. These modules will produce calibrated data and psana-translate will find it and translate it to the hdf5 file. Understanding this flow of data is not necessary for automatic translation, however if users want to customize calibration, some understanding of how psana modules pass data through the event store, and are configured through config files is necessary.  The calibrated data will be distinguished from uncalibrated data with the use of a key in the event store. The key defaults to the value 'calibrated' but this is configurable through the psana.cfg file, in the section for the calibration module used. psana-translate provides special treatment for the calibration key. For psana-translate, the default value for the calibration key is 'calibrated' as well, but again, this is configurable through the psana.cfg file, in the section for Translator.H5Output. If psana-translate sees data with the key calibrated - it defaults to only translate data with the calibrated key and not the raw data. In the hdf5 file, one will find calibrated data where one would have otherwise found uncalibrated data. This is consistent with how o2o-translate translated calibrated cspad data. The calibrated key is not present in the hdf5 path names. This is different than what one finds for keys with ndarrays. For ndarrays the key is part of the h5 path name (see below).  The psana-translate option skip_calibrated can be set to true to get the uncalibrated data instead of calibrated data.

Calibration makes use of calibration constants - such as pedestals and pixel status. A key difference between psana-translate and o2o-translate is where these calibration constants are found, and the datatypes used to store them. For psana-translate they are found in the group CalibStore to the current configure group. For example, if we translate the first event in a run of the cxi tutorial data where we add the cspad calibration module before the psana-translate module:

psana -n 1 -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.output_file=calib.h5 exp=xpptut13:run=71

Then we will see

h5ls -r calib.h5 | grep -i "calibstore\|cspad"  # this command will include the following output

/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0/common_mode Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0/data Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0/element Dataset {1/Inf}
...
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.1/common_mode Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.1/data Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.1/element Dataset {1/Inf}
...
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PedestalsV1/XppGon.0:Cspad2x2.0/pedestals Dataset {185, 388, 2}
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PedestalsV1/XppGon.0:Cspad2x2.1/pedestals Dataset {185, 388, 2}
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PixelStatusV1/XppGon.0:Cspad2x2.0/status Dataset {185, 388, 2}
/Configure:0000/CalibStore/pdscalibdata::CsPadCommonModeSubV1/XppGon.0:Cspad2x2.0/data Dataset {SCALAR}
...

Things to note:

• There are common_mode datasets included with the data
• Both cspad sources have a pedestal dataset in CalibStore
• Only XppGon.0:Cspad2x2.0 has a common mode dataset in the calibstore.

When one looks at the common_mode dataset for XppGon.0:Cspad2x2.1, one sees the values are -10000, indicating no common mode calibration was done. The module loaded in this example to do calibration, cspad_mod.CsPadCalib, is equivalent to what o2o-translate would do. More information on o2o-translate calibration can be found at CsPad calibration in translator. Documentation on the CsPadCalib module is found in psana - Module Catalog.

An issue users may run into is understanding what calibration was done and recovering the raw data just from examining the hdf5 output. In the case of cspad, an understanding of the CsPadCalib module along with the what is in the hdf5 file does allow one to recover the uncalibrated data. This may not be possible with other calibration modules and detectors, in particular if nonlinear calibration algorithms are applied, such as applying a threshold.

Filtering Calibrated Data

By default, when the Translator sees both the original xtc data and a calibrated version of it, it writes the calibrated data in place of the xtc data. For automatic translation, we always load the module that will calibrate cspad if possible. If users do not want the calibrated data but prefer the raw data, they can set the option

skip_calibrated = true

If the user wants neither the calibrated nor the raw cspad, they they should use type filters, that is setting

Cspad = exclude
Cspad2x2 = exclude

There will be no cspad in the translation (including the cspad configuration data as well as event data). This can be useful if one is using other modules to produce ndarrays from the calibrated data and one only wants the final processed ndarrays in the translation.

PNCCD::FullFrame

This data is no longer translated. FullFrame is a copy of Frames with a more convenient interface for psana users, it is not considered to be as useful for hdf5 files. User's interested in having FullFrame written into their hdf5 files rather than the original Frames data should make a feature request.

Filtering Events

Since psana-translate runs as a psana module, it is possible to filter translated events through psana options and other modules. psana options allow you to start at a certain event, and process a certain number of events.  Moreover a user module that is loaded before the Translator module can tell psana that it should not pass this event on to any other modules, hence the Translator.H5Output module will never see the event and it will not get translated.

A downside of having modules loaded before the translator skip events is that updates to epics pv's, or configuration data will not get recorded. One may also wish to record the reason for filtering the event in the hdf5 file, as well as the event id's for the filtered events.  psana-translate provides an interface for doing these things. To use this mechanism, a module must put an object in the eventStore with a key that starts with

do_not_translate

For example, if a C++ module implements the event method to do the following:

  virtual void event(Event& evt, Env& env) {
	boost::shared_ptr<int> message = boost::make_shared<int>();        
    evt.put(message,"do_not_translate");
  }

Then none of the event data will get translated in any of the calib cycles.  The Translator will do the following

• For each calib cycle, it will make a filtered group
  • For instance, if the file has the group /Configure:0000/Run:0000/CalibCycle:0000, then it will also have:
    the group: /Configure:0000/Run:0000/Filtered:0000
• within each Filtered group, a time dataset that holds event id's of the filtered events.  

Suppose a user module has made some measurements that indicate this event should be filtered (for instance the beam energy is wrong). These measurements can be recorded in the hdf5 file by adding data to the event store that the Translator knows how to write.  As discussed below, the Translator can write ndarrays and strings as well as simple new types that user modules register. If a user module implements event to do the following:

// define user Module,

  virtual void event(Event& evt, Env& env) {
    boost::shared_ptr<std::string> message = boost::make_shared<std::string>("The beam energy is bad");        
    evt.put(message,"do_not_translate:message");
    const unsigned shape[1] = {4};
    boost::shared_ptr< ndarray<float,1> measurements = boost::make_shared< ndarray<float,1> >(shape);
    float *data = measurements->data();
    data[0] = 0.4;  data[1] = 1.3; data[2] = 2.2; data[3] = 3.1;
    evt.put(measurements,"do_not_translate:measurements");
  }

Note the use of the key "do_not_translate:xxx" the :xxx is not necessary, but it helps to uniquely quality the event data, and it will become a part of the 'src' group name where the do_not_translate event data is written to.  Since both std::string and a ndarray<float,1> are types that the Translator knows how to write, it will create the following groups and datasets in the hdf5 file:

• /Configure:0000/Run:0000/Filtered:0000/time  - this is as discussed above, the event id's for all filtered events
• /Configure:0000/Run:0000/Filtered:0000/std::string/noSrc__message/data  - this will be a dataset of variable length strings, each entry will be the string "The beam energy is bad"
• /Configure:0000/Run:0000/Filtered:0000/std::string/noSrc__message/time  - this will be a dataset of eventId's for the data above (there need not have been a std::string in all the filtered events).
• /Configure:0000/Run:0000/Filtered:0000/ndarray_float32_1/noSrc__measurements/data - this will be a dataset where each entry is a 1D array of 4 floats, with the values 0.4, 1.3, 2.2, 3.1
• /Configure:0000/Run:0000/Filtered:0000/ndarray_float32_1/noSrc__measurements/time - likewise the event ids for the ndarrays of the filtered events.

Note the src level group names: noSrc__mesage and noSrc__measurements. Since no source was specified with the calls to evt.put, the Translator starts with the string noSrc in the group name. Two underscores, __, separate the source from the keystring.

Note the fully qualified type information about the ndarray's written. This allows translation of different ndarrays in the event store that differ only by this type information (i.e: they have the same key and source).

Filtering from Python Modules

A Python module can use standard psana features to skip events as discussed above. It can also add any Python object into the event store that has the key "do_not_translate". This will create the Filtered:0000/time dataset as above. However to use the Translator filtering features that record user data, the Python module will have to add data that psana knows how to convert for C++ modules. Presently the only types that a Python module can add to the event store which will be seen by C++ modules are a number of ndarrays and str. A Python module will need to add one of these ndarray types or str to filter events.

Below is a complete example. First we make a release environment, and create a package for our Python Module that will filter events:

newrel ana-current myrel
cd myrel
newpkg mypkg
mkdir mypkg/src
sit_setup

Suppose we want to filter based on the photon count of the calibrated cspad from the CxiDs1.0:Cspad.0 source in the tutorial data. Rather then work with the quad's of the cspad, we will use an image producer module so that we can work with a 2D image. Further documentation on the various calibration modules, including image producers, can be found at psana - Module Catalog. We now add the following two files:

[psana]
modules = cspad_mod.CsPadCalib \
          CSPadPixCoords.CSPadImageProducer \
          mypkg.mymod \
          Translator.H5Output 
events = 10
files = exp=cxitut13:run=1150

[CSPadPixCoords.CSPadImageProducer]
source        = DetInfo(CxiDs1.0:Cspad.0)
key           = calibrated
imgkey        = image

[Translator.H5Output]
deflate=-1
shuffle=False
overwrite=True
output_file=cxitut13-run1150-filt.h5

and the file

import psana

class mymod(object):
    def __init__(self):
        self.threshold = self.configInt('threshold',176000000)
        self.source = self.configSrc('source','DetInfo(CxiDs1.0:Cspad.0)')

    def event(self, evt, env):
        image = evt.get(psana.ndarray_int16_2, 
                        self.source, 
                        'image')
	    if image is None: return
        photonCount = image[:].sum()
        if photonCount < self.threshold:
            self.skip()

After putting these files in place, and doing

scons

we can run this example by

psana -c trans.cfg

The configuration file trans.cfg sets up a module chain with 4 modules: cspad_mod.CsPadCalib, CSPadPixCoords.CSPadImageProducer, mypkg.mymod, Translator.H5Output. The first module calibrates all cspad it finds. The second module turns a cspad from a specific source into an image - placing quads and ascics in the correct position based on geometry. After these two modules, we load our own module - mypkg.mymod. Finally the Translator module runs last.

Reading through trans.cfg you will see how the raw cspad moves through the event store. The default behavior of cspad_mod.CsPadCalib is to place calibrated cspad in the Event with the key "calibrated". The CSPadPixCoords.CSPadImageProducer module has been told to look for the "calibrated" input key, for the source DetInfo(CxiDs1.0:Cspad.0) and produce an image with the key "image". 

In mymod.py, we see a class called mymod derived from object. It is important that the class name be the same as the file name. This is part of how psana finds the class. In the event, the module gets data of type psana.ndarray_int16_2. Identifying the correct type to use can be a challenge. Starting with code in event() that does

print evt.keys()

shows what the keys look like. After getting a valid image, a sum and simple threshold is performed.

Note the option

events = 10

in the psana section of the config file. This means one would translate 10 events in the data. This is just for testing and development. One would remove the option, or set it to 0 for a full translation. With events=10, after translation, if one does

h5ls -r cxitut13-run1150-filt.h5  | grep -i "ndarray"
/Configure:0000/Run:0000/CalibCycle:0000/ndarray_const_int16_2/CxiDs1.0:Cspad.0__image/data Dataset {5/Inf}

one sees that only 5 events were translated. The other 5 were skipped.

Another point to make about this example is that the cspad is effectively getting translated twice. The Translator is going to see the event keys:

EventKey(type=psana.CsPad.DataV2, src='DetInfo(CxiDs1.0:Cspad.0)')
EventKey(type=psana.CsPad.DataV2, src='DetInfo(CxiDs1.0:Cspad.0)', key='calibrated')
EventKey(type=psana.ndarray_int16_2, src='DetInfo(CxiDs1.0:Cspad.0)', key='image')

The Translator's default behavior is to treat the key 'calibrated'  as special. Since the first two keys differ only by the keystring 'calibrated', the Translator assumed the one marked 'calibrated' should replace the first in the translation. Hence it will not translate the raw cspad. It only translated the calibrated cspad. However the Translator does not know that the ndarray with key 'image' is a copy of the cspad. If one is only going to work with the 'image' array data and not the 'calibrated' cspad data, one could add the filtering option

Cspad=exclude

To the Translator.H5Output section of the config file. Then none of the cspad data will be translated (including both the configuration cspad object as well as event data) while the 'image' arrays will still be translated.

Filtering Types

The psana.cfg file accepts a number of parameters that will filter out sets of psana types.  For example setting

EBeam = exclude

would cause any of the types Psana::Bld::BldDataEBeamV0, Psana::Bld::BldDataEBeamV1, Psana::Bld::BldDataEBeamV2, Psana::Bld::BldDataEBeamV3 or Psana::Bld::BldDataEBeamV4 to be excluded from translation.

All types are translated by default. To exclude a few types, you can add lines like EBeam = exclude to the psana.cfg file. You can also list them with the type_filter parameter:

type_filter exclude EBeam Andor

The type_filter parameter is useful for including a few types:

type_filter include CsPad Frame

A shortcut is available to turn off translation of all the Xtc data:

type_filter exclude psana

One would use this to only translate user module data, such as ndarrays, strings and newly registered types.

The complete list of type aliases that users can use to filter is found in the default_psana.cfg file included below.

Src Filtering

Specific src's can be filtered by providing a list such as

src_filter = exclude NoDetector.0:Evr.2  CxiDs1.0:Cspad.0  CxiSc2.0:Cspad2x2.1  EBeam  FEEGasDetEnergy  CxiDg2_Pim

the syntax for a src in the filter list is what is supported by the Psana::Source class. This is a flexible syntax allowing for several ways to specify a src. It will match any detector or device number if this is not specified. See the section Psana Configuration File and all Options below for more details. If DAQ src aliases are present in the xtc file, these can be used for src filtering as well.  For example if the alias

acq01 -> SxrEndstation.0:Acqiris.0

is present, one can do

src_filter = exclude acq01

to exclude all data from the SxrEndstation.0:Acqiris.0 src.

Writing User Data

The translator will write NDarrays, C++ std::strings, and C++ types that the user registers.  Presently, registering new types is an advanced feature that requires familiarity with hdf5 programming.

NDArrays and Strings

ndarrays (up to dimension 4 of the standard integral types, floats and doubles) as well as std::string's that are written into the event store will be written to the hdf5 by default.  ndarrays can be passed to the Translator by Python modules as well as C++ modules. These events can be filtered as well.  The example in The XTC-to-HDF5 Translator above illustrates the group names used for ndarrays and strings. Note, the type group name for ndarrays is fully qualified by the template arguments, some examples of type names are

ndarray_int8_1           # a one dimensional array of 8 bit signed integers    (the C type char)
ndarray_uint8_2          # a two dimensional array of 8 bit unsigned integers
ndarray_int32_1          # a one dimensional array of 32 bit signed integers   (the C type int)
ndarray_uint64_3         # a 3D array of 64 bit unsigned integers   
ndarray_float32_2        # a 2D array of 32 bit floats  (the C type float)
ndarray_float64_1        # a 3D array of 64 bit floats (the C type double)

These names agree with what users find in the Python interface to psana.

Less common are the names used to store an ndarray of const data. An example name for such data is

ndarray_const_float32_2 

Fixed Dimensions vs. Variable Dimensions

The Translator defaults to using a fixed set of dimensions for all the ndarrays that go into the same dataset. The array received for the first data of the dataset determine these dimensions. For example, if from python one did

event.put(numpy.zeros((3,4),"mykey")

during the first event, but then

event.put(numpy.zeros((5,4),"mykey")

during the second event, the Translator would throw an error. Both of these arrays are supposed to go into an hdf5 path that ends with

/ndarray_float32_2/noSrc__mykey

but the underlying hdf5 type for this dataset has been set to a 2D array with dimensions (3,4). At present, one can start a new dataset during the second event

event.put(numpy.zeros((5,4),"mykey_larger")

to resolve this. In the near future, the Translator will support translation of ndarrays that vary only in the slow dimension to the same dataset. This feature will be activated with a modified key. One would prepend 'translate_vlen:' to the start of the keys. For example:

event.put(numpy.zeros((3,4),"translate_vlen:mykey")        # event one
event.put(numpy.zeros((5,4),"translate_vlen:mykey")        # event two

Now both ndarrays go to the same dataset, and the underlying hdf5 type changes to a vlen type of 1D arrays with dimension 4. The type name in the hdf5 path changes to indicate vlen, it will be

/ndarray_float32_2_vlen/noSrc__mykey

Registering New Types

C++ modules can register new types. Note, this is an advanced feature that requires familiarity with the Hdf5 programming in C.  Presently this feature is only suitable for simple types. An example is found in the file Translator/src/TestModuleNewWriter.cpp. We go through the example here. First a module will define the data type that they want to store. This type is a simple C struct of native types in the C language:

struct MyData {
  int32_t eventCounter;
  float energy;
};

Next, the module must define functions that create the hdf5 type for MyData, and fill a buffer to be written to the hdf5 file.  These functions must satisfy a particular signature:

  typedef hid_t (*CreateHDF5Type)(const void *userDataType);
  typedef const void * (*FillHdf5WriteBuffer)(const void *userDataType);

Here is what these functions might look like for MyData:

#include "hdf5/hdf5.h"
#include "MsgLogger/MsgLogger.h"

hid_t createMyDataHdf5Type(const void *) {
  static bool firstCall = true;
  static hid_t h5type = -1;
  if (not firstCall) return h5type;
  firstCall = false;
  h5type = H5Tcreate(H5T_COMPOUND, sizeof(MyData));
  
  herr_t status1 = H5Tinsert(h5type, "eventCounter", 
                             offsetof(MyData,eventCounter), 
                             H5T_NATIVE_UINT32);
  herr_t status2 = H5Tinsert(h5type, "energy", 
                             offsetof(MyData,energy), 
                             H5T_NATIVE_FLOAT);
  if ((h5type < 0) or (status1 < 0) or (status2<0)) {
    MsgLog("mydata",fatal,"unable to create MyData compound type");
  }
  MsgLog("mydata",trace,"Created hdf5 type for MyData  " << h5type);  
  return h5type;
}

const void * fillMyDataWriteBuffer(const void *data) {
  return data;
}

The function createMyDataHdf5Type must return an hdf5 type for MyData.  The void * that it is being passed will point to an actual instance of the MyData struct that was found in the eventStore.  Because MyData is so simple, the function createMyDataHdf5Type does not need to use this argument.  However a more complex type may include arrays of different sizes, and so the exact hdf5 type that describes the data cannot be determined without looking at the object.

The function fillMyDataWriteBuffer receives a void pointer to an instance of MyData that was found in the eventStore.  The function must then return a void pointer to a memory buffer that holds the data to be written into the hdf5 file.  Since MyData is so simply, the memory layout of the C++ object coincides with that of the hdf5 type, so we can simply return the original pointer to MyData. For more complex types, this will not be the case and fillMyDataWriteBuffer will have to manage a buffer of memory that persists after the function is called. It would then transfer the data in the complex C++ object into this memory buffer.

To register this new type for writing in the system, the user module must, in the beginJob() function, put a special object in the eventStore.  The Translator module will look for these special objects when it handles the beginJob() function. Then the user module can add MyData into the eventStore during the event() function:

#include "Translator/HdfWriterNew.h"

...
class TestNewHdfWriter : public Module {
public:
  TestNewHdfWriter(std::string moduleName) : Module(moduleName) {}
  virtual void beginJob(Event& evt, Env& env) {
    boost::shared_ptr<Translator::HdfWriterNew> newWriter = 
      boost::make_shared<Translator::HdfWriterNew>(&typeid(MyData), 
                                                   "data", 
                                                   createMyDataHdf5Type, 
                                                   fillMyDataWriteBuffer);
    evt.put(newWriter,"MyDataWriter");
  }
  
  virtual void event(Event& evt, Env& env) {
    boost::shared_ptr<MyData> myData = boost::make_shared<MyData>();
    myData->eventCounter = 11;
    myData->energy = 23.239;
    evt.put(myData,"example");
  }
};

The special type, HdfWriterNew, that is part of the Translator namespace, has the following arguments:

• the C++ std::type_info pointer for the new type being registered (&typeid(MyData) in the example)
• the name of the dataset ("data")
• the function that creates the hdf5 type (which we discussed above)
• the function that returns the memory buffer for writing (which we discussed above)

HdfWriterNew also takes an optional fifth argument that users can use to clean up resources.  Since MyData is so simple, there is no need to use this part of the API.  We will create the hdf5 type once, and not worry about closing it. 

The key "MyDataWriter" added when putting the newWriter in the event store is not important.  Giving it a distinct name can help debug problems that may arise in the Translator.

The translator, in each calib cycle, will make the following groups (for example in calib cycle 0):

• /Configure:0000/Run:0000/CalibCycle:0000/MyData/example
  • Note how the C++ type name, MyData, shows up in the path. 
  • Next the 'src' level group is based on the key "example" passed when putting myData in the event store.
• The dataset: /Configure:0000/Run:0000/CalibCycle:0000/MyData/example/data
  • The name "data" comes from the 2nd parameter to the HdfWriterNew object.
  • The dataset will be a 1D array of the hdf5 compound type with the fields
    • "eventCount"  uint32
    • "energy"  float

Psana Configuration File and all Options

When running the translator as a psana module, if is often convenient to create a psana.cfg file.  The Translator package include the file default_psana.cfg which is a psana configuration file that describes all the options possible, with extensive documentation as to what they mean.  Below we include this file for reference. To use this file, one could it and modify it. However it is not necessary to take the whole file - every value set is set to the default value. One could simply use this as a reference for those options values that one wants to change.

######################################################################
[psana]

# MODULES: any modules that produce data to be translated need be loaded 
# **BEFORE** Translator.H5Output (such as calibrated data or ndarray's)
# event data added by modules listed after Translator.H5Output is not translated.
modules = Translator.H5Output

files = **TODO: SPECIFY INPUT FILES OR DATA SOURCE HERE**

######################################################################
[Translator.H5Output]

# The only option you need to set for the Translator.H5Output module is
# output_file. All other options have default values (explained below).

# TODO: enter the full h5 output file name, including the output directory
output_file = output_directory/h5output.h5

# By default, the Translator will not overwrite the h5 file if it already exists
overwrite = false

# # # # # # # # # # # # # # # # # # # # #
# EPICS FILTERING
# The Translator can store epics pv's in one of three ways, or not at all.
# set store_epics below, to one of the following:
#
# updates_only   stores an EPICS pv when it has been updated in the psana epics store.
#                For xtc input this happens whenever EPICS data is present in a datagram.
#                Note - many EPICS pvs are not present in every shot. A dataset
#                for an EPIC pv is often shorter than the total number of events.
#                Experiments with many short calib cycles may have some calib cycles where
#                no EPICS pv's show up. Users would then have to look back through several 
#                calib cycle's to find the latest value of a pv.
#
# calib_repeat   This is the same as updates_only except that each calib cycle starts with
#                the most recent value of each pv. This makes it easier to find pv's in a 
#                calib cycle. For experiments with many short calib cycles, it can produce
#                more datasets than neccessary.
#
# always         For each event, store the most recent value of the EPICs pv. Produces 
#                longer datasets than neccessary, but makes it easier to find the latest
#                pv for an event.
#
# no             epics pv's will not be stored. You may also want to set Epics=exclude
#                (see below) if you do not want the epics configuration data stored

# The default is 'calib_repeat'

store_epics = calib_repeat

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# FILTERING
# 
# By default, all xtc data is Translated and many ndarrays that user modules (if any) 
# add are translated. Filtering can occur in either the code of user modules, or
# through options in the psana.cfg file. Here in the config file, different groups of 
# data can be filtered. There are four options for filtering data: 
#
#    type filtering   -  for example, exclude all cspad, regardless of the detector source
#    source filtering -  for example, exclude any data from a given detector source
#    key filtering    -  for example, include only ndarrays with a given key string
#    calibration      -  do not translate original xtc if a calibrated version is found
#
# Type filtering is based on sets of Psana data types. If you know what detectors or 
# devices to filter, leave type filtering alone and go to src_filter. 
#
# Type filtering has the highest precedence, then key filtering, then source 
# filtering, and lastly calibration filtering. When the Translator sees new data, 
# it first checks the type filter. If it is a filtered type (or unknown type) no further 
# translation occurs with the data - regardless of src or key. For data that gets 
# past the type filter, the Translator looks at the src and key. If the key 
# string is empty, it checks the source filter. Data with non empty key strings are 
# handled via the key filter. If the src is filtered, but the key is not, then the
# data will be translated. Data with the special calibration key string are handled 
# via the calibration filtering. 
#
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# TYPE FILTERING 
#
# One can include or exclude a class of Psana types with the following 
# options. Only the strings include or exclude are valid for these 
# type filtering options. 
# 
# Note - Epics in the list below refers only to the epicsConfig data
# which is the epics alias list, not the epics pv's. To filter the epics pv's
# see the 'store_epics' option above.

AcqTdc = include               # Psana::Acqiris::TdcConfigV1, Psana::Acqiris::TdcDataV1
AcqWaveform = include          # Psana::Acqiris::ConfigV1, Psana::Acqiris::DataDescV1
Alias = include                # Psana::Alias::ConfigV1
Andor = include                # Psana::Andor::ConfigV1, Psana::Andor::FrameV1
Arraychar = include            # Psana::Arraychar::DataV1
Control = include              # Psana::ControlData::ConfigV1, Psana::ControlData::ConfigV2, Psana::ControlData::ConfigV3
Cspad = include                # Psana::CsPad::ConfigV1, Psana::CsPad::ConfigV2, Psana::CsPad::ConfigV3, Psana::CsPad::ConfigV4, Psana::CsPad::ConfigV5, Psana::CsPad::DataV1, Psana::CsPad::DataV2
Cspad2x2 = include             # Psana::CsPad2x2::ConfigV1, Psana::CsPad2x2::ConfigV2, Psana::CsPad2x2::ElementV1
DiodeFex = include             # Psana::Lusi::DiodeFexConfigV1, Psana::Lusi::DiodeFexConfigV2, Psana::Lusi::DiodeFexV1
EBeam = include                # Psana::Bld::BldDataEBeamV0, Psana::Bld::BldDataEBeamV1, Psana::Bld::BldDataEBeamV2, Psana::Bld::BldDataEBeamV3, Psana::Bld::BldDataEBeamV4, Psana::Bld::BldDataEBeamV5, Psana::Bld::BldDataEBeamV6
Encoder = include              # Psana::Encoder::ConfigV1, Psana::Encoder::ConfigV2, Psana::Encoder::DataV1, Psana::Encoder::DataV2
Epics = include                # Psana::Epics::ConfigV1
Epix = include                 # Psana::Epix::ConfigV1, Psana::Epix::ElementV1, Psana::Epix::ElementV2
Epix100a = include             # Psana::Epix::Config100aV1
Epix10k = include              # Psana::Epix::Config10KV1
EpixSampler = include          # Psana::EpixSampler::ConfigV1, Psana::EpixSampler::ElementV1
Evr = include                  # Psana::EvrData::ConfigV1, Psana::EvrData::ConfigV2, Psana::EvrData::ConfigV3, Psana::EvrData::ConfigV4, Psana::EvrData::ConfigV5, Psana::EvrData::ConfigV6, Psana::EvrData::ConfigV7, Psana::EvrData::DataV3
EvrIO = include                # Psana::EvrData::IOConfigV1
Evs = include                  # Psana::EvrData::SrcConfigV1
FEEGasDetEnergy = include      # Psana::Bld::BldDataFEEGasDetEnergy, Psana::Bld::BldDataFEEGasDetEnergyV1
Fccd = include                 # Psana::FCCD::FccdConfigV1, Psana::FCCD::FccdConfigV2
Fli = include                  # Psana::Fli::ConfigV1, Psana::Fli::FrameV1
Frame = include                # Psana::Camera::FrameV1
FrameFccd = include            # Psana::Camera::FrameFccdConfigV1
FrameFex = include             # Psana::Camera::FrameFexConfigV1
GMD = include                  # Psana::Bld::BldDataGMDV0, Psana::Bld::BldDataGMDV1, Psana::Bld::BldDataGMDV2
GenericPgp = include           # Psana::GenericPgp::ConfigV1
Gsc16ai = include              # Psana::Gsc16ai::ConfigV1, Psana::Gsc16ai::DataV1
Imp = include                  # Psana::Imp::ConfigV1, Psana::Imp::ElementV1
Ipimb = include                # Psana::Ipimb::ConfigV1, Psana::Ipimb::ConfigV2, Psana::Ipimb::DataV1, Psana::Ipimb::DataV2
IpmFex = include               # Psana::Lusi::IpmFexConfigV1, Psana::Lusi::IpmFexConfigV2, Psana::Lusi::IpmFexV1
L3T = include                  # Psana::L3T::ConfigV1, Psana::L3T::DataV1, Psana::L3T::DataV2
OceanOptics = include          # Psana::OceanOptics::ConfigV1, Psana::OceanOptics::ConfigV2, Psana::OceanOptics::DataV1, Psana::OceanOptics::DataV2
Opal1k = include               # Psana::Opal1k::ConfigV1
Orca = include                 # Psana::Orca::ConfigV1
Partition = include            # Psana::Partition::ConfigV1
PhaseCavity = include          # Psana::Bld::BldDataPhaseCavity
PimImage = include             # Psana::Lusi::PimImageConfigV1
Pimax = include                # Psana::Pimax::ConfigV1, Psana::Pimax::FrameV1
Princeton = include            # Psana::Princeton::ConfigV1, Psana::Princeton::ConfigV2, Psana::Princeton::ConfigV3, Psana::Princeton::ConfigV4, Psana::Princeton::ConfigV5, Psana::Princeton::FrameV1, Psana::Princeton::FrameV2
PrincetonInfo = include        # Psana::Princeton::InfoV1
Quartz = include               # Psana::Quartz::ConfigV1
Rayonix = include              # Psana::Rayonix::ConfigV1, Psana::Rayonix::ConfigV2
SharedAcqADC = include         # Psana::Bld::BldDataAcqADCV1
SharedIpimb = include          # Psana::Bld::BldDataIpimbV0, Psana::Bld::BldDataIpimbV1
SharedPim = include            # Psana::Bld::BldDataPimV1
Spectrometer = include         # Psana::Bld::BldDataSpectrometerV0
TM6740 = include               # Psana::Pulnix::TM6740ConfigV1, Psana::Pulnix::TM6740ConfigV2
TimeTool = include             # Psana::TimeTool::ConfigV1, Psana::TimeTool::DataV1
Timepix = include              # Psana::Timepix::ConfigV1, Psana::Timepix::ConfigV2, Psana::Timepix::ConfigV3, Psana::Timepix::DataV1, Psana::Timepix::DataV2
TwoDGaussian = include         # Psana::Camera::TwoDGaussianV1
UsdUsb = include               # Psana::UsdUsb::ConfigV1, Psana::UsdUsb::DataV1
pnCCD = include                # Psana::PNCCD::ConfigV1, Psana::PNCCD::ConfigV2, Psana::PNCCD::FramesV1

# user types to translate from the event store
ndarray_types = include        # ndarray<int8_t,1>, ndarray<int8_t,2>, ndarray<int8_t,3>, ndarray<int8_t,4>, ndarray<int16_t,1>, ndarray<int16_t,2>, ndarray<int16_t,3>, ndarray<int16_t,4>, ndarray<int32_t,1>, ndarray<int32_t,2>, ndarray<int32_t,3>, ndarray<int32_t,4>, ndarray<int64_t,1>, ndarray<int64_t,2>, ndarray<int64_t,3>, ndarray<int64_t,4>, ndarray<uint8_t,1>, ndarray<uint8_t,2>, ndarray<uint8_t,3>, ndarray<uint8_t,4>, ndarray<uint16_t,1>, ndarray<uint16_t,2>, ndarray<uint16_t,3>, ndarray<uint16_t,4>, ndarray<uint32_t,1>, ndarray<uint32_t,2>, ndarray<uint32_t,3>, ndarray<uint32_t,4>, ndarray<uint64_t,1>, ndarray<uint64_t,2>, ndarray<uint64_t,3>, ndarray<uint64_t,4>, ndarray<float,1>, ndarray<float,2>, ndarray<float,3>, ndarray<float,4>, ndarray<double,1>, ndarray<double,2>, ndarray<double,3>, ndarray<double,4>, ndarray<const int8_t,1>, ndarray<const int8_t,2>, ndarray<const int8_t,3>, ndarray<const int8_t,4>, ndarray<const int16_t,1>, ndarray<const int16_t,2>, ndarray<const int16_t,3>, ndarray<const int16_t,4>, ndarray<const int32_t,1>, ndarray<const int32_t,2>, ndarray<const int32_t,3>, ndarray<const int32_t,4>, ndarray<const int64_t,1>, ndarray<const int64_t,2>, ndarray<const int64_t,3>, ndarray<const int64_t,4>, ndarray<const uint8_t,1>, ndarray<const uint8_t,2>, ndarray<const uint8_t,3>, ndarray<const uint8_t,4>, ndarray<const uint16_t,1>, ndarray<const uint16_t,2>, ndarray<const uint16_t,3>, ndarray<const uint16_t,4>, ndarray<const uint32_t,1>, ndarray<const uint32_t,2>, ndarray<const uint32_t,3>, ndarray<const uint32_t,4>, ndarray<const uint64_t,1>, ndarray<const uint64_t,2>, ndarray<const uint64_t,3>, ndarray<const uint64_t,4>, ndarray<const float,1>, ndarray<const float,2>, ndarray<const float,3>, ndarray<const float,4>, ndarray<const double,1>, ndarray<const double,2>, ndarray<const double,3>, ndarray<const double,4>
std_string = include           # std::string


# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# TYPE FILTER SHORTCUT
#
# In addition to filtering Psana types by the options above, one can use
# the type_filter option below. For example:
#
# type_filter = include cspad       # will only translate cspad types. Will not translate
#                                 # ndarrays or strings
# type_filter = exclude Andor evr   # translate all except the Andor or Evr types
# 
# If you do not want to translate what is in the xtc file, use the psana shortcut:
#
# type_filter = exclude psana       # This will only translate ndarray's and strings 
#
# Likewise doing:
#
# type_filter = include psana       # will translate all xtc data, but skip any ndarray's or strings
#
# The default is to include all

type_filter = include all

# note - if type_filter is anything other than 'include all' it takes precedence
# over the classes of type filter options above, like Cspad=include.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# SOURCE FILTERING
#
# The default for the src_filter option is "include all"
# If you want to include a subset of the sources, do
#
# src_filter include srcname1 srcname2  
#
#  or if you want to exclude a subset of sources, do
#
# src_filter exclude srcname1 srcname2
#
# The syntax for specifying a srcname follows that of the Psana Source (discussed in 
# the Psana Users Guide). The Psana Source recognizes DAQ alias names (if present
# in the xtc files), several styles for specifying a Pds Src, as well as detector matches 
# where the detector number, or device number is not known.
# 
# Specifically, format of the match string can be:
#
#       DetInfo(det.detId:dev.devId) - fully or partially specified DetInfo
#       det.detId:dev.devId - same as above
#       DetInfo(det-detId|dev.devId) - same as above
#       det-detId|dev.devId - same as above
#       BldInfo(type) - fully or partially specified BldInfo
#       type - same as above
#       ProcInfo(ipAddr) - fully or partially specified ProcInfo
#
# For example
#        DetInfo(AmoETOF.0.Acqiris.0)  
#        DetInfo(AmoETOF.0.Acqiris)  
#        DetInfo(AmoETOF:Acqiris)
#        AmoETOF:Acqiris
#        AmoETOF|Acqiris
#
# will all match the same data, AmoETOF.0.Acqiris.0. The later ones will match
# additional data (such as detector 1, 2, etc.) if it is present.
#
# A simple way to set up src filtering is to take a look at the sources in the 
# xtc input using the psana EventKeys module.  For example
#
# psana -n 5 -m EventKeys exp=cxitut13:run=22 
#
# Will print the EventKeys in the first 5 events.  If the output includes
#
#   EventKey(type=Psana::EvrData::DataV3, src=DetInfo(NoDetector.0:Evr.2))
#   EventKey(type=Psana::CsPad::DataV2, src=DetInfo(CxiDs1.0:Cspad.0))
#   EventKey(type=Psana::CsPad2x2::ElementV1, src=DetInfo(CxiSc2.0:Cspad2x2.1))
#   EventKey(type=Psana::Bld::BldDataEBeamV3, src=BldInfo(EBeam))
#   EventKey(type=Psana::Bld::BldDataFEEGasDetEnergy, src=BldInfo(FEEGasDetEnergy))
#   EventKey(type=Psana::Camera::FrameV1, src=BldInfo(CxiDg2_Pim))
#
# Then one can filter on these six srcname's:
#
#  NoDetector.0:Evr.2  CxiDs1.0:Cspad.0  CxiSc2.0:Cspad2x2.1  EBeam  FEEGasDetEnergy  CxiDg2_Pim
#

src_filter = include all

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# CALIBRATION FILTERING
#
# Psana calibration modules can produce calibrated versions of different 
# data types. Depending on the module used, you may get an NDArray, an 
# image, or the same data type as was in the xtc but with calibrated data.
#
# If you are doing the latter, the module output will be data of the same type 
# and src as the uncalibrated data, with an additional key, such as 'calibrated'.
# If these modules are configured to use a different key, set calibration_key
# below accordingly:

calibration_key = calibrated

# The Translator defaults to writing calibrated data in place of uncalibrated
# data. If you do not want the calibrated data and would prefer to have the
# original uncalibrated data from the xtc, then set skip_calibrated to true.

skip_calibrated = false

# note, setting skip_calibrated to true will force sets exclude_calibstore 
# (below) to be true as well.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# CALIBSTORE FILTERING
#
# Calibration modules may publish the data they used to produce the calibrated
# event objects. Examples of data would be pedestal values, pixel status (what
# pixels are hot) and common mode algorithm parameters. This data will be published
# in what is called the Psana calibStore. When the Translator sees calibrated 
# event data, it will look for the corresponsinding calibStore data as well.
# If you do not want it to translate calibStore data, set the following to true.

exclude_calibstore = false

# otherwise, the Translator will create a group CalibStore that holds the
# calibstore data. Note, the Translator looks for all calibStore data associated 
# with the calibration modules. If a calibration module was configured to not do 
# certain calibrations (such as gain) but the module still put gain values
# in the config store (even though it did not use them) the Translator 
# would still translate those gain values.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# KEY FILTERING
#
# Psana modules loaded before the translator may put a variety of objects in the event 
# store. Be default, the Translator will translate any new data that it knows about.
# In addition to the psana types, it knows about NDArrays, C++ strings, and has a C++ interface 
# for registering new simple types. NDarray's up to 4 dimensions of 10 basic types 
# (8, 16, 32 and 64 bit signed and unsigned int, float and double) as well as the const 
# versions of these types are translated.
#
# Generally Psana modules will attach keys to these objects (the keys are simply strings).
# To filter the set of keys that are translated, modify the parameter below:

key_filter = include all

# The default is to not look at the key but rather translate all data that the translator
# knows about. An example of including only data with the key finalanswer would be
#
# key_filter = include finalanswer
#
# To exclude a few keys, one can do
#
# key_filter = exclude arrayA arrayB
#
# Note, key filtering does not affect translation of data without keys. For instance
# setting key_filter = include keyA does not turn off translation of data without keys.
# Of all the data with keys, only those where the key is keyA will be translated.
#

# ---------------------------------------
# SPLIT INTO SEPARTE HDF5 FILES BASED ON CALIB CYCLES
#
# There are two reasons to split the Translator output into separate files based on
# calib cycles. The first is to reduce the size of the hdf5 files, and the second is
# to speedup translation by translating separate calib cycles in parallel. The default 
# is to not split:

split = NoSplit

# however the Translator also supports SplitScan mode. The best way to invoke this is to
# run the separate driver program
#
#  h5-mpi-translate
#
# which requires MPI to be available in the environment, but see below for the non-MPI 
# based split scan mode. In SplitScan mode, in addition to the output File, separate files will 
# be made for the calib cycles. The output file (the master file) will include external links 
# to the other files. Several mpi jobs are run simultaneously to divide the work of creating the 
# calib cycle files. For example, running six jobs to produce out.h5 might look like:
#
#   mpirun -n 6 h5-mpi-translate -m Translator.H5Output -o Translator.H5Output.output_file=out.h5 exp=xppd9714:run=16
#
# The driver program, h5-mpi-translate, takes all arguments that psana takes.
# Presently with our implementation, one must give the full path to mpirun, and run
# a driver program that sets up the analysis environment. For example, 
#
#   /reg/g/psdm/sw/releases/ana-current/arch/x86_64-rhel6-gcc44-opt/bin/mpirun -n 6 mpilaunch h5-mpi-translate -m Translator.H5Output -o Translator.H5Output.output_file=out.h5 exp=xppd9714:run=16
#
# where mpilaunch is
#
#  #!/bin/bash
#  . /reg/g/psdm/bin/sit_setup.sh ana-current
#  $@
#
# If six jobs were used, one becomes the master process and the other five are the workers.
# The master process does two things. First it writes the file out.h5 with the external links 
# to the calib files. Second it reads through all the data and finds the calib cycles. When it
# finds a calib cycle, it tells the next available worker where this is. When a worker is done,
# it tells the master process. The master process than adds all neccessary external links from
# out.h5 to the translated calib file produced by the worker.
#
# Generally, there will be one calib cycle file for each calib cycle. However to prevent to many 
# calib cycle files from being produced for experiments that have only a few events per calib cycle, 
# an option controls the minumum number of events per external calib cycle file. The default is

# min_events_per_calib_file = 100

# For example, if there are only 10 events per calib cycle, and assuming the master file is called 
# out.h5, the file output_cc0000.h5 will contain the groups 
#
# /CalibCycle:0000
# /CalibCycle:0001
# ...
# /CalibCycle:0009
#
# and the file output_cc0010.h5 will start with group /CalibCycle:0010
#
# As mentioned above, when workers finish a calib cycle file, they send a message to the master. 
# How frequently the master stops reading through the data to check for these messages is controlled 
# by the following parameter

# num_events_check_done_calib_file = 120

# that is, it defaults to check for a 'done' message from a worker every 120 events.
#
# When running the h5-mpi-translate and specifying user psana modules (perhaps to add ndarrays
# into the translation or dynamically filter events) it is important to note that these modules
# are restarted for each calib cycle file. That is these modules will have their beginJob/endJob
# and beginRun/endRun routines called for each calib file that a worker produces.
#
# If MPI is not available, the Translator supports an additional  split scan mode by setting the 
# split option as follows:
#
# split=SplitScan
#
# In this mode, there is no communication between jobs, and each Translator job reads through all 
# the input, so launching too many jobs will significantly increase the amount of input processing. 
# Dividing the work of this SplitScan mode is done with the parameters

# jobTotal = 1
# jobNumber = 0

# which default to 1 job that is numbered 0. However if jobTotal=3 and jobNumber=1, this 
# Translator will process calibCycle 1, 4, 7, etc. If jobTotal is 3, the user MUST
# make sure to launch 3 Translator jobs with jobNumber being 0,1 and 2 to get all the calib cycle
# files written. jobNumber=0 will write the master file with the external links to the calib
# cycle files. 
#
# For example, the following two command lines:
#
# psana -m Translator.H5Output -o Translator.H5Output.output_file=mydir/split.h5 -o Translator.H5Output.split=SplitScan -o Translator.H5Output.jobNumber=0 -o Translator.H5Output.jobTotal=2 exp=xpp123:run=10
# psana -m Translator.H5Output -o Translator.H5Output.output_file=mydir/split.h5 -o Translator.H5Output.split=SplitScan -o Translator.H5Output.jobNumber=1 -o Translator.H5Output.jobTotal=2 exp=xpp123:run=10
#
# will divide the work into two translator jobs. When they finish, the output will be
# 
# mydir/split.h5
# mydir/split_cc0000.h5
# mydir/split_cc0001.h5
# ...
#
# note, this split scan mode, unlike the MPI version, does not put multiple calib cycles in
# one file. One can get a large number of files if calib cycles only contain a few events.

# The remaining values for split are
#
# split=MPIWorker
# split=MPIMaster
#
# These values are meant to be set by h5-mpi-translate. Generally users should not set
# these values. The h5-mpi-translate driver will set additional options that we 
# document here (again, users should not need to set these options). These options include the
# two discussed above:
#
#   num_events_check_done_calib_file 
#   min_events_per_calib_file
#
# as well as the option
#
# first_calib_cycle_number
#
# which is a 0-up counter for the first calib cycle that the MPIWorker will see.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
# COMPRESSION 
#
# The following options control compression for most all datasets.
# Shuffling improves compression for certain datasets. Valid values for
# deflate (gzip compression level) are 0-9. Setting deflate = -1 turns off
# compression.

shuffle = true
deflate = 1

# if deflate is set to -1, set shuffle to false, as it performs no function without compression.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# TECHNICAL, ADVANCED CONFIGURATION
# 
# ---------------------------------------
# CHUNKING
# The commented options below give the default chunking options.
# Objects per chunk are selected from the target chunk size (16 MB) and 
# adjusted based on min/max objects per chunk, and the max bytes per chunk.
# It is important that the chunkCache (created on a per dataset basis) be 
# large enough to hold at least one chunk, ideally all chunks we need to have
# open at one time when writing to the dataset (usually one, unless we repair
# split events):
 
# chunkSizeTargetInBytes = 1703936 (16MB)
# chunkSizeTargetObjects = 0 (0 means select objects per chunk from chunkSizeInBytes)
# maxChunkSizeInBytes = 10649600  (100MB)
# minObjectsPerChunk = 50              
# maxObjectsPerChunk = 2048
# chunkCacheSizeTargetInChunks = 3
# maxChunkCacheSizeInBytes = 10649600  (100MB)

# ---------------------------------------
# REFINED DATASET CONTROL
#
# There are six classes of datasets for which individual options for shuffle,
# deflate, chunkSizeTargetInBytes and chunkSizeTargetObjects can be specified:
#
# regular (most everything, all psana types)
# epics (all the epics pv's)
# damage (accompanies all regular data from event store)
# ndarrays (new data from other modules)
# string's (new data from other modules)
# eventId (the time dataset that also accompanies all regular data, epics pvs, ndarrays and strings)
#
# The options for regular datasets have been discussed above. The other five datasets 
# get their default values for shuffle, deflate, chunkSizeInBytes and chunkSizeInObjects
# from the regular dataset options except in the cases below:
 
# damageShuffle = false
# stringShuffle = false
# epicsPvShuffle = false
# stringDeflate = -1
# eventIdChunkSizeTargetInBytes = 16384
# epicsPvChunkSizeTargetInBytes = 16384

# The rest of the shuffle, deflate and chunk size options for the other five datasets are:
#
# eventIdShuffle = true
# eventIdDeflate = 1
# damageDeflate = 1
# epicsPvDeflate = 1
# ndarrayShuffle = true
# ndarrayDeflate = 1
# eventIdChunkSizeTargetObjects = 0
# damageChunkSizeTargetInBytes = 1703936
# damageChunkSizeTargetObjects = 0
# stringChunkSizeTargetInBytes = 1703936
# stringChunkSizeTargetObjects = 0
# ndarrayChunkSizeTargetInBytes = 1703936
# ndarrayChunkSizeTargetObjects = 0
# epicsPvChunkSizeTargetObjects = 0

# ---------------------------------------
# SPLIT EVENTS
# When the Translator encounters a split event, it checks a cache to see
# if it has already seen it.  If it has, it fills in any blanks that it can.
# To prevent this cache from growing to large, set the maximum number of
# split events to look back through here (default is 3000):

max_saved_split_events = 3000

Translation and Damage

psana has a specific damage policy that tells it what damaged data is acceptable for psana modules and what data is not. The default behavior is

• configStore - only undamaged data is stored in the configStore
• EventStore - undamaged data, and EBeam data with user damage is stored in the event, all other damage is not stored
  
  

psana-translate records event ids and damage for any xtc data that passes psana's damage policy. So by default, damaged config objects, and damaged events (other then user damaged EBeam data) are not translated. This deviates slightly from what o2o-translate would translate.  o2o-translate would also store out of order damaged event data.  There is a psana option that can be added to the [psana] section of the .cfg file to recover this behavior.  Below we document some special options that control what damaged data psana stores:

• store-out-of-order-damage  - defaults to false, set to true if you want to translate out of order damaged data
• store-user-ebeam-damage  - defaults to true, set to false if you do not want to translate EBeam data that only has user damage
• store-damaged-config - defaults to false, set to true if you want to store damaged config data

Difference's with o2o-translate

Here we cover differences with o2o-translate that we expect will be minor and not affect user code.

Feature's Dropped from o2o-translate

hdf file creation parameters
Only NoSplit is implemented - no family or split drivers.

In general a number of o2o-translate options are no longer supported.  In particular:
-G (long names like CalibCycle:0000 instead of CalibCycle) is always on.

Technical Difference's with o2o-translate

Below is a list of technical differences between psana-translate and o2o-translate. These differences should not affect users.

• File attributes runNumber, runType and experiment not stored, instead expNum, experiment, instrument and jobName are stored (from the psana Env object)
• The attribute :schema:timestamp-format is always "full", there is no option for "short"
• The output file must be explicitly specificed in the psana cfg file. It is not inferred from the input.
• The File attribute origin is now psana-translator as opposed to translator
• The end sec and nanoseconds are not written into the Configure group at the end of the job as there is no EventId in the Event at the end.
• integer size changes - a number of fields have changed size, a few examples are below.  In one quirky case, this caused translation to be different.  The reason was that the data was uninitialized, and the new 32 bit value was different than the old 16 bit value. Data produced from 2014 onward will not include unitialized data in the translation, users will not have to worry about.  Unitialized data is very rare in pre 2014 data and, due to its location, not likely to be used in analysis.
• A few Examples of field size changes:
  • EvrData::ConfigV7/seq_config - sync_source - enum was uint16, now uint32
  • EvrData::ConfigV7/seq_config - beam_source - enum was uint16, now uint32
  • Ipimb::DataV2 - source_id was uint16, now uint8
  • Ipimb::DataV2 - conn_id was uint16 now uint8
  • Ipimb::DataV2 - module was uint16, now uint8

Some types have their field names rearranged. For example with ControlData::ConfigV2 one has:

ControlData::ConfigV2:
  o2o: uses_duration uses_events duration events npvControls npvMonitors npvLabels
psana: events uses_duration uses_events duration npvControls npvMonitors npvLabels

EvrData::ConfigV7:
  o2o: code isReadout isCommand isLatch reportDelay reportWidth releaseCode maskTrigger maskSet maskClear desc readoutGroup
psana: code isReadout isCommand isLatch reportDelay reportWidth maskTrigger maskSet maskClear desc readoutGroup releaseCode

Epics Ctrl datasets (in the configure group as opposed to the calib group) are not chunked.  They are stored as fixed size datasets depending on the number of pv's.

Only one epics pv is stored per name (of course, one epics pv may have any number of elements within it). This is fine as the epic pv name is supposed to uniquely identify the pv.  However in xtc files, you can see several epics pv's with the same pvname, but different pvid's. This scenario should only arise when the same pv is coming from different sources, and replicates the same data.  Psana only stores one epics pv per name (the last one it sees in a datagram). This is the one that psana-translate will pick up and store.

All Epics pv's are stored in the source folder EpicsArch.0:NoDevice.0.  With o2o-translate, some could be split off into other folders (such as AmoVMI.0:Opal1000.0). As epics pv names uniquely identify the data, the source information should not be needed.

Some DAQ config objects include space for a maximum number of entries.  o2o-translate would only write entries for those used, not the maximum entries.  psana-translate does not.  For example:

• The Acqiris::ConfigV1 vert dataset now always prints the max of 20 channels, even if the user will only be using 3.
  • Note, in this case the Acqiris data will still only include the 3 channels being used. o2o-translate was making an adjustment to the config data being written.

psana-translate will write an emtpy output_lookup_table for the Opal1k::ConfigV1 dataset named output_lookup_table, even if output_lookup_table() is enabled.  o2o-translate would not.

psana-translate does not produce the _dim_fix_flag_20103 datasets that o2o-translate did.

Bld::BldDataGMDV  the field fSpare1 has been dropped from this type.

With psana-translate, if all the xtc's coming from a particular source are damaged, you will not see a 'data' dataset in the hdf5 file. You will see the time, _damage and _mask datasets that tell the damage and events where the omitted data lives. o2o-translate may have created a 'data' dataset filled with blanks.

As discussed above, OutOfOrder Damage is not translated by default. o2o-translate translated out of order damage, however psana-translate does not.  psana can be told to include this kind of damaged data by setting store-out-of-order-damage=true in the [psana] section of your .cfg file.

When the number of events is recorded in the control data, o2o-translate would set the chunk size based on this value.  psana-translate does the same.  However o2o-translate also looked at the actual number of events and used this as well to set chunk sizes in future calib cycles.  psana-translate does not do this latter part.

Speed

Comparison with o2o-translate

psana-translate runs about 10% slower than o2o-translate does.

Performance testing was done during November/December of 2013.  Both o2o-translate and psana-translate worked through a 92 GB xtc file using compression=1 on the rhat6 machine psdev105.  They read and wrote the data from /u1. They both used the non-parallel compression library.  o2o-translate produced a 68GB file in 65 minutes and psana-translate produced a 65GB file in 70 minutes.  (Speeds of about 22MB/sec).  Production runs will use the parallel compression library and are faster.

Single psana-translate

I parsed through psana-translate logs on 9/25/2014. An estimate of the median speed is 51MB/sec. The 25th and 75th percentile speeds look to be 37MB/sec and 69MB/sec. I've made some effort to parse logs that only include full translations with the default compression. Among such translations, I would expect the differences on speeds to be based on how much cspad is being calibrated, the load on the filesystem and cores, and how much damaged data appears and is not processed (will make the speed seem artificially higher). I assume that the size of the xtc files on disk is the amount of data translated.

A limitation on translation speed is the time it takes psana to read through all the data. I estimate this to be around 250MB/sec, putting translation at 5 times psana reading. I believe the file system supports 300-400MB/sec, but this depends heavily on the load, however this may not be a good benchmark for psana. A better benchmarch is a simple program that parses through the xtc - touching all the payloads.

Timing MPI Split Scan Translation

Below is a table of some testing results for the MPI split scan translator. All of this was carried out on the offline file system. A typical command was

bsub -q psanaq -a mympi -n 12 -R "span[ptile=1]" h5-mpi-translate -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.output_file=/reg/data/ana01/temp/davidsch/out.h5 exp=xpptut13:run=3

With the environment variables set to load the parallel compression library. All jobs run on the psanaq and write to ana01. Two of the jobs only had one calib cycle (xpp74813:run=69 and xpp40312:run=48) from which we might say the baseline translation speeds were 25-30 mb/sec in these tests. When investigating why this was less than the median 51MB/sec, I did see a high load on the system, however it may also be the overhead of having two readers. When running mpi split scan on a run with only one calib cycle - two different nodes will hit the same xtc on disk for a while. At some point the master process will get further along in the files. However when I ran a normal psana-translate job on the system I saw a 84MB/sec translation.

Some of the columns are explained here:

• WJobs - how many worker jobs. If there are at least 100 events in a calib cycle, it is one calib cycle per worker job. Otherwise as many calib cycles as needed to hit 100.
• CC/WJ - average number of calib cycles per worker job. Usually 1.
• mread - time for the master to read through the data. The rest of the time it waits for worker nodes to finish.
• calib - how many distinct cspad sources were calibrated in the h5.
• WJtime - average time for one worker job. In principle, something close to mread + WJtime would be the minimum time we expect to achieve.
  *wn - for the n workers, report the number of worker jobs done, and the percent of the total time the worker was translating, vs. idly waiting for a worker job. For example, w0=19/96% means worker 0 processed 19 worker jobs in 96% of the total time.