Table of Contents

...

Production: o2o-translate

Presently there are two xtc to hdf5 translators. These are o2o-translate and psana-translate. o2o-translate is being phased out will be replaced with psana-translate. Presently o2o-translate is presently being used in production. The automatic hdf5 translation that users can execute from the web portal uses o2o-translate. psana-translate is part of the analysis release and can be run directly. It will replace o2o-translate for automatic translation soon. This document covers psana-translate. Documentation on o2o-translate, which discusses some history with regards to selecting hdf5 for a scientific data format for general use can be found

...

• Datasets need not be aligned.  That is the 5th image in a detector dataset may come from a different event than the 5th record in a gas detector dataset. One can match up records from different datasets by use the time datasets.
• One should use the _mask datasets to identify valid data. A _mask dataset record is 1 when the corresponding record of the data dataset if valid, 0 if it is not. When the _mask record is 0, the data record will be all zeros and should not be processed.
• The hdf5 group hierarchy has the following levels: run, calib cycle. type, source  - regular event data is organized into datasets that live at the source level, epics has its own place, and configuration data (that usually arrives once) as its own place as well.

Future: psana-translate (BETA Software)

psana-translate is part of the analysis release. It will replace o2o-translate for automatic translation as soon as possible. Presently it is BETA software and subject to change – in particular the output schema may change to better accommodate new features.

The rest of this document covers psana-translate. psana-translate runs as a psana module. As such, we have been able to develop several new features that will be discussed below. However the main technical reason for phasing out o2o-translate is to use a Data Description Language (DDL) to generate code that handles the many data types that different detectors produce. This use of DDL is part of psana-translate.Below we discuss

Running psana-translate

...

You can run psana-translate as you would any other psana module. Either through psana command line options or by writing a psana configuration file.  The module is Translator.H5Output.

...

If this poses a problem for updating code written for o2o-tranlsate hdf5 files, note that the hdf5 library provides a way to identify soft links. All high level interfaces (such as h5py, pytables, Matlab, Octave or IDL) should provide this as well.

One can also exclude these soft links by adding

create_alias_links = false

to the \[Translator.H5Output\] section of the psana config file.

Calibrated Data

o2o-translate knows how to calibrate CsPad data. If o2o-translate was told where a calib-dir was, and calibration constants were deployed (that is written into this calib-dir) then o2o-translate would calibrate cspad data and write the calibrated data instead of the raw xtc data - in the same place where the raw xtc would have gone. It would also write the calibration data used in a special group, and include the common mode values (if calculated, this depends on what files are deployed to the calib-dir) with the calibrated cspad data. This allows users to recover the raw data from the calibrated data.

...