...

Translation from the LCLS data format of XTC to the more general scientific format  HDF5 is carried out by the XTC to HDF5 Translator. We refer to the software that carries out the translation as psana-translate. This distinguishes it from the first earlier version of the software called o2o-translate . o2o-translate is no longer being maintained. While it is still possible to run o2o-translate, it will not understand the latest types of data used in LCLS experiments(which has been deprecated). Translation for experiments should generally be carried out by using the automatic hdf5 translation feature of the Web Portal of Experiments. Below features are discussed that allow for customization of the translation - filtering unwanted events or detectors as well as adding processed data. The web portal provides a mechanism to customize the translation using these features. Experiment POC's should be able to help with using these features. Documentation on o2o-translate, which discusses some history with regards to selecting hdf5 for a scientific data format for general use can be found

...

A Python module can use standard psana features to skip events as discussed above. It can also add any Python object into the event store that has the key "do_not_translate". This will create the Filtered:0000/time dataset as above. However to use the Translator filtering features that record user data, the Python module will have to add data that psana knows how to convert for C++ modules. Presently the only types that a Python module can add to the event store which will be seen by C++ modules are a number of ndarrays and str. A Python module will need to add one of these ndarray types or str to filter events.

Below is a complete example. First we make a release environment, and create a package for our Python Module that will filter events:

newrel ana-current myrel
cd myrel
newpkg mypkg
mkdir mypkg/src

Suppose we want to filter based on the photon count of the calibrated cspad from the CxiDs1.0:Cspad.0 source in the tutorial data. Rather then work with the quad's of the ndarray cspad, we will use an image producer module so that we can work with a 2D image. Further documentation on the various calibration modules, including image producers, can be found at psana - Module Catalog. We now add the following two files:

[psana]
modules = cspad_mod.CsPadCalib \
          CSPadPixCoords.CSPadImageProducer \
          mypkg.mymod \
          Translator.H5Output 
events = 10
files = exp=cxitut13:run=1150

[CSPadPixCoords.CSPadImageProducer]
source        = DetInfo(CxiDs1.0:Cspad.0)
key           = calibrated
imgkey        = image
tiltIsApplied = false
print_bits    = 0

[Translator.H5Output]
deflate=-1
shuffle=False
overwrite=True
output_file=cxitut13-run1150-filt.h5

and the file

import psana
class mymod(object):
    def __init__(self):
        self.threshold = self.configInt('threshold',176000000)
        srcStr = self.configStr('source','DetInfo(CxiDs1.0:Cspad.0)')
        self.source = psana.Source(srcStr)
    def event(self, evt, env):
        image = evt.get(psana.ndarray_int16_2, 
                        self.source, 
                        'image')
	    if image is None: return
        photonCount = image[:].sum()
        if photonCount < self.threshold:
            self.skip()

After putting these files in place, and doing

scons

we can run this example by

psana -c trans.cfg

The configuration file trans.cfg sets up a module chain with 4 modules: cspad_mod.CsPadCalib, CSPadPixCoords.CSPadImageProducer, mypkg.mymod, Translator.H5Output. The first module calibrates all cspad it finds. The second module turns a cspad from a specific source into an image - placing asics in the correct position based on geometry. After these two, we load our own module - mypkg.mymod. Finally the Translator runs last.

Reading through trans.cfg you will see how the raw cspad moves through the event store. The default behavior of cspad_mod.CsPadCalib is to place calibrated cspad in the Event with the key "calibrated". The CSPadPixCoords.CSPadImageProducer module has been told to look for the "calibrated" input key, for the source DetInfo(CxiDs1.0:Cspad.0) and produce an image with the key "image". 

In mymod.py, we see a class called mymod derived from object. It is important that the class name be the same as the file name. This is part of how psana finds the class. In the event, the module gets data of type psana.ndarray_int16_2. Identifying the correct type to use can be a challenge. Starting with code in event() that does

print evt.keys()

shows what the keys look like. After getting a valid image, a sum and simple threshold is performed.

Note the option

events = 10

in the psana section of the config file. This means one would translate 10 events in the data. After translation, if one does

h5ls -r cxitut13-run1150-filt.h5  | grep -i "ndarray"
/Configure:0000/Run:0000/CalibCycle:0000/ndarray_const_int16_2/CxiDs1.0:Cspad.0__image/data Dataset {5/Inf}

one sees that only 5 events were translated. The other 5 were skipped will be recorded in the hdf5 file.

Filtering Types

The psana.cfg file accepts a number of parameters that will filter out sets of psana types.  For example setting

...