The easiest way to try different translator options to write a psana.cfg file. Copy the file default_psana.cfg that is included below or from the Translator package directory and modify option values that you wish. The file default_psana.cfg includes extensive documentation on all the translator options.

New Features

With psana-translate, you can

filter out whole events from translation
filter out certain data, by data type, or by data source, or key string
write ndarray's that other modules add to the event store
write std::string's that other C++ modules add to the event store
advanced: have a C++ module register a new type for translation

New Features Subject to Change

Three aspects of these new features are subject to change. These are highlighted in warning boxes below. In brief, these are

hdf5 group names for ndarray types
How event key strings are incorporated into hdf5 paths
How new C++ types are registered for translation.

Important Changes between o2o-translate and psana-translate

The translation that psana-translate produces is most always backward compatible with what o2o-translate produced. The only difference likely to affect users is where the CsPad calibration constants are found, this is discussed in the The XTC-to-HDF5 Translator section below. There are also a number of minor differences which should make no difference to user code written to process o2o-translate hdf5 files. These are documented in the section Difference's with o2o-translate. hdf5 files created by o2o-translate or psana-translate contain an attribute defining the schema number. Below we document important changes introduced with Schema 4 as implemented in V00-01-00 and above for psana-translate. o2o-translate implemented schema versions 1,2 and 3. These important changes are the use of CalibStore for calibration constants, and dropping PNCCD::FullFrames from translation.

Calibrated Data

o2o-translate knows how to calibrate CsPad data. If o2o-translate was told where a calib-dir was (which it is for automatic translation) and calibration constants have been recorded in this directory (typically carried out by the calibration management tool by processing a dark run) then o2o-translate calibrates cspad data and write the calibrated data instead of the raw xtc data. It writes the calibrated data in the same place where the raw xtc would have gone. It would also write the calibration constants used (such as pedestals and pixel status) in a special group. Finally, if the common mode calibration was done (this depends on what files are deployed to the calib-dir) which is a correction calculated for each event, the source group containing all the event data will include a common_mode dataset with the common mode values. This allows users to recover the raw data from the calibrated data.

With psana-translate calibration is handled by external psana modules. These modules will produce calibrated data and psana-translate will find it and translate it to the hdf5 file. Understanding this flow of data is not necessary for automatic translation, however if users want to customize calibration, some understanding of how psana modules pass data through the event store, and are configured through config files is necessary. The calibrated data will be distinguished from uncalibrated data with the use of a key in the event store. The key defaults to the value 'calibrated' but this is configurable through the psana.cfg file, in the section for the calibration module used. psana-translate provides special treatment for the calibration key. For psana-translate, the default value for the calibration key is 'calibrated' as well, but again, this is configurable through the psana.cfg file, in the section for Translator.H5Output. If psana-translate sees data with the key calibrated - it defaults to only translate data with the calibrated key and not the raw data. In the hdf5 file, one will find calibrated data where one would have otherwise found uncalibrated data. This is consistent with how o2o-translate translated calibrated cspad data. The calibrated key is not present in the hdf5 path names. This is different than what one finds for keys with ndarrays. For ndarrays the key is part of the h5 path name (see below). The psana-translate option skip_calibrated can be set to true to get the uncalibrated data instead of calibrated data.

Calibration makes use of calibration constants - such as pedestals and pixel status. A key difference between psana-translate and o2o-translate is where these calibration constants are found, and the datatypes used to store them. For psana-translate they are found in the group CalibStore to the current configure group. For example, if we translate the first event in a run of the cxi tutorial data where we add the cspad calibration module before the psana-translate module:

psana -n 1 -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.output_file=calib.h5 exp=xpptut13:run=71

Then we will see

Code Block

h5ls -r calib.h5 | grep -i "calibstore\|cspad"  # this command will include the following output

/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0/common_mode Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0/data Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0/element Dataset {1/Inf}
...
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.1/common_mode Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.1/data Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.1/element Dataset {1/Inf}
...
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PedestalsV1/XppGon.0:Cspad2x2.0/pedestals Dataset {185, 388, 2}
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PedestalsV1/XppGon.0:Cspad2x2.1/pedestals Dataset {185, 388, 2}
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PixelStatusV1/XppGon.0:Cspad2x2.0/status Dataset {185, 388, 2}
/Configure:0000/CalibStore/pdscalibdata::CsPadCommonModeSubV1/XppGon.0:Cspad2x2.0/data Dataset {SCALAR}
...

Things to note:

There are common_mode datasets included with the data
Both cspad sources have a pedestal dataset in CalibStore
Only XppGon.0:Cspad2x2.0 has a common mode dataset in the calibstore.

When one looks at the common_mode dataset for XppGon.0:Cspad2x2.1, one sees the values are -10000, indicating no common mode calibration was done. The module loaded in this example to do calibration, cspad_mod.CsPadCalib, is equivalent to what o2o-translate would do. More information on o2o-translate calibration can be found at CsPad calibration in translator. Documentation on the CsPadCalib module is found in psana - Module Catalog.

An issue users may run into is understanding what calibration was done and recovering the raw data just from examining the hdf5 output. In the case of cspad, an understanding of the CsPadCalib module along with the what is in the hdf5 file does allow one to recover the uncalibrated data. This may not be possible with other calibration modules and detectors, in particular if nonlinear calibration algorithms are applied, such as applying a threshold.

PNCCD::FullFrame

This data is no longer translated. FullFrame is a copy of Frames with a more convenient interface for psana users, it is not considered to be as useful for hdf5 files. User's interested in having FullFrame written into their hdf5 files rather than the original Frames data should make a feature request.

Filtering Events

Since psana-translate runs as a psana module, it is possible to filter translated events through psana options and other modules. psana options allow you to start at a certain event, and process a certain number of events. Moreover a user module that is loaded before the Translator module can tell psana that it should not pass this event on to any other modules, hence the Translator.H5Output module will never see the event and it will not get translated.

A downside of having modules loaded before the translator skip events is that updates to epics pv's, or configuration data will not get recorded. One may also wish to record the reason for filtering the event in the hdf5 file, as well as the event id's for the filtered events. psana-translate provides an interface for doing these things. To use this mechanism, a module must put an object in the eventStore with a key that starts with

do_not_translate

For example, if a C++ module implements the event method to do the following:

Code Block

language	cpp
title	filtering

  virtual void event(Event& evt, Env& env) {
	boost::shared_ptr<int> message = boost::make_shared<int>();        
    evt.put(message,"do_not_translate");
  }

Then none of the event data will get translated in any of the calib cycles. The Translator will do the following

For each calib cycle, it will make a filtered group
- For instance, if the file has the group /Configure:0000/Run:0000/CalibCycle:0000, then it will also have:
  the group: /Configure:0000/Run:0000/Filtered:0000
within each Filtered group, a time dataset that holds event id's of the filtered events.

Split Scan Mode

Starting in release ana-0.12.1, the Translator will support split scan mode. In this mode, each calib cycle will be written into a separate hdf5 file. A master file will have external links to the separate calib cycle files. Users need only work with the master file. The master file uses the same schema as one finds without split scan mode, with one exception discussed below.. Otherwise, no modification to users code is required when working with the master file. Not all experiments use more than one calib cycle. For experiments that use one calib cycle per run, split scan mode provides no benefit. Two reasons to use split scan mode is first, that the resulting hdf5 file from normal translation is too large, and second, to parallelize the translation and make it faster.

To run the Translator in split scan mode, three options must be set. These options do the following:

Tell the Translator to split (option split=SplitScan)
Tell the Translator how many jobs to run to parallelize the work (option jobTotal=N)
Tell the Translator which job it is (option jobNumber=K)

Next, one must run N different Translator jobs where the jobNumber parameter varies. An example of using two jobs for translation is:

psana -m Translator.H5Output -o Translator.H5Output.output_file=mydir/split.h5 -o Translator.H5Output.split=SplitScan -o Translator.H5Output.jobNumber=0 -o Translator.H5Output.jobTotal=2 exp=xpp123:run=10
psana -m Translator.H5Output -o Translator.H5Output.output_file=mydir/split.h5 -o Translator.H5Output.split=SplitScan -o Translator.H5Output.jobNumber=1 -o Translator.H5Output.jobTotal=2 exp=xpp123:run=10

Those commands will create the files


mydir/split.h5
mydir/split_cc0000.h5
mydir/split_cc0001.h5
...

Users need only work with split.h5. When moving the files, make sure they all reside in the same directory. The links from the master to the calib cycle files assume they are in the same directory.

Presently, each Translator job reads through the entire set of xtc files. As more and more Translator jobs are run simultaneously, the overall speed of translation diminishes while the load on the network steadily increases. It is recommended that users run no more than 5 Translator jobs. Testing has show 5 jobs provide up to a three times speed up in translation.

Reading While Translating

HDF5 presently has little support for reading a file that is being created, and in general does not recommend this. However the master file is written in a way to support this as well as possible. Most all links from the master file to calib cycle files are not created until the calib cycle file is finished. The exception is the last N links, where N is the number of jobs running. These links may be written before the calib cycle files they link to are finished. To see updates in the master file, users may need to shut down programs like Matlab and h5py and restart them. It is not sufficient to close and reopen the master file within a Python or Matlab session.

New Features

With psana-translate, you can

filter out whole events from translation
filter out certain data, by data type, or by data source, or key string
write ndarray's that other modules add to the event store
write std::string's that other C++ modules add to the event store
advanced: have a C++ module register a new type for translation

New Features Subject to Change

Three aspects of these new features are subject to change. These are highlighted in warning boxes below. In brief, these are

hdf5 group names for ndarray types
How event key strings are incorporated into hdf5 paths
How new C++ types are registered for translation.

Important Changes between o2o-translate and psana-translate

The translation that psana-translate produces is most always backward compatible with what o2o-translate produced. The only difference likely to affect users is where the CsPad calibration constants are found, this is discussed in the The XTC-to-HDF5 Translator section below. There are also a number of minor differences which should make no difference to user code written to process o2o-translate hdf5 files. These are documented in the section Difference's with o2o-translate. hdf5 files created by o2o-translate or psana-translate contain an attribute defining the schema number. Below we document important changes introduced with Schema 4 as implemented in V00-01-00 and above for psana-translate. o2o-translate implemented schema versions 1,2 and 3. These important changes are the use of CalibStore for calibration constants, and dropping PNCCD::FullFrames from translation.

Calibrated Data

o2o-translate knows how to calibrate CsPad data. If o2o-translate was told where a calib-dir was (which it is for automatic translation) and calibration constants have been recorded in this directory (typically carried out by the calibration management tool by processing a dark run) then o2o-translate calibrates cspad data and write the calibrated data instead of the raw xtc data. It writes the calibrated data in the same place where the raw xtc would have gone. It would also write the calibration constants used (such as pedestals and pixel status) in a special group. Finally, if the common mode calibration was done (this depends on what files are deployed to the calib-dir) which is a correction calculated for each event, the source group containing all the event data will include a common_mode dataset with the common mode values. This allows users to recover the raw data from the calibrated data.

With psana-translate calibration is handled by external psana modules. These modules will produce calibrated data and psana-translate will find it and translate it to the hdf5 file. Understanding this flow of data is not necessary for automatic translation, however if users want to customize calibration, some understanding of how psana modules pass data through the event store, and are configured through config files is necessary. The calibrated data will be distinguished from uncalibrated data with the use of a key in the event store. The key defaults to the value 'calibrated' but this is configurable through the psana.cfg file, in the section for the calibration module used. psana-translate provides special treatment for the calibration key. For psana-translate, the default value for the calibration key is 'calibrated' as well, but again, this is configurable through the psana.cfg file, in the section for Translator.H5Output. If psana-translate sees data with the key calibrated - it defaults to only translate data with the calibrated key and not the raw data. In the hdf5 file, one will find calibrated data where one would have otherwise found uncalibrated data. This is consistent with how o2o-translate translated calibrated cspad data. The calibrated key is not present in the hdf5 path names. This is different than what one finds for keys with ndarrays. For ndarrays the key is part of the h5 path name (see below). The psana-translate option skip_calibrated can be set to true to get the uncalibrated data instead of calibrated data.

Calibration makes use of calibration constants - such as pedestals and pixel status. A key difference between psana-translate and o2o-translate is where these calibration constants are found, and the datatypes used to store them. For psana-translate they are found in the group CalibStore to the current configure group. For example, if we translate the first event in a run of the cxi tutorial data where we add the cspad calibration module before the psana-translate module:

psana -n 1 -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.output_file=calib.h5 exp=xpptut13:run=71

Then we will see

Code Block

h5ls -r calib.h5 | grep -i "calibstore\|cspad"  # this command will include the following output

/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0/common_mode Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0/data Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.0/element Dataset {1/Inf}
...
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.1/common_mode Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.1/data Dataset {1/Inf}
/Configure:0000/Run:0000/CalibCycle:0000/CsPad2x2::ElementV1/XppGon.0:Cspad2x2.1/element Dataset {1/Inf}
...
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PedestalsV1/XppGon.0:Cspad2x2.0/pedestals Dataset {185, 388, 2}
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PedestalsV1/XppGon.0:Cspad2x2.1/pedestals Dataset {185, 388, 2}
/Configure:0000/CalibStore/pdscalibdata::CsPad2x2PixelStatusV1/XppGon.0:Cspad2x2.0/status Dataset {185, 388, 2}
/Configure:0000/CalibStore/pdscalibdata::CsPadCommonModeSubV1/XppGon.0:Cspad2x2.0/data Dataset {SCALAR}
...

Things to note:

There are common_mode datasets included with the data
Both cspad sources have a pedestal dataset in CalibStore
Only XppGon.0:Cspad2x2.0 has a common mode dataset in the calibstore.

When one looks at the common_mode dataset for XppGon.0:Cspad2x2.1, one sees the values are -10000, indicating no common mode calibration was done. The module loaded in this example to do calibration, cspad_mod.CsPadCalib, is equivalent to what o2o-translate would do. More information on o2o-translate calibration can be found at CsPad calibration in translator. Documentation on the CsPadCalib module is found in psana - Module Catalog.

An issue users may run into is understanding what calibration was done and recovering the raw data just from examining the hdf5 output. In the case of cspad, an understanding of the CsPadCalib module along with the what is in the hdf5 file does allow one to recover the uncalibrated data. This may not be possible with other calibration modules and detectors, in particular if nonlinear calibration algorithms are applied, such as applying a threshold.

PNCCD::FullFrame

This data is no longer translated. FullFrame is a copy of Frames with a more convenient interface for psana users, it is not considered to be as useful for hdf5 files. User's interested in having FullFrame written into their hdf5 files rather than the original Frames data should make a feature request.

Filtering Events

Since psana-translate runs as a psana module, it is possible to filter translated events through psana options and other modules. psana options allow you to start at a certain event, and process a certain number of events. Moreover a user module that is loaded before the Translator module can tell psana that it should not pass this event on to any other modules, hence the Translator.H5Output module will never see the event and it will not get translated.

A downside of having modules loaded before the translator skip events is that updates to epics pv's, or configuration data will not get recorded. One may also wish to record the reason for filtering the event in the hdf5 file, as well as the event id's for the filtered events. psana-translate provides an interface for doing these things. To use this mechanism, a module must put an object in the eventStore with a key that starts with

do_not_translate

For example, if a C++ module implements the event method to do the Suppose a user module has made some measurements that indicate this event should be filtered (for instance the beam energy is wrong). These measurements can be recorded in the hdf5 file by adding data to the event store that the Translator knows how to write. As discussed below, the Translator can write ndarrays and strings as well as simple new types that user modules register. If a user module implements event to do the following:

Code Block

language	cpp
title	C++ do not translate example - with logging	filtering

// define user Module,

  virtual void event(Event& evt, Env& env) {
    	boost::shared_ptr<std::string>ptr<int> message = boost::make_shared<std::string>("The beam energy is bad"shared<int>();        
    evt.put(message,"do_not_translate:message");
    const unsigned }

Then none of the event data will get translated in any of the calib cycles. The Translator will do the following

For each calib cycle, it will make a filtered group
- For instance, if the file has the group /Configure:0000/Run:0000/CalibCycle:0000, then it will also have:
  the group: /Configure:0000/Run:0000/Filtered:0000
within each Filtered group, a time dataset that holds event id's of the filtered events.

Suppose a user module has made some measurements that indicate this event should be filtered (for instance the beam energy is wrong). These measurements can be recorded in the hdf5 file by adding data to the event store that the Translator knows how to write. As discussed below, the Translator can write ndarrays and strings as well as simple new types that user modules register. If a user module implements event to do the following:

Code Block

language	cpp
title	C++ do not translate example - with logging

// define user Module,

  virtual void event(Event& evt, Env& env) {
    boost::shared_ptr<std::string> message = boost::make_shared<std::string>("The beam energy is bad");        shape[1] = {4};
    boost::shared_ptr< ndarray<float,1> measurements = boost::make_shared< ndarray<float,1> >(shape);
    float *data = measurements->data();
    data[0] = 0.4;  data[1] = 1.3; data[2] = 2.2; data[3] = 3.1;
    evt.put(measurementsmessage,"do_not_translate:measurementsmessage");
    const }unsigned shape[1] = {4};
    boost::shared_ptr< ndarray<float,1> measurements = boost::make_shared< ndarray<float,1> >(shape);
    float *data = measurements->data();
    data[0] = 0.4;  data[1] = 1.3; data[2] = 2.2; data[3] = 3.1;
    evt.put(measurements,"do_not_translate:measurements");
  }

Note the use of the key "do_not_translate:xxx" the :xxx is not necessary, but it helps to uniquely quality the event data, Note the use of the key "do_not_translate:xxx" the :xxx is not necessary, but it helps to uniquely quality the event data, and it will become a part of the 'src' group name where the do_not_translate event data is written to. Since both std::string and a ndarray<float,1> are types that the Translator knows how to write, it will create the following groups and datasets in the hdf5 file:

...

When running the translator as a psana module, if is often convenient to create a psana.cfg file. The Translator package include the file default_psana.cfg which is a psana configuration file that describes all the options possible, with extensive documentation as to what they mean. Below we include this file for reference. To use this file, one could it and modify it. However it is not necessary to take the whole file - every value set is set to the default value. One could simply use this as a reference for those options values that one wants to change.documentation as to what they mean. Below we include this file for reference. To use this file, one could it and modify it. However it is not necessary to take the whole file - every value set is set to the default value. One could simply use this as a reference for those options values that one wants to change.

Code Block

language	bash
title	psana-translate default_psana.cfg file - all options
collapse	true

######################################################################
[psana]

# MODULES: any modules that produce data to be translated need be loaded 
# **BEFORE** Translator.H5Output (such as calibrated data or NDArray's)
# event data added by modules listed after Translator.H5Output is not translated.
modules = Translator.H5Output

files = **TODO: SPECIFY INPUT FILES OR DATA SOURCE HERE**

######################################################################
[Translator.H5Output]

# The only option you need to set for the Translator.H5Output module is
# output_file. All other options have default values (explained below).

# TODO: enter the full h5 output file name, including the output directory
output_file = output_directory/h5output.h5

# By default, the Translator will not overwrite the h5 file if it already exists
overwrite = false

# # # # # # # # # # # # # # # # # # # # #
# EPICS FILTERING
# The Translator can store epics pv's in one of two ways, or not at all.
# set store_epics below, to one of the following:
#
# updates_only   stores an epic pv when it has changed. The pv is stored 
#                in the current calib cycle.  For mutli calib cycle experiments, 
#                users may have to look back through several calib cycle's to 
#                find the latest value of a pv.
#
# calib_repeat   each calib cycle will include the latest value of all the epics 
#                pv's.  This can make it easier to find pv's for a calib cycle. 
#                For experiments with many short calib cycles, it produces

Code Block

language	bash
title	psana-translate default_psana.cfg file - all options
collapse	true

######################################################################
[psana]

# MODULES: any modules that produce data to be translated need be loaded 
# **BEFORE** Translator.H5Output (such as calibrated data or NDArray's)
# event data added by modules listed after Translator.H5Output is not translated.
modules = Translator.H5Output

files = **TODO: SPECIFY INPUT FILES OR DATA SOURCE HERE**

######################################################################
[Translator.H5Output]

# The only option you need to set for the Translator.H5Output module is
# output_file. All other options have default values (explained below).

# TODO: enter the full h5 output file name, including the output directory
output_file = output_directory/h5output.h5

# By default, the Translator will not overwrite the h5 file if it already exists
overwrite = false

# # # # # # # # # # # # # # # # # # # # #
# EPICS FILTERING
# The Translator can store epics pv's in one of two ways, or not at all.
# set store_epics below, to one of the following:
#
# updates_only   stores an epic pv when it has changed. The pv is stored 
#                inmany themore currentdatasets calibthan cycleneccessary.
#
# no For mutli calib cycle experiments, 
#       epics pv's will not be stored. You may also users may havewant to look back through several calib cycle's to 
#set Epics=exclude
#                (see below) if you do not findwant the latest value of a pv.
#
# epics configuration data stored

# The default is 'calib_repeat'

store_epics = calib_repeat

# # # #  each calib cycle will include the latest value of all the epics 
#                pv's.  This can make it easier to find pv's for a calib cycle. 
#                For experiments with many short calib cycles, it produces 
#                many more datasets than neccessary.
#
# no             epics pv's will not be stored. You may also want to set Epics=exclude
#                (see below) if you do not want the epics configuration data stored

# The default is 'calib_repeat'

store_epics = calib_repeat

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# FILTERING
# 
# By default, all xtc data is Translated and many ndarrays that user modules (if any) 
# add are translated. Filtering can occur in either the code of user modules, or
# through options in the psana.cfg file. Here in the config file, different groups of 
# data can be filtered. There are four options for filtering data: 
#
#    type filtering   -  for example, exclude all cspad, regardless of the detector source
#    source filtering -  for example, exclude any data from a given detector source
#    key filtering    -  for example, include only ndarrays with a given key string
#    calibration      -  do not translate original xtc if a calibrated version is found
#
# Type filtering is based on sets of Psana data types. If you know what detectors or 
# devices to filter, leave type filtering alone and go to src_filter. 
#
# Type filtering has the highest precedence, then key filtering, then source 
# filtering, and lastly calibration filtering. When the Translator sees new data, 
# it first checks the type filter. If it is a filtered type (or unknown type) no further 
# translation occurs with the data - regardless of src or key. For data that gets 
# past the type filter, the Translator looks at the src and key. If the key 
# string is empty, it checks the source filter. Data with non empty key strings are 
# handled via the key filter. If the src is filtered, but the key is not, then the
# data will be translated. Data with the special calibration key string are handled 
# via the calibration filtering. 
#
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# TYPE FILTERING 
#
# One can include or exclude a class of Psana types with the following 
# options. Only the strings include or exclude are valid for these 
# type filtering options. 
# 
# Note - Epics in the list below refers only to the epicsConfig data
# which is the epics alias list, not the epics pv's. To filter the epics pv's
# see the 'store_epics' option above.

AcqTdc# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# FILTERING
# 
# By default, all xtc data is Translated and many ndarrays that user modules (if any) 
# add are translated. Filtering can occur in either the code of user modules, or
# through options in the psana.cfg file. Here in the config file, different groups of 
# data can be filtered. There are four options for filtering data: 
#
#    type filtering   -  for example, exclude all cspad, regardless of the detector source
#    source filtering -  for example, exclude any data from a given detector source
#    key filtering    -  for example, include only ndarrays with a given key string
#    calibration      -  do not translate original xtc if a calibrated version is found
#
# Type filtering is based on sets of Psana data types. If you know what detectors or 
# devices to filter, leave type filtering alone and go to src_filter. 
#
# Type filtering has the highest precedence, then key filtering, then source 
# filtering, and lastly calibration filtering. When the Translator sees new data, 
# it first checks the type filter. If it is a filtered type (or unknown type) no further 
# translation occurs with the data - regardless of src or key. For data that gets 
# past the type filter, the Translator looks at the src and key. If the key 
# string is empty, it checks the source filter. Data with non empty key strings are 
# handled via the key filter. If the src is filtered, but the key is not, then the
# data will be translated. Data with the special calibration key string are handled 
# via the calibration filtering. 
#
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# TYPE FILTERING 
#
# One can include or exclude a class of Psana types with the following 
# options. Only the strings include or exclude are valid for these 
# type filtering options. 
# 
# Note - Epics in the list below refers only to the epicsConfig data
# which is the epics alias list, not the epics pv's. To filter the epics pv's
# see the 'store_epics' option above.

AcqTdc = include               # Psana::Acqiris::TdcConfigV1, Psana::Acqiris::TdcDataV1
AcqWaveform = include          # Psana::Acqiris::ConfigV1, Psana::Acqiris::DataDescV1
Alias = include                # Psana::Alias::ConfigV1
Andor = include                # Psana::Andor::ConfigV1, Psana::Andor::FrameV1
Arraychar = include            # Psana::Arraychar::DataV1
Control = include              # Psana::ControlData::ConfigV1, Psana::ControlData::ConfigV2, Psana::ControlData::ConfigV3
Cspad = include                # Psana::CsPad::ConfigV1, Psana::CsPad::ConfigV2, Psana::CsPad::ConfigV3, Psana::CsPad::ConfigV4, Psana::CsPad::ConfigV5, Psana::CsPad::DataV1, Psana::CsPad::DataV2
Cspad2x2 = include             # Psana::CsPad2x2::ConfigV1, Psana::CsPad2x2::ConfigV2, Psana::CsPad2x2::ElementV1
DiodeFex = include             # Psana::Lusi::DiodeFexConfigV1, Psana::Lusi::DiodeFexConfigV2, Psana::Lusi::DiodeFexV1
EBeam = include                # Psana::Bld::BldDataEBeamV0, Psana::Bld::BldDataEBeamV1, Psana::AcqirisBld::BldDataEBeamV2, Psana::Bld::BldDataEBeamV3, Psana::Bld::TdcConfigV1BldDataEBeamV4, Psana::AcqirisBld::TdcDataV1BldDataEBeamV5
AcqWaveformEncoder = include              # Psana::AcqirisEncoder::ConfigV1, Psana::Encoder::ConfigV2, Psana::Encoder::AcqirisDataV1, Psana::Encoder::DataDescV1DataV2
AliasEpics = include                # Psana::AliasEpics::ConfigV1
AndorEpix = include                 # Psana::AndorEpix::ConfigV1, Psana::AndorEpix::FrameV1ElementV1
ArraycharEpixSampler = include            # Psana::Arraychar::DataV1
Control = include              # Psana::ControlData::ConfigV1, Psana::ControlData::ConfigV2EpixSampler::ConfigV1, Psana::ControlDataEpixSampler::ConfigV3ElementV1
CspadEvr = include                  # Psana::CsPadEvrData::ConfigV1, Psana::CsPadEvrData::ConfigV2, Psana::CsPadEvrData::ConfigV3, Psana::CsPadEvrData::ConfigV4, Psana::CsPadEvrData::ConfigV5, Psana::CsPadEvrData::ConfigV6, Psana::EvrData::DataV1ConfigV7, Psana::CsPad::DataV2
Cspad2x2EvrData::DataV3
EvrIO = include                # Psana::EvrData::IOConfigV1
Evs = include                  # Psana::CsPad2x2::ConfigV1,EvrData::SrcConfigV1
FEEGasDetEnergy = include      # Psana::Bld::BldDataFEEGasDetEnergy
Fccd = include                 # Psana::CsPad2x2FCCD::ConfigV2FccdConfigV1, Psana::CsPad2x2FCCD::ElementV1FccdConfigV2
DiodeFexFli = include                  # Psana::LusiFli::DiodeFexConfigV1ConfigV1, Psana::LusiFli::DiodeFexConfigV2, Psana::Lusi::DiodeFexV1
EBeamFrameV1
Frame = include                # Psana::BldCamera::BldDataEBeamV0, Psana::Bld::BldDataEBeamV1, Psana::Bld::BldDataEBeamV2, Psana::Bld::BldDataEBeamV3, Psana::Bld::BldDataEBeamV4,FrameV1
FrameFccd = include            # Psana::BldCamera::BldDataEBeamV5FrameFccdConfigV1
EncoderFrameFex = include              # Psana::Encoder::ConfigV1, Psana::Encoder::ConfigV2, Psana::Encoder::DataV1, Psana::Encoder::DataV2
EpicsCamera::FrameFexConfigV1
GMD = include                  # Psana::Bld::BldDataGMDV0, Psana::EpicsBld::ConfigV1BldDataGMDV1
EpixGsc16ai = include                 # Psana::EpixGsc16ai::ConfigV1, Psana::EpixGsc16ai::ElementV1DataV1
EpixSamplerImp = include                  # Psana::EpixSamplerImp::ConfigV1, Psana::EpixSamplerImp::ElementV1
EvrIpimb = include                  # Psana::EvrDataIpimb::ConfigV1, Psana::EvrDataIpimb::ConfigV2, Psana::EvrDataIpimb::ConfigV3DataV1, Psana::EvrData::ConfigV4, Psana::EvrData::ConfigV5, Psana::EvrData::ConfigV6, Psana::EvrData::ConfigV7, Psana::EvrData::DataV3
EvrIO = include                # Psana::EvrData::IOConfigV1
Evs = include     Ipimb::DataV2
IpmFex = include             # Psana::EvrData::SrcConfigV1
FEEGasDetEnergy = include      # Psana::Lusi::IpmFexConfigV1, Psana::Lusi::IpmFexConfigV2, Psana::BldLusi::BldDataFEEGasDetEnergyIpmFexV1
FccdL3T = include                  # Psana::FCCDL3T::FccdConfigV1ConfigV1, Psana::FCCDL3T::FccdConfigV2DataV1
FliOceanOptics = include          #        #Psana::OceanOptics::ConfigV1, Psana::OceanOptics::ConfigV2, Psana::FliOceanOptics::ConfigV1DataV1, Psana::FliOceanOptics::FrameV1DataV2
FrameOpal1k = include                # Psana::CameraOpal1k::FrameV1ConfigV1
FrameFccdOrca = include                 # Psana::CameraOrca::FrameFccdConfigV1ConfigV1
FrameFexPartition = include             # Psana::CameraPartition::FrameFexConfigV1ConfigV1
GMDPhaseCavity = include                  # Psana::Bld::BldDataGMDV0, Psana::Bld::BldDataGMDV1
Gsc16aiBldDataPhaseCavity
PimImage = include              # Psana::Gsc16aiLusi::ConfigV1, Psana::Gsc16ai::DataV1
ImpPimImageConfigV1
Pimax = include                  # Psana::ImpPimax::ConfigV1, Psana::ImpPimax::ElementV1FrameV1
IpimbPrinceton = include            # Psana::Princeton::ConfigV1, Psana::Princeton::ConfigV2,  # Psana::Princeton::ConfigV3, Psana::IpimbPrinceton::ConfigV1ConfigV4, Psana::IpimbPrinceton::ConfigV2ConfigV5, Psana::IpimbPrinceton::DataV1FrameV1, Psana::IpimbPrinceton::DataV2FrameV2
IpmFexPrincetonInfo = include        # Psana::Princeton::InfoV1
Quartz = include    # Psana::Lusi::IpmFexConfigV1, Psana::Lusi::IpmFexConfigV2,           # Psana::LusiQuartz::IpmFexV1ConfigV1
L3TRayonix = include                  # Psana::L3TRayonix::ConfigV1, Psana::L3T::DataV1
OceanOptics::Rayonix::ConfigV2
SharedAcqADC = include         # Psana::Bld::BldDataAcqADCV1
SharedIpimb = include          # Psana::OceanOpticsBld::ConfigV1BldDataIpimbV0, Psana::OceanOpticsBld::ConfigV2, Psana::OceanOptics::DataV1, Psana::OceanOptics::DataV2
Opal1kBldDataIpimbV1
SharedPim = include               # Psana::Opal1kBld::ConfigV1BldDataPimV1
OrcaSpectrometer = include                 # Psana::OrcaBld::ConfigV1BldDataSpectrometerV0
PartitionTM6740 = include               # Psana::Pulnix::TM6740ConfigV1, Psana::PartitionPulnix::ConfigV1TM6740ConfigV2
PhaseCavityTimepix = include              # Psana::BldTimepix::BldDataPhaseCavity
PimImage = include    ConfigV1, Psana::Timepix::ConfigV2, Psana::Timepix::ConfigV3, Psana::Timepix::DataV1, Psana::Timepix::DataV2
TwoDGaussian = include         # Psana::LusiCamera::PimImageConfigV1TwoDGaussianV1
PimaxUsdUsb = include                # Psana::PimaxUsdUsb::ConfigV1, Psana::PimaxUsdUsb::FrameV1DataV1
PrincetonpnCCD = include            # Psana::Princeton::ConfigV1, Psana::Princeton::ConfigV2, Psana::Princeton::ConfigV3, Psana::Princeton::ConfigV4, # Psana::PrincetonPNCCD::ConfigV5ConfigV1, Psana::PrincetonPNCCD::FrameV1ConfigV2, Psana::PrincetonPNCCD::FrameV2FramesV1

PrincetonInfo# =user includetypes to translate from the    # Psana::Princeton::InfoV1
Quartzevent store
ndarray_types = include        # ndarray<int8_t,1>, ndarray<int8_t,2>, ndarray<int8_t,3>, ndarray<int8_t,4>, ndarray<int16_t,1>, ndarray<int16_t,2>,  # Psana::Quartz::ConfigV1
Rayonix = include              # Psana::Rayonix::ConfigV1, Psana::Rayonix::ConfigV2
SharedAcqADC = include         # Psana::Bld::BldDataAcqADCV1
SharedIpimb = include          # Psana::Bld::BldDataIpimbV0, Psana::Bld::BldDataIpimbV1
SharedPim = include            # Psana::Bld::BldDataPimV1
Spectrometer = include         # Psana::Bld::BldDataSpectrometerV0
TM6740 = include               # Psana::Pulnix::TM6740ConfigV1, Psana::Pulnix::TM6740ConfigV2
Timepix = include              # Psana::Timepix::ConfigV1, Psana::Timepix::ConfigV2, Psana::Timepix::ConfigV3, Psana::Timepix::DataV1, Psana::Timepix::DataV2
TwoDGaussian = includendarray<int16_t,3>, ndarray<int16_t,4>, ndarray<int32_t,1>, ndarray<int32_t,2>, ndarray<int32_t,3>, ndarray<int32_t,4>, ndarray<int64_t,1>, ndarray<int64_t,2>, ndarray<int64_t,3>, ndarray<int64_t,4>, ndarray<uint8_t,1>, ndarray<uint8_t,2>, ndarray<uint8_t,3>, ndarray<uint8_t,4>, ndarray<uint16_t,1>, ndarray<uint16_t,2>, ndarray<uint16_t,3>, ndarray<uint16_t,4>, ndarray<uint32_t,1>, ndarray<uint32_t,2>, ndarray<uint32_t,3>, ndarray<uint32_t,4>, ndarray<uint64_t,1>, ndarray<uint64_t,2>, ndarray<uint64_t,3>, ndarray<uint64_t,4>, ndarray<float,1>, ndarray<float,2>, ndarray<float,3>, ndarray<float,4>, ndarray<double,1>, ndarray<double,2>, ndarray<double,3>, ndarray<double,4>, ndarray<const int8_t,1>, ndarray<const int8_t,2>, ndarray<const int8_t,3>, ndarray<const int8_t,4>, ndarray<const int16_t,1>, ndarray<const int16_t,2>, ndarray<const int16_t,3>, ndarray<const int16_t,4>, ndarray<const int32_t,1>, ndarray<const int32_t,2>, ndarray<const int32_t,3>, ndarray<const int32_t,4>, ndarray<const int64_t,1>, ndarray<const int64_t,2>, ndarray<const int64_t,3>, ndarray<const int64_t,4>, ndarray<const uint8_t,1>, ndarray<const uint8_t,2>, ndarray<const uint8_t,3>, ndarray<const uint8_t,4>, ndarray<const uint16_t,1>, ndarray<const uint16_t,2>, ndarray<const uint16_t,3>, ndarray<const uint16_t,4>, ndarray<const uint32_t,1>, ndarray<const uint32_t,2>, ndarray<const uint32_t,3>, ndarray<const uint32_t,4>, ndarray<const uint64_t,1>, ndarray<const uint64_t,2>, ndarray<const uint64_t,3>, ndarray<const uint64_t,4>, ndarray<const float,1>, ndarray<const float,2>, ndarray<const float,3>, ndarray<const float,4>, ndarray<const double,1>, ndarray<const double,2>, ndarray<const double,3>, ndarray<const double,4>
std_string = include           # Psanastd::Camera::TwoDGaussianV1
UsdUsb = include               # Psana::UsdUsb::ConfigV1, Psana::UsdUsb::DataV1
pnCCD = include                # Psana::PNCCD::ConfigV1, Psana::PNCCD::ConfigV2, Psana::PNCCD::FramesV1

# user types to translate from the event store
ndarray_types = include string


# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# TYPE FILTER SHORTCUT
#
# In addition to filtering Psana types by the options above, one can use
# the type_filter option below. For example:
#
# type_filter include cspad       # ndarray<int8_t,1>, ndarray<int8_t,2>, ndarray<int8_t,3>, ndarray<int8_t,4>, ndarray<int16_t,1>, ndarray<int16_t,2>, ndarray<int16_t,3>, ndarray<int16_t,4>, ndarray<int32_t,1>, ndarray<int32_t,2>, ndarray<int32_t,3>, ndarray<int32_t,4>, ndarray<int64_t,1>, ndarray<int64_t,2>, ndarray<int64_t,3>, ndarray<int64_t,4>, ndarray<uint8_t,1>, ndarray<uint8_t,2>, ndarray<uint8_t,3>, ndarray<uint8_t,4>, ndarray<uint16_t,1>, ndarray<uint16_t,2>, ndarray<uint16_t,3>, ndarray<uint16_t,4>, ndarray<uint32_t,1>, ndarray<uint32_t,2>, ndarray<uint32_t,3>, ndarray<uint32_t,4>, ndarray<uint64_t,1>, ndarray<uint64_t,2>, ndarray<uint64_t,3>, ndarray<uint64_t,4>, ndarray<float,1>, ndarray<float,2>, ndarray<float,3>, ndarray<float,4>, ndarray<double,1>, ndarray<double,2>, ndarray<double,3>, ndarray<double,4>, ndarray<const int8_t,1>, ndarray<const int8_t,2>, ndarray<const int8_t,3>, ndarray<const int8_t,4>, ndarray<const int16_t,1>, ndarray<const int16_t,2>, ndarray<const int16_t,3>, ndarray<const int16_t,4>, ndarray<const int32_t,1>, ndarray<const int32_t,2>, ndarray<const int32_t,3>, ndarray<const int32_t,4>, ndarray<const int64_t,1>, ndarray<const int64_t,2>, ndarray<const int64_t,3>, ndarray<const int64_t,4>, ndarray<const uint8_t,1>, ndarray<const uint8_t,2>, ndarray<const uint8_t,3>, ndarray<const uint8_t,4>, ndarray<const uint16_t,1>, ndarray<const uint16_t,2>, ndarray<const uint16_t,3>, ndarray<const uint16_t,4>, ndarray<const uint32_t,1>, ndarray<const uint32_t,2>, ndarray<const uint32_t,3>, ndarray<const uint32_t,4>, ndarray<const uint64_t,1>, ndarray<const uint64_t,2>, ndarray<const uint64_t,3>, ndarray<const uint64_t,4>, ndarray<const float,1>, ndarray<const float,2>, ndarray<const float,3>, ndarray<const float,4>, ndarray<const double,1>, ndarray<const double,2>, ndarray<const double,3>, ndarray<const double,4>
std_string = include           # std::string


# # # # # # # #  will only translate cspad types. Will not translate
#                                 # ndarrays or strings
# type_filter exclude Andor evr   # translate all except the Andor or Evr types
# 
# If you do not want to translate what is in the xtc file, use the psana shortcut:
#
# type_filter exclude psana       # This will only translate ndarray's and strings 
#
# Likewise doing:
#
# type_filter include psana       # will translate all xtc data, but skip any ndarray's or strings
#
# The default is to include all

type_filter include all

# note - if type_filter is anything other than 'include all' it takes precedence
# over the classes of type filter options above, like Cspad=include.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# TYPESOURCE FILTER SHORTCUTFILTERING
#
# The Indefault additionfor tothe filteringsrc_filter Psanaoption typesis by the options above, one can use
# the type_filter option below. For example:"include all"
# If you want to include a subset of the sources, do
#
# typesrc_filter include cspadsrcname1 srcname2  
#
#  or if #you willwant onlyto translateexclude cspada types.subset Willof not translatesources, do
#
# src_filter exclude srcname1 srcname2
#
# The syntax for specifying a srcname follows that of the Psana Source (discussed in 
# the Psana Users Guide). The Psana Source recognizes DAQ alias names (if  present
# ndarraysin orthe strings
# type_filter exclude Andor evr   # translate all except the Andor or Evr types
# 
# If you do not want to translate what is in the xtc file, use the psana shortcutxtc files), several styles for specifying a Pds Src, as well as detector matches 
# where the detector number, or device number is not known.
# 
# Specifically, format of the match string can be:
#
#  type_filter exclude psana   DetInfo(det.detId:dev.devId) - fully or partially specified DetInfo
# This  will only translate ndarray's and strings 
#
# Likewise doing:
#
# type_filter include psana       # will translate all xtc data, but skip any ndarray's or strings
#
# The default is to include all

type_filter include all

# note - if type_filter is anything other than 'include all' it takes precedence
# over the classes of type filter options above, like Cspad=include.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# SOURCE FILTERING
#
# The default for the src_filter option is "include all"
# If you want to include a subset of the sources, do
#
# src_filter include srcname1 srcname2  
#
#  or if you want to exclude a subset of sources, do
#
# src_filter exclude srcname1 srcname2
#
# The syntax for specifying a srcname follows that of the Psana Source (discussed in 
# the Psana Users Guide). The Psana Source recognizes DAQ alias names (if present
# in the xtc files), several styles for specifying a Pds Src, as well as detector matches 
# where the detector number, or device number is not known.
# 
# Specifically, format of the match string can be:
#
#       DetInfo(det.detId:dev.devId) - fully or partially specified DetInfo
#       det.detId:dev.devId - same as above
#       DetInfo(det-detId|dev.devId) - same as above
#       det-detId|dev.devId - same as above
#       BldInfo(type) - fully or partially specified BldInfo
#       type - same as above
#       ProcInfo(ipAddr) - fully or partially specified ProcInfo
#
# For example
#        DetInfo(AmoETOF.0.Acqiris.0)  
#        DetInfo(AmoETOF.0.Acqiris)  
#        DetInfo(AmoETOF:Acqiris)
#        AmoETOF:Acqiris
#        AmoETOF|Acqiris
#
# will all match the same data, AmoETOF.0.Acqiris.0. The later ones will match
# additional data (such as detector 1, 2, etc.) if it is present.
#
# A simple way to set up src filtering is to take a look at the sources in the 
# xtc input using the psana EventKeys module.  For example
#
# psana -n 5 -m EventKeys exp=cxitut13:run=22 
#
# Will print the EventKeys in the first 5 events.  If the output includes
#
#   EventKey(type=Psana::EvrData::DataV3, src=DetInfo(NoDetector.0:Evr.2))
#   EventKey(type=Psana::CsPad::DataV2, src=DetInfo(CxiDs1.0:Cspad.0))
#   EventKey(type=Psana::CsPad2x2::ElementV1, src=DetInfo(CxiSc2.0:Cspad2x2.1))
#   EventKey(type=Psana::Bld::BldDataEBeamV3, src=BldInfo(EBeam))
#   EventKey(type=Psana::Bld::BldDataFEEGasDetEnergy, src=BldInfo(FEEGasDetEnergy))
#   EventKey(type=Psana::Camera::FrameV1, src=BldInfo(CxiDg2_Pim))
#
# Then one can filter on these six srcname's:
#
#  NoDetector.0:Evr.2  CxiDs1.0:Cspad.0  CxiSc2.0:Cspad2x2.1  EBeam  FEEGasDetEnergy  CxiDg2_Pim
#

src_filter = include all

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# CALIBRATION FILTERING
#
# Psana calibration modules can produce calibrated versions of different 
# data types. Depending on the module used, you may get an NDArray, an 
# image, or the same data type as was in the xtc but with calibrated data.
#
# If you are doing the latter, the module output will be data of the same type 
# and src as the uncalibrated data, with an additional key, such as 'calibrated'.
# If these modules are configured to use a different key, set calibration_key
# below accordingly:

calibration_key = calibrated

# The Translator defaults to writing calibrated data in place of uncalibrated
# data. If you do not want the calibrated data, set skip_calibrated to true.

skip_calibrated = false

# note, setting skip_calibrated to true will force sets exclude_calibstore 
# (below) to be true as well.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# CALIBSTORE FILTERING
#
# Calibration modules may publish the data they used to produce the calibrated
# event objects. Examples of data would be pedestal values, pixel status (what
# pixels are hot) and common mode algorithm parameters. This data will be published
# in what is called the Psana calibStore. When the Translator sees calibrated 
# event data, it will look for the corresponsinding calibStore data as well.
# If you do not want it to translate calibStore data, set the following to true.

exclude_calibstore = false

# otherwise, the Translator will create a group CalibStore that holds the
# calibstore data. Note, the Translator looks for all calibStore data associated 
# with the calibration modules. If a calibration module was configured to not do 
# certain calibrations (such as gain) but the module still put gain values
# in the config store (even though it did not use them) the Translator 
# would still translate those gain values.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# KEY FILTERING
#
# Psana modules loaded before the translator may put a variety of objects in the event 
# store. Be default, the Translator will translate any new data that it knows about.
# In addition to the psana types, it knows about NDArrays, C++ strings, and has a C++ interface 
# for registering new simple types. NDarray's up to 4 dimensions of 10 basic types 
# (8, 16, 32 and 64 bit signed and unsigned int, float and double) as well as the const 
# versions of these types are translated.
#
# Generally Psana modules will attach keys to these objects (the keys are simply strings).
# To filter the set of keys that are translated, modify the parameter below:

key_filter = include all

# The default is to not look at the key but rather translate all data that the translator
# knows about. An example of including only data with the key finalanswer would be
#
# key_filter = include finalanswer
#
# To exclude a few keys, one can do
#
# key_filter = exclude arrayA arrayB
#
# Note, key filtering does not affect translation of data without keys. For instance
# setting key_filter = include keyA does not turn off translation of data without keys.
# Of all the data with keys, only those where the key is keyA will be translated det.detId:dev.devId - same as above
#       DetInfo(det-detId|dev.devId) - same as above
#       det-detId|dev.devId - same as above
#       BldInfo(type) - fully or partially specified BldInfo
#       type - same as above
#       ProcInfo(ipAddr) - fully or partially specified ProcInfo
#
# For example
#        DetInfo(AmoETOF.0.Acqiris.0)  
#        DetInfo(AmoETOF.0.Acqiris)  
#        DetInfo(AmoETOF:Acqiris)
#        AmoETOF:Acqiris
#        AmoETOF|Acqiris
#
# will all match the same data, AmoETOF.0.Acqiris.0. The later ones will match
# additional data (such as detector 1, 2, etc.) if it is present.
#
# A simple way to set up src filtering is to take a look at the sources in the 
# xtc input using the psana EventKeys module.  For example
#
# psana -n 5 -m EventKeys exp=cxitut13:run=22 
#
# Will print the EventKeys in the first 5 events.  If the output includes
#
#   EventKey(type=Psana::EvrData::DataV3, src=DetInfo(NoDetector.0:Evr.2))
#   EventKey(type=Psana::CsPad::DataV2, src=DetInfo(CxiDs1.0:Cspad.0))
#   EventKey(type=Psana::CsPad2x2::ElementV1, src=DetInfo(CxiSc2.0:Cspad2x2.1))
#   EventKey(type=Psana::Bld::BldDataEBeamV3, src=BldInfo(EBeam))
#   EventKey(type=Psana::Bld::BldDataFEEGasDetEnergy, src=BldInfo(FEEGasDetEnergy))
#   EventKey(type=Psana::Camera::FrameV1, src=BldInfo(CxiDg2_Pim))
#
# Then one can filter on these six srcname's:
#
#  NoDetector.0:Evr.2  CxiDs1.0:Cspad.0  CxiSc2.0:Cspad2x2.1  EBeam  FEEGasDetEnergy  CxiDg2_Pim
#

src_filter = include all

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# CALIBRATION FILTERING
#
# Psana calibration modules can produce calibrated versions of different 
# data types. Depending on the module used, you may get an NDArray, an 
# image, or the same data type as was in the xtc but with calibrated data.
#
# If you are doing the latter, the module output will be data of the same type 
# and src as the uncalibrated data, with an additional key, such as 'calibrated'.
# If these modules are configured to use a different key, set calibration_key
# below accordingly:

calibration_key = calibrated

# The Translator defaults to writing calibrated data in place of uncalibrated
# data. If you do not want the calibrated data, set skip_calibrated to true.

skip_calibrated = false

# note, setting skip_calibrated to true will force sets exclude_calibstore 
# (below) to be true as well.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# CALIBSTORE FILTERING
#
# Calibration modules may publish the data they used to produce the calibrated
# event objects. Examples of data would be pedestal values, pixel status (what
# pixels are hot) and common mode algorithm parameters. This data will be published
# in what is called the Psana calibStore. When the Translator sees calibrated 
# event data, it will look for the corresponsinding calibStore data as well.
# If you do not want it to translate calibStore data, set the following to true.

exclude_calibstore = false

# otherwise, the Translator will create a group CalibStore that holds the
# calibstore data. Note, the Translator looks for all calibStore data associated 
# with the calibration modules. If a calibration module was configured to not do 
# certain calibrations (such as gain) but the module still put gain values
# in the config store (even though it did not use them) the Translator 
# would still translate those gain values.

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# KEY FILTERING
#
# Psana modules loaded before the translator may put a variety of objects in the event 
# store. Be default, the Translator will translate any new data that it knows about.
# In addition to the psana types, it knows about NDArrays, C++ strings, and has a C++ interface 
# for registering new simple types. NDarray's up to 4 dimensions of 10 basic types 
# (8, 16, 32 and 64 bit signed and unsigned int, float and double) as well as the const 
# versions of these types are translated.
#
# Generally Psana modules will attach keys to these objects (the keys are simply strings).
# To filter the set of keys that are translated, modify the parameter below:

key_filter = include all

# The default is to not look at the key but rather translate all data that the translator
# knows about. An example of including only data with the key finalanswer would be
#
# key_filter = include finalanswer
#
# To exclude a few keys, one can do
#
# key_filter = exclude arrayA arrayB
#
# Note, key filtering does not affect translation of data without keys. For instance
# setting key_filter = include keyA does not turn off translation of data without keys.
# Of all the data with keys, only those where the key is keyA will be translated.
#
# ---------------------------------------
# SPLIT INTO SEPARTE HDF5 FILES BASED ON CALIB CYCLES
#
# There are two reasons to split the Translator output, the resulting hdf5 file is to 
# large, and to parallelize the translation and make it faster. The default is to 
# not split:

split = NoSplit

# however the Translator also supports SplitScan mode:
#
# split=SplitScan
#
# In SplitScan mode, in addition to the output File, one file will be made for every
# calib cycle. The output file (the master file) will include external links to the other files. 
# Several translator jobs may run simultaneously to divide the work of creating the calib cycle files.
# At this time, each Translator job reads through all the input, so launching too many jobs will
# significantly increase the amount of input processing.
# Dividing the work of SplitScan mode is done with the parameters

# jobTotal = 1
# jobNumber = 0

# which default to 1 job that is numbered 0. However if jobTotal=3 and jobNumber=1, this 
# Translator will process calibCycle 1, 4, 7, etc. If jobTotal is 3, the user MUST
# make sure to launch 3 Translator jobs with jobNumber being 0,1 and 2 to get all the calib cycle
# files written. jobNumber=0 will write the master file with the external links to the calib
# cycle files. 
#
# For example, the following two command lines:
#
# psana -m Translator.H5Output -o Translator.H5Output.output_file=mydir/split.h5 -o Translator.H5Output.split=SplitScan -o Translator.H5Output.jobNumber=0 -o Translator.H5Output.jobTotal=2 exp=xpp123:run=10
# psana -m Translator.H5Output -o Translator.H5Output.output_file=mydir/split.h5 -o Translator.H5Output.split=SplitScan -o Translator.H5Output.jobNumber=1 -o Translator.H5Output.jobTotal=2 exp=xpp123:run=10
#
# will divide the work into two translator jobs. When they finish, the output will be
# 
# mydir/split.h5
# mydir/split_cc0000.h5
# mydir/split_cc0001.h5
# ...
#
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 
# COMPRESSION 
#
# The following options control compression for most all datasets.
# Shuffling improves compression for certain datasets. Valid values for
# deflate (gzip compression level) are 0-9. Setting deflate = -1 turns off
# compression.

shuffle = true
deflate = 1

# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# TECHNICAL, ADVANCED CONFIGURATION
# 
# ---------------------------------------
# CHUNKING
# The commented options below give the default chunking options.
# Objects per chunk are selected from the target chunk size (16 MB) and 
# adjusted based on min/max objects per chunk, and the max bytes per chunk.
# It is important that the chunkCache (created on a per dataset basis) be 
# large enough to hold at least one chunk, ideally all chunks we need to have
# open at one time when writing to the dataset (usually one, unless we repair
# split events):
 
# chunkSizeTargetInBytes = 1703936 (16MB)
# chunkSizeTargetObjects = 0 (0 means select objects per chunk from chunkSizeInBytes)
# maxChunkSizeInBytes = 10649600  (100MB)
# minObjectsPerChunk = 50              
# maxObjectsPerChunk = 2048
# chunkCacheSizeTargetInChunks = 3
# maxChunkCacheSizeInBytes = 10649600  (100MB)

# ---------------------------------------
# REFINED DATASET CONTROL
#
# There are six classes of datasets for which individual options for shuffle,
# deflate, chunkSizeTargetInBytes and chunkSizeTargetObjects can be specified:
#
# regular (most everything, all psana types)
# epics (all the epics pv's)
# damage (accompanies all regular data from event store)
# ndarrays (new data from other modules)
# string's (new data from other modules)
# eventId (the time dataset that also accompanies all regular data, epics pvs, ndarrays and strings)
#
# The options for regular datasets have been discussed above. The other five datasets 
# get their default values for shuffle, deflate, chunkSizeInBytes and chunkSizeInObjects
# from the regular dataset options except in the cases below:
 
# damageShuffle = false
# stringShuffle = false
# epicsPvShuffle = false
# stringDeflate = -1
# eventIdChunkSizeTargetInBytes = 16384
# epicsPvChunkSizeTargetInBytes = 16384

# The rest of the shuffle, deflate and chunk size options for the other five datasets are:
#
# eventIdShuffle = true
# eventIdDeflate = 1
# damageDeflate = 1
# epicsPvDeflate = 1
# ndarrayShuffle = true
# ndarrayDeflate = 1
# eventIdChunkSizeTargetObjects = 0
# damageChunkSizeTargetInBytes = 1703936
# damageChunkSizeTargetObjects = 0
# stringChunkSizeTargetInBytes = 1703936
# stringChunkSizeTargetObjects = 0
# ndarrayChunkSizeTargetInBytes = 1703936
# ndarrayChunkSizeTargetObjects = 0
# epicsPvChunkSizeTargetObjects = 0

# ---------------------------------------
# SPLIT EVENTS
# When the Translator encounters a split event, it checks a cache to see
# if it has already seen it.  If it has, it fills in any blanks that it can.
# To prevent this cache from growing to large, set the maximum number of
# split events to look back through here (default is 3000):

max_saved_split_events = 3000

# ---------------------------------------
# HDF5 FILE PROPERTIES
#
# split large files, presently we only support NoSplit. Future options may be: Family and SplitScan
# for future splitting, splitSize defaults to 10 GB
split = NoSplit
splitSize = 10737418240

Translation and Damage

psana has a specific damage policy that tells it what damaged data is acceptable for psana modules and what data is not. The default behavior is

...

Page tree

Versions Compared

Old Version 52

New Version 53

Key

New Features

New Features Subject to Change

Important Changes between o2o-translate and psana-translate

Calibrated Data

PNCCD::FullFrame

Filtering Events

Split Scan Mode

Reading While Translating

New Features

New Features Subject to Change

Important Changes between o2o-translate and psana-translate

Calibrated Data

PNCCD::FullFrame

Filtering Events

Translation and Damage

Page tree

Page History

Versions Compared

Old Version 52

New Version 53

Key

New Features

New Features Subject to Change

Important Changes between o2o-translate and psana-translate

Calibrated Data

PNCCD::FullFrame

Filtering Events

Split Scan Mode

Reading While Translating

New Features

New Features Subject to Change

Important Changes between o2o-translate and psana-translate

Calibrated Data

PNCCD::FullFrame

Filtering Events

Translation and Damage