...

The rest of this document covers psana-translate. psana-translate runs as a psana module. As such, we have been able to develop several new features that will be discussed below. However the main technical reason for phasing out o2o-translate is to use a Data Description Language (DDL) to generate code that handles the many data types that different detectors produce. This use of DDL is part of psana-translate.

Running psana-translate

When translating entire runs of an experiment, users should use the web portal of experiments. Translation can produce large amounts of data that needs to be managed by LCLS. The automatic system launches jobs through the batch system using the appropriate queues. The queue used can affect I/O performance. In addition, a faster parallel compression library is used.

You can also run psana-translate by hand. You run it as you would any other psana module. Either through psana command line options or by writing a psana configuration file.  The module is Translator.H5Output. When using the psana command line interface to run the module, the only option that is required to give to the Translator is the name of the output file.  This must be a fully qualified filename, with the output directory.  For example:

...

would invoke the translator.  It will translate all the xtc files in run 71 of the xpptut13 dataset. This runs with default values for all the translator options. These are the recommended option values to use for translation.  The options include gzip compression at level 1 and no filtering on events or data. The Translator does not overwrite an existing h5 output file by default (set the option overwrite=true to overwrite the output). There are many different options you can set for translation that are discussed below. The easiest way to try different translator options is to write a psana.cfg file.  Copy the file default_psana.cfg that is included below or from the Translator package directory and modify option values that you wish.  The file default_psana.cfg includes extensive documentation on all the translator options.

Split Scan Mode

Starting in release ana-0.12.1, the Translator supports split scan mode. In this mode, each calib cycle will be written into a separate hdf5 file. A master file will have external links to the separate calib cycle files. Users need only work with the master file. The master file uses the same schema as one finds without split scan mode, with one exception discussed below (the logging of filtered events). Otherwise, little modification to users code is required when working with the master file. What is required, is following external links (see below for tips on this). Not all experiments use more than one calib cycle. For experiments that use one calib cycle per run, split scan mode provides no benefit. Two reasons to use split scan mode is first, that the resulting hdf5 file from normal translation is too large, and second, to parallelize the translation and make it faster.

When running the Translator directly, you can make use of the same parallel compression library that the automatic translation system uses by setting the following environment variables (below is for bash):

export PAZLIB_MAX_THREADS=8
export LD_PRELOAD=libpazlib.so

before running. Automatic translation also calibrates all cspad data for which calibration constants have been deployed. To achieve this by hand, one can load the module cspad_mod.CsPadCalib as well (it must be loaded before the Translator). When running this module, it is useful to get some of its diagnostic messages to verify that it has found calibration files. After setting PAZLIB_MAX_THREADS and LD_PRELOAD, one could additionally set

export MSGLOGCONFIG=CalibDataProxy=trace

before running

psana -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.output_file=exp-run001.h5 exp=xpptut13:run=71

to translate in the same manner as the automatic translation system. Better I/O is achieved by writing output to the scratch folder of the experiment rather than a users home directory (on the LCSL system) and running translation through the batch system using the appropriate queue.

Split Scan Mode

The Translator supports split scan mode. In this mode, calib cycles are written to separate hdf5 files. A master file will have external links to these separate hdf5 files. Users need only work with the master file. The master file uses the same schema as one finds without split scan mode, with one exception discussed below (the logging of filtered events). Otherwise, little modification to user code is required when working with the master file. What is required, is following external links (see below for tips on this). Not all experiments use more than one calib cycle. For experiments that use one calib cycle per run, split scan mode provides no benefit. Two reasons to use split scan mode is first, that the resulting hdf5 file from normal translation is too large, and second, to make translation faster by translating the data in parallel.

Two versions of split scan mode have been implemented. The most recent (soon to be available in ana-0.13.2 and later) is an MPI (Message Passing Interface) based version. This is the recommended way to Split scan mode. However the previous version is still available and is documented below.

MPI Split Scan

Here is an example command for launching the mpi based splitscan Translator:

bsub -q psanaq -a mympi -n 9 -R "span[ptile=1]" -o translate_%J.out h5-mpi-translate -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.printenv=True -o Translator.H5Output.output_file=mydir/split.h5 exp=xpptut13:run=71

The first few arguments for bsub set up the batch job. The batch job uses 9 processes that are distributed (the ptile=1) onto distinct nodes in the mpi cluster. We use distinct nodes since each translator job will use up to 9 threads with the parallel compression library. As you read through the above command line, you will see that h5-mpi-translate, rather than psana, is the driver program for mpi based split scan translation. An additional option, printenv=True, has been added. This option is only picked up by h5-mpi-translate and can be useful for debugging, but is not necessary (it prints all the environment variables the Translator sees). One node (with the master process) will read through the data, identify the start of calib cycles, and communicate via MPI with the other 8 nodes. The other 8 nodes will do all the translation - telling the master when they are done and ready for a new job. The master will add links to the master h5 output file only after the separate calib files are done.

Generally, there will be one calib cycle file for each calib cycle. However to prevent too many calib cycle files from being produced for experiments that have only a few events per calib cycle, small calib cycles are grouped into one file. They are grouped until the total number of events in the file exceeds a threshold. The threshold defaults to 100 but is configurable.

For the above command, assuming there are multiple calib cycles in run 71 of xpptut13 that each have at least 100 events, you will produce the following files:

mydir/split.h5
mydir/split_cc0000.h5
mydir/split_cc0001.h5
...

When moving the files, make sure they all reside in the same directory. The links from the master to the calib cycle files assume they are in the same directory.

Non-MPI Split Scan

To run the Translator in split scan mode without MPI, three options must be set. These options do the following:

...

psana -m Translator.H5Output -o Translator.H5Output.output_file=mydir/split.h5 -o Translator.H5Output.split=SplitScan -o Translator.H5Output.jobNumber=0 -o Translator.H5Output.jobTotal=2 exp=xpp123:run=10
psana -m Translator.H5Output -o Translator.H5Output.output_file=mydir/split.h5 -o Translator.H5Output.split=SplitScan -o Translator.H5Output.jobNumber=1 -o Translator.H5Output.jobTotal=2 exp=xpp123:run=10

Those commands will create the files

...

(This mode always writes one calib cycle per file).

In this mode

Users need only work with split.h5. When moving the files, make sure they all reside in the same directory. The links from the master to the calib cycle files assume they are in the same directory.

Presently, each Translator job reads through the entire set of xtc files. As more and more Translator jobs are run simultaneously, the overall speed of translation diminishes while the load on the network steadily increases. It is recommended that users run no more than 5 Translator jobs. Testing has show 5 jobs provide up to a three times speed up in translation.

...