Page History

...

bsub -q psanaq -a mympi -n 9 -R "span[ptile=1]" -o translate_%J.out h5-mpi-translate -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.printenv=True -o Translator.H5Output.output_file=mydir/split.h5 exp=xpptut13:run=71

More study is needed to see if this is the optimal way to launch a job. The first few arguments for bsub set up the batch job, everything after h5-mpi-translate are arguments for the mpi split scan translator.

batch arguemnts

• The queue used is psanaq - standard for the offline file system. See Submitting Batch Jobs for more details. When using a different queue, that is the high priority queues (psnehprior, psfehprior) while taking data during an experiment, note these should only be used to read data from the ffb - they do not have fast disk access to the offline file system.
• The batch job uses 9 processes that are distributed (the ptile=1) onto distinct nodes in the mpi cluster. We use distinct nodes since each translator job will use up to 9 threads with the parallel compression library - however this uses significantly more resources in the queue.
• The job output file (translate_39283.out) will record timing information for the master and workers.
  
  

Translator arguments

As you read through the above command line, you will see that h5-mpi-translate, rather than psana, is the driver program for mpi based split scan translation. An additional option, printenv=True, h5-mpi-translate takes the same arguments as psana does. It understands one additional option, printenv=True, which has been added. This option is only picked up by h5-mpi-translate and can be useful for debugging, but is not necessary (it prints all the environment variables the Translator sees). In particular, h5-mpi-translate understands the -c file.cfg option so that options may be specified in a configuration file.

Operation

When h5-mpi-translate runs, one . One node (with the master process) will read through the data, identify the start of calib cycles, and communicate via MPI with the other 8 nodes. The other 8 nodes will do all the translation - telling the master when they are done and ready for a new job. The master will add links to the master h5 output file only after the separate calib files are done.

Generally, there will be one calib cycle file for each calib cycle. However to prevent too many calib cycle files from being produced for experiments that have only a few events per calib cycle, small calib cycles are grouped into one file. They are grouped until the total number of events in the file exceeds a threshold. The threshold defaults to around 100 but is configurable.

For the above command, assuming there are multiple calib cycles in run 71 of xpptut13 that each have at least 100 events, and that the batch job was 39283, you will produce the following files:

...

When moving the hdf5 files, make sure they all reside in the same directory. The links from the master to the calib cycle files assume they are in the same directory.

Calib files in sub directory

While the split scan translator defaults to putting all files, the master file, and translated calib cycle files, into the same directory, users can optionally put the calib files into a sub-directory. The option is "split_cc_in_subdir". For example, when translating the above datasource, if one were to use a config file and add an option like so

[Translator.H5Output]
split_cc_in_subdir=True

Then one will get all the calib files in a directory called xpptut13_r0071_ccfiles and the translated files there:

mydir/xpptut13-r0071.h5
mydir/xpptut13-r0071_ccfiles/xpptut13-r0071_cc0000.h5
mydir/xpptut13-r0071_ccfiles/xpptut13-r0071_cc0001.h5
...

Live Data and Fast Indexing 

For online analysis with live data, one of the impediments to keeping up with the data is the time it takes h5-mpi-translate's to read through the data to find the calib cycles. Typically the data is recorded in 6 or more separate files and each must be read through to identify the start of calib cycles. Unfortunately this read speed can be 30-40hz for a typical experiment - far short of the 120hz we'd like to obtain. A recent feature added to h5-mpi-translate takes advantage of the unique signature of each new calib cycle, combined with the regular structure of the separate data files in order to limit the reading to just one of the files. In this way, the h5-mpi-translate master rank need only get through the data it reads/searches at 20hz to keep up with the data. We have had good success with this feature recently, but it is not guaranteed to work. It is a temporary solution until a more robust way to do fast/live indexing is put in place. However in the meantime, starting with analysis release ana-0.13.17, the translator supports the following options to turn on fast indexing and controlling how much time is spent searching the other files:

fast_index=1                 # do the fast indexing, be default it is off
fi_mb_half_block=12          # when fast indexing is on, use 12MB on each side, or 24MB for each block that is searched
fi_num_blocks=3              # this it half the number of 'other' blocks to try. The translator will try 1 + 2*3 = 7 blocks if this is 3 

More information can be found in the Psana Configuration File and All the Options section belowThe job output file (translate_39283.out) will record timing information for the master and workers. Identifying the optimal number of nodes for h5-mpi-translate requires further analysis, however you can identify idle workers or a poor distribution of the work by looking at the report in the batch job output file.

Non-MPI Split Scan

To run the Translator in split scan mode without MPI, three options must be set. These options do the following:

...