Page History

...

Two versions of split scan mode have been implemented. The most recent (soon to be available in ana-0.13.2 and later) is an MPI (Message Passing Interface) based version. This is the recommended way to Split run split scan mode. However the previous version is still available and is documented below.

...

HDF5 presently has little support for reading a file that is being created, and in general does not recommend this. However the master file is written in a way to support this as well as possible. Most all links from When using the MPI split scan translator, links are not added to the master file to after the calib cycle files are not created until the calib cycle file is finished. The exception is done. Thus it is always safe to traverse those links. With the non-MPI split scan translator this is true except for the last N links, where N is the number of jobs running. These links may be written before the calib cycle files they link to are finished. To see updates in the master file, users may need to shut down programs like Matlab and h5py and restart them. It is not sufficient to close and reopen the master file within a Python or Matlab session.

...

hdf file creation parameters
Only NoSplit is implemented - no family or split drivers.

In general a number of o2o-translate options are no longer supported.  In particular:
-G (long names like CalibCycle:0000 instead of CalibCycle) is always on.

Speed

Comparison with o2o-translate

psana-translate runs about 10% slower than o2o-translate does.

Performance testing was done during November/December of 2013.  Both o2o-translate and psana-translate worked through a 92 GB xtc file using compression=1 on the rhat6 machine psdev105.  They read and wrote the data from /u1. They both used the non-parallel compression library.  o2o-translate produced a 68GB file in 65 minutes and psana-translate produced a 65GB file in 70 minutes.  (Speeds of about 22MB/sec).  Production runs will use the parallel compression library and are expected to run at faster speeds (about 50MB/sec).faster.

Single psana-translate

I parsed through psana-translate logs on 9/25/2014. An estimate of the median speed is 51MB/sec. The 25th and 75th percentile speeds look to be 37MB/sec and 69MB/sec. I've made some effort to parse logs that only include full translations with the default compression. Among such translations, I would expect the differences on speeds to be based on how much cspad is being calibrated, the load on the filesystem and cores, and how much damaged data appears and is not processed (will make the speed seem artificially higher). I assume that the size of the xtc files on disk is the amount of data translated.

A limitation on translation speed is the time it takes psana to read through all the data. I estimate this to be around 250MB/sec, putting translation at 5 times psana reading. I believe the file system supports 300-400MB/sec, but this depends heavily on the load, however this may not be a good benchmark for psana. A better benchmarch is a simple program that parses through the xtc - touching all the payloads.

Timing MPI Split Scan Translation

Below is a table of some testing results for the MPI split scan translator. All of this was carried out on the offline file system. A typical command was

bsub -q psanaq -a mympi -n 12 -R "span[ptile=1]" h5-mpi-translate -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.output_file=/reg/data/ana01/temp/davidsch/out.h5 exp=xpptut13:run=3

With the environment variables set to load the parallel compression library. All jobs run on the psanaq and write to ana01. Two of the jobs only had one calib cycle (xpp74813:run=69 and xpp40312:run=48) from which we might say the baseline translation speeds were 25-30 mb/sec in these tests. When investigating why this was less than the median 51MB/sec, I did see a high load on the system -however it may also be the overhead of having two readers. When running mpi split scan on a run with only one calib cycle - two different nodes will hit the same xtc on disk for a while. At some point the master process will get further along in the files.

Some of the columns are explained here:

• WJobs - how many worker jobs. If there are at least 100 events in a calib cycle, it is one calib cycle per worker job. Otherwise as many calib cycles as needed to hit 100.
• CC/WJ - average number of calib cycles per worker job. Usually 1.
• mread - time for the master to read through the data. The rest of the time it waits for worker nodes to finish.
• calib - how many distinct cspad sources were calibrated in the h5.
• WJtime - average time for one worker job. In principle, something close to mread + WJtime would be the minimum time we expect to achieve.
  *wn - for the n workers, report the number of worker jobs done, and the percent of the total time the worker was translating, vs. idly waiting for a worker job. For example, w0=19/96% means worker 0 processed 19 worker jobs in 96% of the total time.

Technical Difference's with o2o-translate

...