...

hdf file creation parameters
Only NoSplit is implemented - no family or split drivers.

In general a number of o2o-translate options are no longer supported.  In particular:
-G (long names like CalibCycle:0000 instead of CalibCycle) is always on.

Speed

...

Technical Difference's with o2o-translate

Below is a list of technical differences between psana-translate runs about 10% slower than and o2o-translate does.
Performance testing was done during November/December of 2013.  Both o2o-translate and psana-translate worked through a 92 GB xtc file using compression=1 on the rhat6 machine psdev105.  They read and wrote the data from /u1. They both used the non-parallel compression library.  o2o-translate produced a 68GB file in 65 minutes and psana-translate produced a 65GB file in 70 minutes.  (Speeds of about 22MB/sec).  Production runs will use the parallel compression library and are faster.

Single psana-translate

I parsed through psana-translate logs on 9/25/2014. An estimate of the median speed is 51MB/sec. The 25th and 75th percentile speeds look to be 37MB/sec and 69MB/sec. I've made some effort to parse logs that only include full translations with the default compression. Among such translations, I would expect the differences on speeds to be based on how much cspad is being calibrated, the load on the filesystem and cores, and how much damaged data appears and is not processed (will make the speed seem artificially higher). I assume that the size of the xtc files on disk is the amount of data translated.

A limitation on translation speed is the time it takes psana to read through all the data. I estimate this to be around 250MB/sec, putting translation at 5 times psana reading. I believe the file system supports 300-400MB/sec, but this depends heavily on the load, however this may not be a good benchmark for psana. A better benchmarch is a simple program that parses through the xtc - touching all the payloads.

Timing MPI Split Scan Translation

Below is a table of some testing results for the MPI split scan translator. All of this was carried out on the offline file system. A typical command was

bsub -q psanaq -a mympi -n 12 -R "span[ptile=1]" h5-mpi-translate -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.output_file=/reg/data/ana01/temp/davidsch/out.h5 exp=xpptut13:run=3

With the environment variables set to load the parallel compression library. All jobs run on the psanaq and write to ana01. Two of the jobs only had one calib cycle (xpp74813:run=69 and xpp40312:run=48) from which we might say the baseline translation speeds were 25-30 mb/sec in these tests. When investigating why this was less than the median 51MB/sec, I did see a high load on the system -however it may also be the overhead of having two readers. When running mpi split scan on a run with only one calib cycle - two different nodes will hit the same xtc on disk for a while. At some point the master process will get further along in the files.

Some of the columns are explained here:

• WJobs - how many worker jobs. If there are at least 100 events in a calib cycle, it is one calib cycle per worker job. Otherwise as many calib cycles as needed to hit 100.
• CC/WJ - average number of calib cycles per worker job. Usually 1.
• mread - time for the master to read through the data. The rest of the time it waits for worker nodes to finish.
• calib - how many distinct cspad sources were calibrated in the h5.
• WJtime - average time for one worker job. In principle, something close to mread + WJtime would be the minimum time we expect to achieve.
  *wn - for the n workers, report the number of worker jobs done, and the percent of the total time the worker was translating, vs. idly waiting for a worker job. For example, w0=19/96% means worker 0 processed 19 worker jobs in 96% of the total time.

Technical Difference's with o2o-translate

Below is a list of technical differences between psana-translate and o2o-translate. These differences should not affect users.

• File attributes runNumber, runType and experiment not stored, instead expNum, experiment, instrument and jobName are stored (from the psana Env object)
• The attribute :schema:timestamp-format is always "full", there is no option for "short"
• The output file must be explicitly specificed in the psana cfg file. It is not inferred from the input.
• The File attribute origin is now psana-translator as opposed to translator
• The end sec and nanoseconds are not written into the Configure group at the end of the job as there is no EventId in the Event at the end.
• integer size changes - a number of fields have changed size, a few examples are below.  In one quirky case, this caused translation to be different.  The reason was that the data was uninitialized, and the new 32 bit value was different than the old 16 bit value. Data produced from 2014 onward will not include unitialized data in the translation, users will not have to worry about.  Unitialized data is very rare in pre 2014 data and, due to its location, not likely to be used in analysis.
• A few Examples of field size changes:
  • EvrData::ConfigV7/seq_config - sync_source - enum was uint16, now uint32
  • EvrData::ConfigV7/seq_config - beam_source - enum was uint16, now uint32
  • Ipimb::DataV2 - source_id was uint16, now uint8
  • Ipimb::DataV2 - conn_id was uint16 now uint8
  • Ipimb::DataV2 - module was uint16, now uint8

Some types have their field names rearranged. For example with ControlData::ConfigV2 one has:

...

. These differences should not affect users.

• File attributes runNumber, runType and experiment not stored, instead expNum, experiment, instrument and jobName are stored (from the psana Env object)
• The attribute :schema:timestamp-format is always "full", there is no option for "short"
• The output file must be explicitly specificed in the psana cfg file. It is not inferred from the input.
• The File attribute origin is now psana-translator as opposed to translator
• The end sec and nanoseconds are not written into the Configure group at the end of the job as there is no EventId in the Event at the end.
• integer size changes - a number of fields have changed size, a few examples are below.  In one quirky case, this caused translation to be different.  The reason was that the data was uninitialized, and the new 32 bit value was different than the old 16 bit value. Data produced from 2014 onward will not include unitialized data in the translation, users will not have to worry about.  Unitialized data is very rare in pre 2014 data and, due to its location, not likely to be used in analysis.
• A few Examples of field size changes:
  • EvrData::ConfigV7/seq_config - sync_source - enum was uint16, now uint32
  • EvrData::ConfigV7/seq_config - beam_source - enum was uint16, now uint32
  • Ipimb::DataV2 - source_id was uint16, now uint8
  • Ipimb::DataV2 - conn_id was uint16 now uint8
  • Ipimb::DataV2 - module was uint16, now uint8

Some types have their field names rearranged. For example with ControlData::ConfigV2 one has:

ControlData::ConfigV2:
  o2o: uses_duration uses_events duration events npvControls npvMonitors npvLabels
psana: events uses_duration uses_events duration npvControls npvMonitors npvLabels

EvrData::ConfigV7:
  o2o: code isReadout isCommand isLatch reportDelay reportWidth releaseCode maskTrigger maskSet maskClear desc readoutGroup
psana: code isReadout isCommand isLatch reportDelay reportWidth maskTrigger maskSet maskClear desc readoutGroup releaseCode

Epics Ctrl datasets (in the configure group as opposed to the calib group) are not chunked.  They are stored as fixed size datasets depending on the number of pv's.

Only one epics pv is stored per name (of course, one epics pv may have any number of elements within it). This is fine as the epic pv name is supposed to uniquely identify the pv.  However in xtc files, you can see several epics pv's with the same pvname, but different pvid's. This scenario should only arise when the same pv is coming from different sources, and replicates the same data.  Psana only stores one epics pv per name (the last one it sees in a datagram). This is the one that psana-translate will pick up and store.

All Epics pv's are stored in the source folder EpicsArch.0:NoDevice.0.  With o2o-translate, some could be split off into other folders (such as AmoVMI.0:Opal1000.0). As epics pv names uniquely identify the data, the source information should not be needed.

Some DAQ config objects include space for a maximum number of entries.  o2o-translate would only write entries for those used, not the maximum entries.  psana-translate does not.  For example:

• The Acqiris::ConfigV1 vert dataset now always prints the max of 20 channels, even if the user will only be using 3.
  • Note, in this case the Acqiris data will still only include the 3 channels being used. o2o-translate was making an adjustment to the config data being written.

psana-translate will write an emtpy output_lookup_table for the Opal1k::ConfigV1 dataset named output_lookup_table, even if output_lookup_table() is enabled.  o2o-translate would not.

psana-translate does not produce the _dim_fix_flag_20103 datasets that o2o-translate did.

Bld::BldDataGMDV  the field fSpare1 has been dropped from this type.

With psana-translate, if all the xtc's coming from a particular source are damaged, you will not see a 'data' dataset in the hdf5 file. You will see the time, _damage and _mask datasets that tell the damage and events where the omitted data lives. o2o-translate may have created a 'data' dataset filled with blanks.

As discussed above, OutOfOrder Damage is not translated by default. o2o-translate translated out of order damage, however psana-translate does not.  psana can be told to include this kind of damaged data by setting store-out-of-order-damage=true in the [psana] section of your .cfg file.

When the number of events is recorded in the control data, o2o-translate would set the chunk size based on this value.  psana-translate does the same.  However o2o-translate also looked at the actual number of events and used this as well to set chunk sizes in future calib cycles.  psana-translate does not do this latter part.

Speed

Comparison with o2o-translate

psana-translate runs about 10% slower than o2o-translate does.

Performance testing was done during November/December of 2013.  Both o2o-translate and psana-translate worked through a 92 GB xtc file using compression=1 on the rhat6 machine psdev105.  They read and wrote the data from /u1. They both used the non-parallel compression library.  o2o-translate produced a 68GB file in 65 minutes and psana-translate produced a 65GB file in 70 minutes.  (Speeds of about 22MB/sec).  Production runs will use the parallel compression library and are faster.

Single psana-translate

I parsed through psana-translate logs on 9/25/2014. An estimate of the median speed is 51MB/sec. The 25th and 75th percentile speeds look to be 37MB/sec and 69MB/sec. I've made some effort to parse logs that only include full translations with the default compression. Among such translations, I would expect the differences on speeds to be based on how much cspad is being calibrated, the load on the filesystem and cores, and how much damaged data appears and is not processed (will make the speed seem artificially higher). I assume that the size of the xtc files on disk is the amount of data translated.

A limitation on translation speed is the time it takes psana to read through all the data. I estimate this to be around 250MB/sec, putting translation at 5 times psana reading. I believe the file system supports 300-400MB/sec, but this depends heavily on the load, however this may not be a good benchmark for psana. A better benchmarch is a simple program that parses through the xtc - touching all the payloads.

Timing MPI Split Scan Translation

Below is a table of some testing results for the MPI split scan translator. All of this was carried out on the offline file system. A typical command was

bsub -q psanaq -a mympi -n 12 -R "span[ptile=1]" h5-mpi-translate -m cspad_mod.CsPadCalib,Translator.H5Output -o Translator.H5Output.output_file=/reg/data/ana01/temp/davidsch/out.h5 exp=xpptut13:run=3

With the environment variables set to load the parallel compression library. All jobs run on the psanaq and write to ana01. Two of the jobs only had one calib cycle (xpp74813:run=69 and xpp40312:run=48) from which we might say the baseline translation speeds were 25-30 mb/sec in these tests. When investigating why this was less than the median 51MB/sec, I did see a high load on the system -however it may also be the overhead of having two readers. When running mpi split scan on a run with only one calib cycle - two different nodes will hit the same xtc on disk for a while. At some point the master process will get further along in the files.

Some of the columns are explained here:

• WJobs - how many worker jobs. If there are at least 100 events in a calib cycle, it is one calib cycle per worker job. Otherwise as many calib cycles as needed to hit 100.
• CC/WJ - average number of calib cycles per worker job. Usually 1.
• mread - time for the master to read through the data. The rest of the time it waits for worker nodes to finish.
• calib - how many distinct cspad sources were calibrated in the h5.
• WJtime - average time for one worker job. In principle, something close to mread + WJtime would be the minimum time we expect to achieve.
  *wn - for the n workers, report the number of worker jobs done, and the percent of the total time the worker was translating, vs. idly waiting for a worker job. For example, w0=19/96% means worker 0 processed 19 worker jobs in 96% of the total time.

Epics Ctrl datasets (in the configure group as opposed to the calib group) are not chunked.  They are stored as fixed size datasets depending on the number of pv's.

Only one epics pv is stored per name (of course, one epics pv may have any number of elements within it). This is fine as the epic pv name is supposed to uniquely identify the pv.  However in xtc files, you can see several epics pv's with the same pvname, but different pvid's. This scenario should only arise when the same pv is coming from different sources, and replicates the same data.  Psana only stores one epics pv per name (the last one it sees in a datagram). This is the one that psana-translate will pick up and store.

All Epics pv's are stored in the source folder EpicsArch.0:NoDevice.0.  With o2o-translate, some could be split off into other folders (such as AmoVMI.0:Opal1000.0). As epics pv names uniquely identify the data, the source information should not be needed.

Some DAQ config objects include space for a maximum number of entries.  o2o-translate would only write entries for those used, not the maximum entries.  psana-translate does not.  For example:

• The Acqiris::ConfigV1 vert dataset now always prints the max of 20 channels, even if the user will only be using 3.
  • Note, in this case the Acqiris data will still only include the 3 channels being used. o2o-translate was making an adjustment to the config data being written.

psana-translate will write an emtpy output_lookup_table for the Opal1k::ConfigV1 dataset named output_lookup_table, even if output_lookup_table() is enabled.  o2o-translate would not.

psana-translate does not produce the _dim_fix_flag_20103 datasets that o2o-translate did.

Bld::BldDataGMDV  the field fSpare1 has been dropped from this type.

With psana-translate, if all the xtc's coming from a particular source are damaged, you will not see a 'data' dataset in the hdf5 file. You will see the time, _damage and _mask datasets that tell the damage and events where the omitted data lives. o2o-translate may have created a 'data' dataset filled with blanks.

As discussed above, OutOfOrder Damage is not translated by default. o2o-translate translated out of order damage, however psana-translate does not.  psana can be told to include this kind of damaged data by setting store-out-of-order-damage=true in the [psana] section of your .cfg file.

When the number of events is recorded in the control data, o2o-translate would set the chunk size based on this value.  psana-translate does the same.  However o2o-translate also looked at the actual number of events and used this as well to set chunk sizes in future calib cycles.  psana-translate does not do this latter part.

...