SWMR and Hdf5 Translation

Recently (Jan 29 2015) I (davidsch) took a look at the beta work of the SWMR model (single writer multiple readers) that the hdf5 group is working on. As it is planned now, there are a few limitations that would necessitate changes in how the DAQ records data, as well as in the Translator. Below is an email exchange with the hdf5 group that covers this. The summary is that SWMR will only support the datasets, all the groups have to be known and created before SWMR can take place.

Another limitation that we gathered after meeting with Quincy at SLAC sometime in late 2014 was that you cannot start moving hdf5 files as soon as they are written. That is once the file is closed, the library needs to do some things to the header of the file. This would affect our data management model were we to write natively to hdf5.

email transcript:

From: help@hdfgroup.org [help@hdfgroup.org]
Sent: Thursday, January 29, 2015 6:54 AM
To: Schneider, David A.
Subject: Re: SWMR (fwd)

Hi David,

I received word from the developer. His replies are interspersed in the
message below.

-Barbara

> I maintain and develop the LCLS HDF5 Translator at SLAC, I work for Amedeo Perazzo. 
> I wrote a while ago (10/20/2014)and Barbara Jones wrote back and sent me theselinks to the SWMR prototype:
>
> ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/SWMR/[ftp.hdfgroup.uiuc.edu]
> The documentation is located here: http://www.hdfgroup.org/HDF5/docNewFeatures/
>
> I was wondering if those are still the best current links?

        Yes, I believe those are still the best links.

> I've been reading the documentation on the above link, and there are a few things about the model that may 
> pose problems for us, the way we do things presently.I wanted to see if they are limitations of the prototype only, or will be part of the production SWMR model.
>
> These are:
>
> 1. The writer is not allowed to add any new objects to the file such as groups, datasets, links,
> committed datatypes, and attributes.
>
> Right now, the Data Acquisition system can deliver data in what we call calib cycles. There can be many calib cycles in a run, and each calib cycle 
> triggers a new set of groups to be created in the hdf5 file. Then, when we start receiving data for a calib cycle, we create additional subgroups 
> based on each of the detectors that are emitting data. Presently, information about what these groups will be is not as robust as waiting until 
> we get the first piece of data from a given detector. At that point we create the group that will hold its dataset.

Yes, this limitation is planned to be in place, unless we can get some additional funding to finish out "full SWMR support" 
for all the metadata in the HDF5 file.  It's definitely something we know how to do, it's just a matter of getting enough funded 
manpower behind the effort.

> 3. The writer is not allowed to modify or append to any data items containing variable-size
> datatypes (including string and region references datatypes).
>
> we would not need to rewrite a particular vlen entry in a dataset, but we would want to append items to a vlen dataset.

        Also fixable, with some additional funding.

> Also for this:
> WORD of CAUTION: Please note that HDF5 files created by the SWMR HDF5 library may not be read by
> the tools based on the HDF5 1.8 libraries due to the new features introduced in the HDF5 development
> branch (see the link above). An application built with the HDF5 Library with SWMR capability can always
> read and modify files created by all previous versions of HDF5.
>
> Does that only apply to reading the file while it is being written? After the SWMR writer closes the file, 
> could existing 1.8 hdf5 based tools read the hdf5 file?

Currently, it applies both while the file is open and after it's closed.  However, we are planning to write a tool to "downgrade" 
the file format so that the 1.8.x releases can read/write the file produced.


-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Barbara Jones
The HDF Helpdesk
help@hdfgroup.org

Support Services:  http://www.hdfgroup.org/services/

To let us know who you are or how we can improve our
products or services, see:

   http://www.hdfgroup.org/downloads/registration.php

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Parallel I/O and HDF5 Translation

The Hdf5 library currently supports parallel I/O using MPI underneath. In the meeting with Qunicy at SLAC in late 2014 we get a good sense of how we would have to change things to use this model. Here are the main points and consequences:

• In this model, you create all your groups and datasets collectively, and then different MPI ranks fill in the datasets.
• No dataset can contain variable length data nor can a dataset contain compressed data.

This would be a big change for the Translator. It seems we would have to wait for the xtc to be written so that we know what datasets to create, and how long each one is (how many events we took, etc). So we wouldn't be able to do 'live' translation, and let users read parts of the translation (like a translated calib cycle) while the whole run is getting translated (see The XTC-to-HDF5 Translator - the MPI split scan section). Also no compression is a big limitation for users moving files offsite. However I would expect faster translation.

It may be that the hdf5 group can lift these limitations and are working on it - I believe I saw a reference to parallel compressed data in a white paper Amedeo sent around once, but no details.

Virtual View and HDF5 Translation

The hdf5 group has announced (around Jan 2015) their plans for a virtual view layer. I believe this is coming within 1-2 years? Now we can create a master hdf5 file that contains links to hdf5 groups or whole datasets in other hdf5 files. My understanding of the virtual I/O layer is that within a dataset in the master file, we can link each dataset entry to a different dataset entry in another hdf5 file. This would allow us to translate each of the DAQ streams in parallel, into separate hdf5 files, while maintaining an easy to navigate master file that orders the events appropriately from the different streams.

Compression and HDF5 Translation

Presently when we compress, we limit ourselves to the standard compression algorithms that are available with the Hdf5 library. When users export their data offsite, their offsite installation of hdf5 will be able to decompress the data. Although we could use a custom compression algorithm and get much better performance, we would then we responsible for maintaining builds of the custom compression algorithm for all of our users target platforms, or a mechanism for users to build the library themselves. In a recent technical talk that Qunicy gave (January 2015) I asked if they had thought about including the decompression filter in the hdf5 file in some way, perhaps as Java byte code. He said they had, but one of the goals of hdf5 is longevity of the data - on the order of 40 years. Including java byte code or even a Python script, adds a dependency on Java or Python that is not desirable. He then contrasted this to the current dependency that one gets by using the standard compression algorithms, such as on an external gzip library - but since those algorithms are so common, it is not as problematic of a dependency.