Table of Contents

SWMR and Hdf5 Translation

...

The hdf5 group has announced (around Jan 2015) their plans for a virtual view layer. I believe this is coming within 1-2 years? Now we can create a master hdf5 file that contains links to hdf5 groups or whole datasets in other hdf5 files. My understanding of the virtual I/O layer is that within a dataset in the master file, we can link each dataset entry to a different dataset entry in another hdf5 file. This would allow us to translate each of the DAQ streams in parallel, into separate hdf5 files, while maintaining an easy to navigate master file that orders the events appropriately from the different streams.

Compression and HDF5 Translation

Presently when we compress, we limit ourselves to the standard compression algorithms that are available with the Hdf5 library. When users export their data offsite, their offsite installation of hdf5 will be able to decompress the data. Although we could use a custom compression algorithm and get much better performance, we would then we responsible for maintaining builds of the custom compression algorithm for all of our users target platforms, or a mechanism for users to build the library themselves. In a recent technical talk that Qunicy gave (January 2015) I asked if they had thought about including the decompression filter in the hdf5 file in some way, perhaps as Java byte code. He said they had, but one of the goals of hdf5 is longevity of the data - on the order of 40 years. Including java byte code or even a Python script, adds a dependency on Java or Python that is not desirable. He then contrasted this to the current dependency that one gets by using the standard compression algorithms, such as on an external gzip library - but since those algorithms are so common, it is not as problematic of a dependency.