Page History

From: help@hdfgroup.org [help@hdfgroup.org]
Sent: Thursday, January 29, 2015 6:54 AM
To: Schneider, David A.
Subject: Re: SWMR (fwd)

Hi David,

I received word from the developer. His replies are interspersed in the
message below.

-Barbara

> I maintain and develop the LCLS HDF5 Translator at SLAC, I work for Amedeo Perazzo. 
> I wrote a while ago (10/20/2014)and Barbara Jones wrote back and sent me theselinks to the SWMR prototype:
>
> ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/SWMR/[ftp.hdfgroup.uiuc.edu]
> The documentation is located here: http://www.hdfgroup.org/HDF5/docNewFeatures/
>
> I was wondering if those are still the best current links?

        Yes, I believe those are still the best links.

> I've been reading the documentation on the above link, and there are a few things about the model that may 
> pose problems for us, the way we do things presently.I wanted to see if they are limitations of the prototype only, or will be part of the production SWMR model.
>
> These are:
>
> 1. The writer is not allowed to add any new objects to the file such as groups, datasets, links,
> committed datatypes, and attributes.
>
> Right now, the Data Acquisition system can deliver data in what we call calib cycles. There can be many calib cycles in a run, and each calib cycle 
> triggers a new set of groups to be created in the hdf5 file. Then, when we start receiving data for a calib cycle, we create additional subgroups 
> based on each of the detectors that are emitting data. Presently, information about what these groups will be is not as robust as waiting until 
> we get the first piece of data from a given detector. At that point we create the group that will hold its dataset.

        Yes, this limitation is planned to be in place, unless we can get some additional funding to finish out "full SWMR support" 
for all the metadata in the HDF5 file.  It's definitely something we know how to do, it's just a matter of getting enough funded 
manpower behind the effort.

> 3. The writer is not allowed to modify or append to any data items containing variable-size
> datatypes (including string and region references datatypes).
>
> we would not need to rewrite a particular vlen entry in a dataset, but we would want to append items to a vlen dataset.

        Also fixable, with some additional funding.

> Also for this:
> WORD of CAUTION: Please note that HDF5 files created by the SWMR HDF5 library may not be read by
> the tools based on the HDF5 1.8 libraries due to the new features introduced in the HDF5 development
> branch (see the link above). An application built with the HDF5 Library with SWMR capability can always
> read and modify files created by all previous versions of HDF5.
>
> Does that only apply to reading the file while it is being written? After the SWMR writer closes the file, 
> could existing 1.8 hdf5 based tools read the hdf5 file?

        Currently, it applies both while the file is open and after it's closed.  However, we are planning to write a tool to "downgrade" 
the file format so that the 1.8.x releases can read/write the file produced.


-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Barbara Jones
The HDF Helpdesk
help@hdfgroup.org

Support Services:  http://www.hdfgroup.org/services/

To let us know who you are or how we can improve our
products or services, see:

   http://www.hdfgroup.org/downloads/registration.php

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+