Fixed chunk size
The test scripts are stored here.
https://github.com/slac-lcls/lcls2/tree/master/evtbuild/hdf5_tests/
I evaluated the performance of HDF5 read/write with lots of small appends. The test scripts write a 10 GB file by appending a single 1 MB element at a time. The file is then read back in, again one element at a time. I tested both HDF5's fixed integer and variable length datatypes.
Below I plot histograms of the measured speeds for 1024 and 16 element chunking. In both cases the chunk cache was set to twice the size of the chunk.
The HDF5 group indicated that variable length performance would be poor. However, I see it outperform fixed size in most cases. Variable length writing is significantly faster for both chunking cases and reading 1024 element chunked data. Fixed size reading for 16 element chunks is about 10% faster than variable length.
Overview
Content Tools