Fixed chunk size
The test scripts are stored here.
https://github.com/slac-lcls/lcls2/tree/master/evtbuild/hdf5_tests/
I evaluated the performance of HDF5 read/write with lots of small appends. The test scripts write a 32 GB file by appending a single 1 MB element at a time. The file is then read back in, again one element at a time. I tested both HDF5's fixed integer and variable length datatypes. For comparison, I repeated the test with analogous calls to fread()/fwrite() to a binary file format. The reads/writes are done sequentially with a single core.
Below I plot histograms of the measured speeds for 16 element chunking. The chunk cache was set to twice the size of the chunk.
The HDF5 group indicated that variable length performance would be poor. I find that variable length data is slightly faster (5%) in writing and significantly slower (20%) for reading as compared to fixed length. HDF5 is slower in all cases than the binary files by 20-50%.
Overview
Content Tools