Page History
Fixed chunk size
The test scripts are stored here.
...
I evaluated the performance of HDF5 read/write with lots of small appends. The test scripts write a 10 32 GB file by appending a single 1 MB element at a time. The file is then read back in, again one element at a time. I tested both HDF5's fixed integer and variable length datatypes. For comparison, I repeated the test with analogous calls to fread()/fwrite() to a binary file format. The reads/writes are done sequentially with a single core.
Below I plot histograms of the measured speeds for 1024 and 16 element chunking. In both cases the The chunk cache was set to twice the size of the chunk.
The HDF5 group indicated that variable length performance would be poor. However, I see it outperform fixed size in most cases. Variable length writing is significantly faster for both chunking cases and reading 1024 element chunked data. Fixed size reading for 16 element chunks is about 10% faster than variable length.I find that variable length data is slightly faster (5%) in writing and significantly slower (20%) for reading as compared to fixed length. HDF5 is slower in all cases than the binary files by 20-50%.
Overview
Content Tools