Compression methods supported by HDF5
HDF5 supports
- gzip (deflate), gzip - CLI, zlib - API
- Szip, n-bit, scale-offset, and shuffling (with deflate) compression filters. SZIP Compression in HDF
- Users defined compression filters Third-party compression filters
gzip performance
gzip -c test.xtc > test.xtc.gz -rw-r--r-- 1 dubrovin br 168126152 Jan 23 14:48 test.xtc -rw-r--r-- 1 dubrovin br 88829288 Jan 23 14:51 test.xtc.gz compression factor = 1.89, time 30sec
zlib compress/decompress single cspad image
exp=cxitut13:run=10 event 11:
Load data from file nda-cxitut13-r0010-e000011.npy raw data: shape:(32, 185, 388) size:2296960 dtype:int16 [1028 1082 1101 1072 1131]... Start zlib.compress zlib level=0: data size (bytes) in/out = 4593957/4594663 = 1.000 time(sec)=0.025749 t(decomp)=0.005665 zlib level=1: data size (bytes) in/out = 4593957/2922633 = 1.572 time(sec)=0.108629 t(decomp)=0.026618 zlib level=2: data size (bytes) in/out = 4593957/2908156 = 1.580 time(sec)=0.125363 t(decomp)=0.029112 zlib level=3: data size (bytes) in/out = 4593957/2884917 = 1.592 time(sec)=0.170814 t(decomp)=0.027699 zlib level=4: data size (bytes) in/out = 4593957/2886850 = 1.591 time(sec)=0.158719 t(decomp)=0.029466 zlib level=5: data size (bytes) in/out = 4593957/2885665 = 1.592 time(sec)=0.261296 t(decomp)=0.030550 zlib level=6: data size (bytes) in/out = 4593957/2834066 = 1.621 time(sec)=0.597133 t(decomp)=0.027355 zlib level=7: data size (bytes) in/out = 4593957/2828951 = 1.624 time(sec)=0.609569 t(decomp)=0.026842 zlib level=8: data size (bytes) in/out = 4593957/2828951 = 1.624 time(sec)=0.636173 t(decomp)=0.027226 zlib level=9: data size (bytes) in/out = 4593957/2828951 = 1.624 time(sec)=0.611562 t(decomp)=0.027042
References
Overview
Content Tools