Support BITSHUFFLE option of Blosc >= v0.7.3 #899
Conversation
Blosc 0.7.3 now treats the blosc "shuffle" option as an Integer rather than a Bool. This allows the use of all three of blosc's shuffling options: no shuffling (0), byte shuffling (1), or bit shuffling (2). The `shuffle` field of the BloscFilter struct is not restricted to Bool, but it was effectively being converted to Bool via `!=(0)` in the `blosc_filter()` function of the H5Zblosc plugin before passing to the `blosc_compress()` function. This prevented the use of bit shuffling because `2 != 0` is `true` which is the same as 1 (byte shuffling). The H5Zblosc plugin no longer converts the given value to Bool and also validates the value against the three supported shuffle option values. These values are also imported into the H5Zblosc module so users can refer to them as one of: - H5Zblosc.NOSHUFFLE - H5Zblosc.SHUFFLE - H5Zblosc.BITSHUFFLE
|
Thank you! |
|
We really should consider a strategy of shifting everything to some version of Blosc2 possibly starting with incorporation of Blosc2.jl into JuliaIO. |
|
Agreed |
|
https://github.com/waralex/Blosc2.jl says it's WIP, I wonder what's still missing. Is he open potentially to moving to JuliaIO? |
|
With regard to this filter we should also review how similar it is to these reference implementations: |
|
These are all great comments vis a vis Blosc2, but perhaps they would be better placed in a new Blosc2 feature request issue so they don't get lost here. Just saying... |
|
The operating issue is JuliaIO/Blosc2.jl#6 |
Blosc 0.7.3 now treats the blosc "shuffle" option as an Integer rather
than a Bool. This allows the use of all three of blosc's shuffling
options: no shuffling (0), byte shuffling (1), or bit shuffling (2).
The
shufflefield of the BloscFilter struct is not restricted to Bool,but it was effectively being converted to Bool via
!=(0)in theblosc_filter()function of the H5Zblosc plugin before passing to theblosc_compress()function. This prevented the use of bit shufflingbecause
2 != 0istruewhich is the same as 1 (byte shuffling).The H5Zblosc plugin no longer converts the given value to Bool and also
validates the value against the three supported shuffle option values.
These values are also imported into the H5Zblosc module so users can
refer to them as one of:
closes #285