szip support with libaec doesn't work with hdf5 =1.14.1

[dwr-psandhu]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

szip support was added with this pull request #198
However it stopped working with hdf5=1.14.1. See this discussion at the end of the pull request
#198 (comment)

Opening this issue to ask @mkitti to investigate.

Installed packages

# packages in environment at c:\Users\psandhu\Miniconda3\envs\hdf5_test_szip_latest:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2023.7.22            h56e8100_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
h5py                      3.9.0           nompi_py311hc8b35be_101    conda-forge
hdf5                      1.14.1          nompi_h73e8ff5_100    conda-forge
intel-openmp              2023.2.0         h57928b3_49496    conda-forge
krb5                      1.21.2               heb0366b_0    conda-forge
libaec                    1.0.6                h63175ca_1    conda-forge
libblas                   3.9.0              17_win64_mkl    conda-forge
libcblas                  3.9.0              17_win64_mkl    conda-forge
libcurl                   8.2.1                hd5e4a3a_0    conda-forge
libexpat                  2.5.0                h63175ca_1    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
libhwloc                  2.9.2           nocuda_h15da153_1008    conda-forge
libiconv                  1.17                 h8ffe710_0    conda-forge
liblapack                 3.9.0              17_win64_mkl    conda-forge
libsqlite                 3.42.0               hcfcfb64_0    conda-forge
libssh2                   1.11.0               h7dfc565_0    conda-forge
libxml2                   2.11.5               hc3477c8_0    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
mkl                       2022.1.0           h6a75c08_874    conda-forge
numpy                     1.25.2          py311h0b4df5a_0    conda-forge
openssl                   3.1.2                hcfcfb64_0    conda-forge
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pthreads-win32            2.9.1                hfa6e2cd_3    conda-forge
python                    3.11.4          h2628c8c_0_cpython    conda-forge
python_abi                3.11                    3_cp311    conda-forge
setuptools                68.0.0             pyhd8ed1ab_0    conda-forge
tbb                       2021.10.0            h91493d7_0    conda-forge
tk                        8.6.12               h8ffe710_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
vc                        14.3                h64f974e_17    conda-forge
vc14_runtime              14.36.32532         hfdfe4a8_17    conda-forge
vs2015_runtime            14.36.32532         h05e6639_17    conda-forge
wheel                     0.41.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge

Environment info

(hdf5_test_szip_latest) D:\dev\pydsm>conda info

     active environment : hdf5_test_szip_latest
    active env location : c:\Users\psandhu\Miniconda3\envs\hdf5_test_szip_latest
            shell level : 3
       user config file : C:\Users\psandhu\.condarc
 populated config files : C:\Users\psandhu\.condarc
          conda version : 23.5.0
    conda-build version : 3.23.3
         python version : 3.8.16.final.0
       virtual packages : __archspec=1=x86_64
                          __cuda=10.2=0
                          __win=0=0
       base environment : c:\Users\psandhu\Miniconda3  (writable)
      conda av data dir : c:\Users\psandhu\Miniconda3\etc\conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : c:\Users\psandhu\Miniconda3\pkgs
                          C:\Users\psandhu\.conda\pkgs
                          C:\Users\psandhu\AppData\Local\conda\conda\pkgs
       envs directories : c:\Users\psandhu\Miniconda3\envs
                          C:\Users\psandhu\.conda\envs
                          C:\Users\psandhu\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/23.5.0 requests/2.29.0 CPython/3.8.16 Windows/10 Windows/10.0.19044 solver/libmamba conda-libmamba-solver/22.8.1 libmambapy/1.3.1
          administrator : False
             netrc file : None
           offline mode : False

[dwr-psandhu]

Here's a small test that could be added based on @mkitti previous work #197 (comment)

# test szip
import h5py
import os

def test_szip_available():
    with h5py.File("test.h5", "w") as h5f:
        ds = h5f.create_dataset("test", (1024, 1024), chunks=(16,16), dtype="uint16", compression="szip", compression_opts=('ec',32))
        assert ds.compression == "szip"
    # Optional: Clean up the created file after the test
    if os.path.exists("test.h5"):
        os.remove("test.h5")

[mkitti]

I suspect this is a fixed upstream issue
HDFGroup/hdf5#3036

[mkitti]

develop branch fix HDFGroup/hdf5#3035

[mkitti]

I think this may be fixed by 1.14.2:
HDFGroup/hdf5@e03ffa1

[dwr-psandhu]

Looks like the patch for windows libaec was removed
fef23d9

1.14.2 is out so when will the feedstock update? It will need to back out this commit if it has to build
ab2be8d

Both changes made by @hmaarrfk

[hmaarrfk]

hmm, sorry, i think i just forgot to re-include it. often these build patches get merged upstream quickly and i must have wrongly assumed. Are you able to verify that including the patch fixes things on windows?

[hmaarrfk]

@conda-forge-admin please rerender

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-webservice.

I just wanted to let you know that I started rerendering the recipe in #208.

[mkitti]

My understanding is that patch appears to have been incorporated upstream. It is unclear if we need to modify the configuration in response.

[dwr-psandhu]

Is there a way to me to test the newly built package ? It has not been pushed to the channel yet and I don't know if there is a way to get hold of the built artifact.

[hmaarrfk]

your example is with h5py. it is dependent on the patch version of hdf5, so you will have to wait for h5py to be rebuilt.

[dwr-psandhu]

Ok I see it now. Pretty narrow version limiting. However I also see you as the maintainer 😊

[hmaarrfk]

one of them..... mkitti actually understand the ins and outs of the library....

[mkitti]

I will test this more thoroughly tomorrow when I have better access to a Windows computer.

[dwr-psandhu]

I was able to test with installing hdf5 version 1.14.2 using conda-forge and h5py 3.9.0 installed via pip. That combo fails ☹

ValueError: Compression filter "szip" is unavailable

[mkitti]

This program returns that the filter is present in hdf5-feedstock.

#include "hdf5.h"
#include <stdio.h>

int main() {
        H5open();
        printf("H5Z DEFLATE filter: %d\n", H5Zfilter_avail(1));
        printf("H5Z SZIP filter: %d\n", H5Zfilter_avail(4));
        return 0;
}

h5py via pip is probably installing it's own wheel.

The corresponding h5py rebuild seems to be failing due to Cython 3 issues.

conda-forge/h5py-feedstock#130

I recommend closing this issue because the hdf5-feedstock is now fixed at 1.14.2. The issue is now with the h5py-feedstock.

[hmaarrfk]

I just "worked around" the cython issues in h5py. Please try updating the conda stuff and report back. I don't have windows handy.

[dwr-psandhu]

Confirmed! it works on windows. Awesome effort guys! Thanks (★‿★)