HDF5 1.10 Compatibility w/ netCDF4

[WardF]

With the release of HDF5 1.10, we are faced with a couple issues. First and foremost, netcdf4 files written using the 1.10 library (compiled with or without 1.10 API, it makes no matter) cannot be read by a libnetcdf compiled against HDF5 1.8.x. This has serious implications for the Unidata datastreams, so long as there are nodes using 1.8 trying to read data generated by nodes using 1.10.

The full changes from HDF5 1.8 to 1.10 can be found here, at the time of this writing:

• https://www.hdfgroup.org/HDF5/doc/ADGuide/Changes.html

The API compatibility report may be found here:

• https://www.hdfgroup.org/HDF5/doc/ADGuide/Compatibility_Report/CR_1.10.0.html

49.2% incompatibility!

It appears that there may be command line tools (5hformat_convert) for re-writing the index data so that 1.10 generated files can be read by 1.8, but lit review is required to determine if this is true. There is also a new function that is listed as "yet to be documented", e.g. H5[D/F]format_convert.

This issue is opened to track progress on how we mitigate these changes for our user community.

[lesserwhirls]

From some initial testing, it looks like the ninth byte of the Superblock (the version number of the superblock) has been changed with 1.10:

With version 1.8.15-patch1, the int value is 2.
With version 1.10.0, the int value is 3.

The first 8 bytes have stayed the same.

What does a superblock of 3 mean? Well, so far it's not documented, but at least it got incremented.

https://www.hdfgroup.org/HDF5/doc/H5.format.html#Superblock

[Dave-Allured]

It appears there is a compatibility mode in HDF5 1.10 for writing downward compatible netcdf-4 files. Would someone please confirm my interpretation of the release documentation? Please read under "Obtaining, Testing, and Experimenting with This Software" on this page:

https://www.hdfgroup.org/HDF5/docNewFeatures/

Files will be written in the older format if these two conditions are met:

1. Set write compatibility:

   H5Pset_libver_bounds (file_id, low=H5F_LIBVER_EARLIEST_F, high=H5F_LIBVER_LATEST_F)

2. Avoid using the so-called "new features " in HDF5 1.10. In other words, do not change current HDF5 function sequences for writing Netcdf-4 files.

--Dave

[WardF]

@Dave-Allured That is a good catch; I was hoping there would be something like that available once I'd had time for proper lit review. I'll need to do some testing to determine the exact behavior of H5Pset_libver_bounds, but I believe you are reading it correctly. We will need to add the explicit call to H5Pset_libver_bounds, assuming it works as expected.

• HDF5 Documentation on H5Pset_libver_bounds

[WardF]

I've created a new branch, gh250, and have roughed in a test that will see if netcdf can open a set of files which will have been generated by a netcdf library linked against hdf5 1.10. These currently fail as expected when using libnetcdf+1.8. This will help us in determining if/how/when we've fixed the problem.

[WardF]

I've got a tentative fix in place that anecdotally appears to work. Will have to do further tests but will try to have this out and in a proper release very quickly.

• TODO: Wire test in to autotools, not just cmake.

[WardF]

The base issue is fixed and merged into master. @DennisHeimbigner is working on a patch which inserts metadata into the file (as a special attribute) which records the version of libhdf5 used to write the file. Once that is done, we'll roll out the 4.4.1-rc1 release, and I'll close out this issue. Leaving it open for now as a discussion hub.

[Dave-Allured]

Please close this issue #250. This was resolved by release 4.4.1 on 2016 June 28, "quite a while ago". Heh heh. This does not need to show up on Milestone:Future in 2019 February. ;-)