Implement preferred_chunks for netcdf 4 backends

[mraspaud]

According to the open_dataset documentation, using chunks="auto" or chunks={} should yield datasets with variables chunked depending on the preferred chunks of the backend. However neither the netcdf4 nor the h5netcdf backend seem to implement the preferred_chunks encoding attribute needed for this to work.

This PR adds this attribute to the encoding upon data reading. This results in chunks="auto" in open_dataset returning variables with chunk sizes multiples of the chunks in the nc file, and for chunks={}, returning the variables with then exact nc chunk sizes.

• Closes If a NetCDF file is chunked on disk, open it with compatible dask chunks #1440
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

[mraspaud]

Hope this is up to the standards! Please tell me if there is anything I can do to improve this PR.

[djhoese]

Is the resulting chunk size a multiple of the on-disk file chunk size or is it exactly the file chunk size? So if I open a file with a file chunk size of 256, do I get a dask array with chunk size 256 or some larger chunk size that is a multiple of 256. By itself, 256 would perform worse than one large chunk for a lot of arrays sizes just from the overhead.

[mraspaud]

@djhoese if chunks={}, the same chunks size as the on-disk file will be used. if chunks="auto", a multiple of the on-disk chunk size will be used to accommodate the dask chunk size preferences. This is in line with what is said in the open_dataset docstring as I understand it.
For our case, using chunks="auto" with this patch resulted in a significant performance boost in processing the data as compared to the main branch.

[jhamman]

Thanks for taking this on @mraspaud. This has been something folks have wanted for some time now!

[dcherian]

Found 5 errors in 1 file (checked 142 source files)
xarray/tests/test_backends.py:1568: error: Need type annotation for "tmp_file"  [var-annotated]
xarray/tests/test_backends.py:1577: error: Argument 1 to "contextmanager" has incompatible type "Callable[[NetCDF4Base, Any, Any], None]"; expected "Callable[[NetCDF4Base, Any, Any], Iterator[<nothing>]]"  [arg-type]
xarray/tests/test_backends.py:1578: error: The return type of a generator function should be "Generator" or one of its supertypes  [misc]
xarray/tests/test_backends.py:1599: error: Need type annotation for "tmp_file"  [var-annotated]

@Illviljan @headtr1ck help!

[headtr1ck]

Suggested change

	def create_chunked_file(self, array_shape, chunk_sizes) -> None:
	def create_chunked_file(self, array_shape: tuple[int, int, int], chunk_sizes: tuple[int, int, int]) -> Generator[str, None, None]:

I haven't checked it though, but it should bring you in the right direction.

[mraspaud]

I attempted to fix the type annotation problem, please tell me if there is more.

[mraspaud]

@dcherian @jhamman anything more I can do on this?

[dcherian]

LGTM! Thanks for your patience here @mraspaud

Can you add a note to whats-new to advertise this great improvement please?

[mraspaud]

What's new now updated

[dcherian]

The mypy failures are. related to these changes I think:

xarray/tests/test_backends.py:1559: error: "str" has no attribute "data"  [attr-defined]
xarray/tests/test_backends.py:1559: error: Invalid index type "str" for "str"; expected type "SupportsIndex | slice"  [index]
xarray/tests/test_backends.py:1576: error: "str" has no attribute "data"  [attr-defined]
xarray/tests/test_backends.py:1576: error: Invalid index type "str" for "str"; expected type "SupportsIndex | slice"  [index]
xarray/tests/test_backends.py:1610: error: "str" has no attribute "encoding"  [attr-defined]
xarray/tests/test_backends.py:1610: error: Invalid index type "str" for "str"; expected type "SupportsIndex | slice"  [index]

@Illviljan can you take a look please?

[Illviljan]

Thanks, @mraspaud !

[mraspaud]

My pleasure! thanks for merging!