test_roundtrip failing builds

[abkfenris]

xpublish/tests/test_zarr_compat.py

Lines 140 to 156 in c4134db

	def test_roundtrip(start, end, freq, nlats, nlons, var_const, calendar, use_cftime):
	ds = create_dataset(
	start=start,
	end=end,
	freq=freq,
	nlats=nlats,
	nlons=nlons,
	var_const=var_const,
	use_cftime=use_cftime,
	calendar=calendar,
	)
	ds = ds.chunk(ds.dims)
	mapper = TestMapper(SingleDatasetRest(ds).app)
	actual = xr.open_zarr(mapper, consolidated=True)
	xr.testing.assert_identical(actual, ds)

Is causing builds with unrelated changes to fail recently.

It looks like the issue is due to the time encoding.

FAILED tests/test_zarr_compat.py::test_roundtrip[2018-01-01-2021-01-01-6H-180-360-True-gregorian-False] - AssertionError: Left and right Dataset objects are not identical

Differing coordinates:
L * time         (time) datetime64[ns] 2018-01-01 2018-01-01 ... 2020-12-31
    bounds: time_bounds
R * time         (time) datetime64[ns] 2018-01-01 ... 2020-12-31T12:00:00
    bounds: time_bounds
Differing data variables:
L   tmin         (time, lat, lon) float32 dask.array<chunksize=(4384, 180, 360), meta=np.ndarray>
R   tmin         (time, lat, lon) float32 dask.array<chunksize=(4384, 180, 360), meta=np.ndarray>
L   time_bounds  (time, d2) datetime64[ns] dask.array<chunksize=(4384, 2), meta=np.ndarray>
R   time_bounds  (time, d2) datetime64[ns] dask.array<chunksize=(4384, 2), meta=np.ndarray>
L   tmax         (time, lat, lon) float32 dask.array<chunksize=(4384, 180, 360), meta=np.ndarray>
R   tmax         (time, lat, lon) float32 dask.array<chunksize=(4384, 180, 360), meta=np.ndarray>

A warning is getting thrown earlier

  /home/runner/work/xpublish/xpublish/xpublish/utils/zarr.py:144: UserWarning: Times can't be serialized faithfully to int64 with requested units 'days since 1980-01-01'. Resolution of 'hours' needed. Serializing times to floating point instead. Set encoding['dtype'] to integer dtype to serialize to int64. Set encoding['dtype'] to floating point dtype to silence this warning.
    encoded_da = encode_zarr_variable(dvar, name=key)

Which is from here:

xpublish/xpublish/utils/zarr.py

Lines 132 to 155 in c4134db

	def create_zmetadata(dataset: xr.Dataset) -> dict:
	"""Helper function to create a consolidated zmetadata dictionary."""
	zmeta = {
	'zarr_consolidated_format': ZARR_CONSOLIDATED_FORMAT,
	'metadata': {},
	}
	zmeta['metadata'][group_meta_key] = {'zarr_format': ZARR_FORMAT}
	zmeta['metadata'][attrs_key] = _extract_dataset_zattrs(dataset)
	for key, dvar in dataset.variables.items():
	da = dataset[key]
	encoded_da = encode_zarr_variable(dvar, name=key)
	encoding = extract_zarr_variable_encoding(dvar)
	zattrs = _extract_dataarray_zattrs(encoded_da)
	zattrs = _extract_dataarray_coords(da, zattrs)
	zmeta['metadata'][f'{key}/{attrs_key}'] = zattrs
	zmeta['metadata'][f'{key}/{array_meta_key}'] = _extract_zarray(
	encoded_da,
	encoding,
	encoded_da.dtype,
	)
	return zmeta

But is really coming from the Xarray Zarr backend.

https://github.com/pydata/xarray/blob/25e6e084aa18c49d92934db298a3efff9c712766/xarray/backends/zarr.py#L288-L318

[abkfenris]

Nothing has changed in encode_zarr_variable in the last few years, so it may be deeper.

[abkfenris]

Digging deeper. conventions.encode_cf_variable calls times.CFDatetimeCoder() which then calls encode_cf_datetime() which has had some recent changes.

https://github.com/pydata/xarray/blob/25e6e084aa18c49d92934db298a3efff9c712766/xarray/conventions.py#L154-L195

https://github.com/pydata/xarray/blob/25e6e084aa18c49d92934db298a3efff9c712766/xarray/coding/times.py#L803-L823

https://github.com/pydata/xarray/blob/25e6e084aa18c49d92934db298a3efff9c712766/xarray/coding/times.py#L670-L747

pydata/xarray#8201 landed in Xarray v2023.09.0 which then caused a new bug that looks similar pydata/xarray#8271 which should be fixed by pydata/xarray#8272

If I anti-pin Xarray to resolve to another version (xarray!=v2023.09.0), tests at least seem to pass locally.

[abkfenris]

If I test locally against upstream (git+https://github.com/pydata/xarray.git in dev-requirements.txt) that also works as it looks like pydata/xarray#8272 does the trick.

For now I'm going to anti-pin Xarray in requirements.

[abkfenris]

Which explains why the dev builds worked.