You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
dataslice.update_slice checks whether the data store is consolidated by calling zarr.open_consolidated on it, and catching the resulting exception if it isn't consolidated. It then uses xarray.open_zarr to open the data store as an xarray.Dataset. This means that, if the store is consolidated, the consolidated metadata object is read twice, which is inefficient. We should try to improve on this.
Measure performance with current implementation, using a remote S3 data store.
Implement a more efficient method of checking for consolidation.
Measure performance with the new implementation to confirm and quantify the improvement.
The text was updated successfully, but these errors were encountered:
On further investigation, I think there's a simple solution: xarray.open_zarr (at least as of version 0.19.0) also accepts an explicit consolidation parameter and throws an exception when trying to open unconsolidated data with this parameter set to True. So we can attempt a consolidated xarray.open_zarr and either use the dataset directly or catch the exception and fall back to unconsolidated, noting the consolidation state for later reference.
dataslice.update_slice
checks whether the data store is consolidated by callingzarr.open_consolidated
on it, and catching the resulting exception if it isn't consolidated. It then usesxarray.open_zarr
to open the data store as anxarray.Dataset
. This means that, if the store is consolidated, the consolidated metadata object is read twice, which is inefficient. We should try to improve on this.The text was updated successfully, but these errors were encountered: