Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use google tensorstore to read/write to zarr? #187

Closed
TomNicholas opened this issue May 26, 2023 · 2 comments · Fixed by #483
Closed

Use google tensorstore to read/write to zarr? #187

TomNicholas opened this issue May 26, 2023 · 2 comments · Fixed by #483
Labels

Comments

@TomNicholas
Copy link
Member

I expect you're already aware of this @tomwhite , but I wanted to ask whether or not you thought the google-tensorstore project might be useful in cubed. @rabernat suggested benchmarking its performance against zarr-python + fsspec. Presumably any improvement to the I/O speed would considerably increase the performance of cubed? Stephan recently released an xarray interface to tensorstore but it seems like that's not the right entry point for using it with cubed.

@tomwhite
Copy link
Member

Yes, I saw this and think it could be very interesting for Cubed.

Presumably any improvement to the I/O speed would considerably increase the performance of cubed?

Hopefully, but we'd have to try it out to see.

@tomwhite tomwhite added the zarr label Jun 1, 2023
@tomwhite
Copy link
Member

tomwhite commented Oct 9, 2023

I tried swapping out Zarr for TensorStore here: https://github.com/tomwhite/cubed/tree/tensorstore. All unit test pass, except for test_resume (since TensorStore doesn't have a way of accessing Zarr's nchunks_initialized).

There were a few challenges due to differences in handling structured arrays, which we use for reductions like mean, but it looks like things would work. We probably will want a way to specify the array storage in the spec so users can select the one they want.

I haven't done any performance testing, but the current branch is sufficiently developed to try that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants