Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Zarr Streaming - hand through compressed buckets #6180

Open
leowe opened this issue Apr 28, 2022 · 1 comment
Open

Zarr Streaming - hand through compressed buckets #6180

leowe opened this issue Apr 28, 2022 · 1 comment

Comments

@leowe
Copy link
Contributor

leowe commented Apr 28, 2022

Detailed Description

Follow-up for #6144.

When using zarr remote datasets, it is possible to define a compression scheme in the .zarray file.

Currently, this feature is unused and all requested buckets are decompressed before sending them to the client. We should hand through the compressed files if possible saving time de-compressing and on the transmission. For that it is necessary to write new data request code paths that don't decompress the read buckets.

@normanrz
Copy link
Member

normanrz commented May 2, 2022

This would be a great addition. In a first iteration, I would only apply this optimization to datasets, where the stored chunk size matches the output chunk size. Otherwise we could create too much server load.

@normanrz normanrz added the zarr label May 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants