Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Asynchronous cache for zarr chunk loading #6165

Merged
merged 4 commits into from
Apr 26, 2022
Merged

Asynchronous cache for zarr chunk loading #6165

merged 4 commits into from
Apr 26, 2022

Conversation

fm3
Copy link
Member

@fm3 fm3 commented Apr 25, 2022

Switches the chunk cache inside zarr array implementation from a synchronous to asynchronous one. This means that concurrent incoming requests that are all cache misses do not each trigger chunk fetching. Only one does and the other requests hook into that.

I use the caching library from akka for this. Side effects include:

  • The new cache is no longer LRU but LFU (frequency based). I do not think this will have much noticeable impact.
  • The new cache has expiry times, which I think is a nice addition made simple by the new akka cache
  • The signatures of many functions had to becomes asynchronous, which makes the code a bit less readable but should be acceptable. It also paves the way for other asynchronous action
  • The exception handling inside ZarrBucketProvider could no longer use the tryo shortcut, instead I use Fox.recover to achieve the same effect.

URL of deployed dev instance (used for testing):

Steps to test:

  • Load remote zarr dataset, ideally one with very different chunk shape from the requested 32² (talk to me about a datasource-properties.json)
  • The first few requests should cause a lot less load, be answered more quickly.
  • wkw datasets and local zarr files should still load

Issues:


@fm3 fm3 self-assigned this Apr 25, 2022
@fm3 fm3 changed the title [WIP] Asynchronous cache for zarr chunk loading Asynchronous cache for zarr chunk loading Apr 26, 2022
@fm3 fm3 marked this pull request as ready for review April 26, 2022 10:01
@fm3 fm3 requested a review from jstriebel April 26, 2022 10:02
Copy link
Contributor

@jstriebel jstriebel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@fm3 fm3 enabled auto-merge (squash) April 26, 2022 11:59
@fm3 fm3 mentioned this pull request Apr 26, 2022
7 tasks
@fm3 fm3 merged commit d42694a into master Apr 26, 2022
@fm3 fm3 deleted the zarr-alru branch April 26, 2022 12:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants