Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip typing data chunk arrays if no rechunking is needed #7370

Merged
merged 6 commits into from
Oct 16, 2023

Conversation

fm3
Copy link
Member

@fm3 fm3 commented Oct 9, 2023

In case the MultiArray chunks on disk are the same shape as the requested buckets, no rechunking is needed. There was already a performance shortcut for this to avoid the partial copy itself, but now we also skip typing the array at all (and converting it back to bytes).

todo

  • memory leak? (cache weight bug? or reference leaked?) → turned out the metadata shape was wrong, leading to the calculated chunk cache weight being 0, leading to the cache to grow without limits

URL of deployed dev instance (used for testing):

Steps to test:

  • Load zarr dataset with nice 32³ buckets, should still look the same and be faster
  • load any other zarr dataset, should still look the same

  • Updated changelog
  • Needs datastore update after deployment

@fm3 fm3 self-assigned this Oct 9, 2023
@fm3 fm3 changed the title WIP: skip typing the array if no partial copying is needed Skip typing data chunk arrays if no rechunking is needed Oct 9, 2023
@fm3 fm3 marked this pull request as ready for review October 9, 2023 13:52
@fm3 fm3 requested a review from normanrz October 9, 2023 13:53
@normanrz normanrz requested a review from frcroth October 10, 2023 09:53
@fm3 fm3 enabled auto-merge (squash) October 16, 2023 09:02
@fm3 fm3 merged commit bdd6a47 into master Oct 16, 2023
2 checks passed
@fm3 fm3 deleted the shortcut-skip-typing branch October 16, 2023 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants