Skip to content

Use unsigned bytes to back Buffer #2738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Apr 19, 2025
Merged

Use unsigned bytes to back Buffer #2738

merged 6 commits into from
Apr 19, 2025

Conversation

QuLogic
Copy link
Contributor

@QuLogic QuLogic commented Jan 21, 2025

This makes compressors consistent with v2, and seems more correct than signed bytes.

Fixes #2735

Note that this does not implement accepting both signed and unsigned, as there wasn't a conclusion in #2735 about that.

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/user-guide/*.rst
  • Changes documented in docs/release-notes.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@QuLogic
Copy link
Contributor Author

QuLogic commented Jan 21, 2025

PS, going to ignore tests for now.

@QuLogic QuLogic mentioned this pull request Jan 22, 2025
Closed
@QuLogic
Use unsigned bytes to back Buffer
610689e 
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.

Fixes zarr-developers#2735
@github-actions github-actions bot added the needs release notes Automatically applied to PRs which haven't added release notes label Jan 25, 2025
@dstansby
Copy link
Contributor

I just fixed uploading code coverage for the GPU tests at #2767, so hopefully when I merge in main here it should improve the patch coverage 🤞

@maxrjones maxrjones mentioned this pull request Apr 7, 2025
Open
@maxrjones
Copy link
Member

maxrjones commented Apr 11, 2025 •
edited
Loading

Hi @QuLogic and @dstansby, thanks for working on this! Could I help at all with resolving the codecov issue? I'd be super excited to see this completed to enable further trying out imagecodecs with Zarr V3 (e.g., maxrjones/virtual-tiff#4).

@jakirkham
Copy link
Member

(clicking update branch in case that fixes it)

@jakirkham
Copy link
Member

Closing and reopening to hopefully fix RTD (it has a git checkout error)

@jakirkham jakirkham closed this Apr 11, 2025
@jakirkham jakirkham reopened this Apr 11, 2025
@jakirkham
Copy link
Member

Ok better RTD checked out the code 😌

@jakirkham
Copy link
Member

Looks like Codecov got stuck somewhere. Let's try another close/reopen

@jakirkham jakirkham closed this Apr 11, 2025
@jakirkham jakirkham reopened this Apr 11, 2025
@maxrjones
Copy link
Member

I don't think that the codecov/patch target will be hit unless the overall coverage actually increases, rather than remains constant. The issue is that two of the lines changed (L87 and L110) are not tested in main either - https://app.codecov.io/gh/zarr-developers/zarr-python/blob/main/src%2Fzarr%2Fcore%2Fbuffer%2Fgpu.py, so the PR isn't decreasing code coverage but still will not hit the 80% target for code coverage for patches (i.e., all new/modified lines of code). Do you prefer to ignore the codecov/patch target or to include unit tests for adding and creating zero length buffers?

@jakirkham
Copy link
Member

If someone can add some tests, that would be welcome

What was concerning to me was one Codecov job says "failing after 1s". Though this may be a non-issue and just the report percentage (as you suggested)

This was referenced Apr 11, 2025
Open
Open
@maxrjones
Copy link
Member

If someone can add some tests, that would be welcome

I added test coverage for the GPU buffer in #2978. However, the test for the GPU buffer with the sharding codec is failing. Debugging that failure would require a cuda-compatable GPU but I don't have one easily accessible. I am wondering if you'd consider merging this PR prior to addressing the failures in #2978 for a few reasons:

  1. The impact on the CPU buffer is fully tested and much more commonly used
  2. The sharding codec failure with the GPU buffer is independent of this PR, such that these changes would not introduce new bugs (Add more unit tests for GPU buffer #2978)
  3. Using unsigned bytes in the arrays in the buffer class would unblock imagecodecs usage for both V2 arrays (Buffer uses signed bytes with v2 compressors #2735) and V3 arrays (e.g., https://github.com/maxrjones/virtual-tiff/blob/main/src/virtual_tiff/imagecodecs.py, which currently requires installing from this PR)

Thanks for your consideration, and @d-v-b for the helpful discussion and recommendation to post this suggestion at the Zarr-Python dev call today!

@d-v-b
Copy link
Contributor

d-v-b commented Apr 18, 2025

thanks for pushing this along @maxrjones. Based on the changes in this PR, I'm happy merging this regardless of what codecov says at the moment.

d-v-b
d-v-b approved these changes Apr 19, 2025
View reviewed changes
Copy link
Contributor

@d-v-b d-v-b left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to repeat what I said in the linked issue, the particular numpy dtype we use in our buffer implementation should be considered an implementation detail by consumers. But for the same reason we can alter that dtype as needed, e.g. to make life easier for imagecodecs users, which is what this PR does. I'm going to approve and merge this despite the low test coverage because adding those tests depends on getting the gpu buffer to work with sharding, which is IMO out of scope for this PR.

@d-v-b d-v-b merged commit bb55f0c into zarr-developers:main Apr 19, 2025
29 of 30 checks passed
d-v-b added a commit to d-v-b/zarr-python that referenced this pull request Apr 20, 2025
@QuLogic @d-v-b
@dstansby @jakirkham
Use unsigned bytes to back Buffer (zarr-developers#2738)
99231c6 
This makes compressors consistent with v2, and buffers consistents with
`bytes` types.

Fixes zarr-developers#2735

Co-authored-by: Davis Bennett <[email protected]>
Co-authored-by: David Stansby <[email protected]>
Co-authored-by: jakirkham <[email protected]>
tomwhite added a commit to tomwhite/zarr-python that referenced this pull request Apr 21, 2025
@tomwhite
No need to take unsigned bytes view following zarr-developers#2738
bf3c713
@TomAugspurger TomAugspurger mentioned this pull request Apr 22, 2025
Merged
rapids-bot bot pushed a commit to rapidsai/kvikio that referenced this pull request Apr 22, 2025
@TomAugspurger
zarr-python 3.0.7 compatibility (#699)
2514648 
This adds compatibility for zarr-python 3.0.7, which changed the dtype for representing bytes in zarr-developers/zarr-python#2738.

Authors:
  - Tom Augspurger (https://github.com/TomAugspurger)

Approvers:
  - Tianyu Liu (https://github.com/kingcrimsontianyu)
  - Benjamin Zaitlen (https://github.com/quasiben)

URL: #699
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@d-v-b d-v-b d-v-b approved these changes

Assignees
No one assigned
Labels
needs release notes Automatically applied to PRs which haven't added release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Buffer uses signed bytes with v2 compressors
5 participants
@QuLogic @dstansby @maxrjones @jakirkham @d-v-b