Drop old frame splitting in NumPy serialization #3868
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jakirkham
force-pushed
the
drop_split_frames_numpy
branch
from
9ac4bc3 to
de43c05
Compare
Member
|
Looks good to me. Thanks for sifting through this part of the codebase @jakirkham Linting error: If memory serves you prefer that others don't push to your branches, so I'll hold off on pushing a flake8 commit. |
This is leftover from when Blosc compression was still being handled with the NumPy serialization code path itself. These days frame splitting already happens as part of serialization more generally before compressing data. Also this cutoff is not relevant for when splitting occurs as that is determined by a limit in the size of buffers Blosc can handle, which `frame_split_size` already knows. So go ahead and drop this special casing in NumPy serialization and rely on these commonly shared code paths that all serialization passes through.
Since the frames are not being split in serialization and other common code paths to all deserialization already make sure frames match their expected length, there should be no need to merge frames in NumPy deserialization. So drop this as well.
jakirkham
force-pushed
the
drop_split_frames_numpy
branch
from
de43c05 to
8914042
Compare
Member
Author
|
Good catch. Thanks! Fixed in the latest commit. Also we shouldn't need the |
To simplify the code a bit and assert the number of frames is exactly 1, unpack `frames` into a single `frame` variable.
Member
|
Thanks @jakirkham for the work and @mrocklin for the review. Merging in |
Member
Author
|
Thanks Ben and Matt! 😄 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is leftover from when Blosc compression was still being handled with the NumPy serialization code path itself ( #777 ) ( #830 ) ( #832 ) ( #998 ). These days frame splitting already happens as part of serialization more generally before compressing data. Also this cutoff is not relevant for when splitting occurs as that is determined by a limit in the size of buffers Blosc can handle, which
frame_split_sizealready knows. So go ahead and drop this special casing in NumPy serialization and rely on these commonly shared code paths that all serialization passes through.