You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This PR extends Vec<u8> extraction to support u8-compatible buffer-protocol exporters (e.g. memoryview, array('B'), and custom types implementing __getbuffer__) by performing a contiguous copy via PyBuffer<u8>.
When the object exports a buffer that is not compatible with u8 (e.g. array('I')), extraction falls back to the existing sequence semantics, preserving current behavior.
Why
Vec<u8> is a common "byte payload" type. Today:
bytes / bytearray already have a specialized path via u8::sequence_extractor.
Other buffer exporters can still hit element-wise sequence extraction, and non-sequence buffer exporters cannot be extracted into Vec<u8> at all.
This change treats the buffer protocol as a first-class source for Vec<u8> when it is semantically valid (u8-compatible).
Implementation notes
Extends the u8::sequence_extractor specialization to recognize buffer exporters.
Makes the internal FromPyObjectSequence::to_vec fallible (PyResult<Vec<_>>) so buffer copies can propagate errors cleanly.
Uses the existing PyBuffer<u8> implementation to perform the copy.
Compatibility
No change to str -> Vec<_> rejection.
bytes / bytearray behavior unchanged.
For incompatible buffers (e.g. array('I')), behavior is preserved via fallback to sequence semantics.
Tests
Added to tests/test_buffer_protocol.rs:
test_extract_vec_u8_from_buffer_exporter (custom buffer exporter, not a sequence)
I am unsure I am comfortable with having this be built-in to the Vec<u8> extraction - see https://alexgaynor.net/2022/oct/23/buffers-on-the-edge/ - there is no guarantee that the buffer object is properly synchronized. By leaving this to user code we let them make the choice.
Hopefully the Python C API will one day support better guarantees for buffer objects.
As per the above, I will close this PR. Thank you for the contribution; we can continue to discuss this in issues, perhaps, to find a long-term solution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR extends
Vec<u8>extraction to support u8-compatible buffer-protocol exporters (e.g.memoryview,array('B'), and custom types implementing__getbuffer__) by performing a contiguous copy viaPyBuffer<u8>.When the object exports a buffer that is not compatible with
u8(e.g.array('I')), extraction falls back to the existing sequence semantics, preserving current behavior.Why
Vec<u8>is a common "byte payload" type. Today:bytes/bytearrayalready have a specialized path viau8::sequence_extractor.Vec<u8>at all.This change treats the buffer protocol as a first-class source for
Vec<u8>when it is semantically valid (u8-compatible).Implementation notes
u8::sequence_extractorspecialization to recognize buffer exporters.FromPyObjectSequence::to_vecfallible (PyResult<Vec<_>>) so buffer copies can propagate errors cleanly.PyBuffer<u8>implementation to perform the copy.Compatibility
str -> Vec<_>rejection.bytes/bytearraybehavior unchanged.array('I')), behavior is preserved via fallback to sequence semantics.Tests
Added to
tests/test_buffer_protocol.rs:test_extract_vec_u8_from_buffer_exporter(custom buffer exporter, not a sequence)test_extract_vec_u8_falls_back_when_buffer_incompatible(array('I')fallback)