-
Notifications
You must be signed in to change notification settings - Fork 786
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ObjectStore] Non-Contiguous Write Payloads #5514
Labels
enhancement
Any new improvement worthy of a entry in the changelog
object-store
Object Store Interface
Comments
tustvold
added
enhancement
Any new improvement worthy of a entry in the changelog
object-store
Object Store Interface
labels
Mar 14, 2024
tustvold
added a commit
to tustvold/arrow-rs
that referenced
this issue
Mar 21, 2024
tustvold
added a commit
to tustvold/arrow-rs
that referenced
this issue
Apr 4, 2024
tustvold
added a commit
to tustvold/arrow-rs
that referenced
this issue
Apr 4, 2024
tustvold
added a commit
to tustvold/arrow-rs
that referenced
this issue
Apr 9, 2024
tustvold
added a commit
to tustvold/arrow-rs
that referenced
this issue
Apr 9, 2024
tustvold
added a commit
to tustvold/arrow-rs
that referenced
this issue
Apr 9, 2024
alamb
changed the title
Non-Contiguous Write Payloads
[ObjectStore] Non-Contiguous Write Payloads
Apr 11, 2024
tustvold
added a commit
that referenced
this issue
Apr 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
enhancement
Any new improvement worthy of a entry in the changelog
object-store
Object Store Interface
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently
ObjectStore::put
accepts aBytes
object, which is a contiguous memory region. This is simple and efficient where payload sizes are known ahead of time, and buffers can therefore be sized appropriately up-front, however, it may force additional copies in situations where this is not known.Describe the solution you'd like
I am not sure, the HTTP-based stores are fairly coupled to HTTP requests containing a
Bytes
body, and whilstreqwest
does have the ability to wrap a byte stream, this would break retries, AWS content signing and possibly some other things.Describe alternatives you've considered
With #5500 it becomes possible to do chunked writes with no additional copies, this potentially reduces the need for this functionality as users can just allocate data in 5 MiB chunks.
Additional context
#5500 adds a put_part API with a similar requirement
The text was updated successfully, but these errors were encountered: