-
Notifications
You must be signed in to change notification settings - Fork 373
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Upgrade arrow2/convert and use native buffers for the tensor u8 types #1375
Conversation
Converted to draft until we have an |
902bb08
to
92b759c
Compare
Sadly, this does not help with the memory duplication of tensors. We see that the The only real solution to this is: |
c41dba2
to
755b81a
Compare
@jleibs do we have the necessary arrow2convert version by now? |
No; there hasn't been a new |
755b81a
to
90f51b6
Compare
…#1375) * Use buffers for the tensor u8 types
Depends on upstream: DataEngineeringLabs/arrow2-convert#103
Using Buffer types here should keep us from making a copy on deserialize.
TODO:
Checklist
CHANGELOG.md
(if this is a big enough change to warrant it)