Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

arrow2_convert primitive (de)serialization performance issues #1712

Closed
Tracked by #1899
teh-cmc opened this issue Mar 27, 2023 · 3 comments
Closed
Tracked by #1899

arrow2_convert primitive (de)serialization performance issues #1712

teh-cmc opened this issue Mar 27, 2023 · 3 comments
Labels
🏹 arrow concerning arrow 🪳 bug Something isn't working 🚀 performance Optimization, memory use, etc

Comments

@teh-cmc
Copy link
Member

teh-cmc commented Mar 27, 2023

Serializing and (maybe?) deserializing primitive types through arrow2-convert's APIs is much slower than doing so using raw arrow2 arrays directly.

See #1742 for detailed benchmarks.

@teh-cmc teh-cmc added 🪳 bug Something isn't working 🏹 arrow concerning arrow 🚀 performance Optimization, memory use, etc labels Mar 27, 2023
@teh-cmc teh-cmc changed the title Creating primitive component arrays through high-level APIs is extremely slow arrow2_convert primitive (de)serialization performance issues Mar 31, 2023
@teh-cmc
Copy link
Member Author

teh-cmc commented Apr 1, 2023

  • Thankfully only really impacts serialization, not deserialization
  • Need to introduce specialized fast paths for serializing control & component columns that are backed by primitive types, which shouldn't be much of an issue

@teh-cmc
Copy link
Member Author

teh-cmc commented Apr 4, 2023

@teh-cmc
Copy link
Member Author

teh-cmc commented Oct 9, 2023

Closing: we're moving away from arrow2-convert.

@teh-cmc teh-cmc closed this as not planned Won't fix, can't repro, duplicate, stale Oct 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🏹 arrow concerning arrow 🪳 bug Something isn't working 🚀 performance Optimization, memory use, etc
Projects
None yet
Development

No branches or pull requests

1 participant