Clean up Arrow extension hell, implement RERUN:component_name
#3360
Assignees
Labels
Comments
teh-cmc
added
🏹 arrow
This was referenced Oct 9, 2023
5 tasks
teh-cmc
added a commit
that referenced
this issue
May 31, 2024
A `TransportChunk` is a `Chunk` that is ready for transport and/or storage. It is very cheap to go from `Chunk` to a `TransportChunk` and vice-versa. A `TransportChunk` maps 1:1 to a native Arrow `RecordBatch`. It has a stable ABI, and can be cheaply send across process boundaries. `arrow2` has no `RecordBatch` type; we will get one once we migrate to `arrow-rs`. A `TransportChunk` is self-describing: it contains all the data _and_ metadata needed to index it into storage. We rely heavily on chunk-level and field-level metadata to communicate Rerun-specific semantics over the wire, e.g. whether some columns are already properly sorted. The Arrow metadata system is fairly limited -- it's all untyped strings --, but for now that seems good enough. It will be trivial to switch to something else later, if need be. - Fixes #1760 - Fixes #1692 - Fixes #3360 - Fixes #1696 --- Part of a PR series to implement our new chunk-based data model on the client-side (SDKs): - #6437 - #6438 - #6439 - #6440 - #6441
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'll write something down at some point, but for the moment, refer to this thread.
The text was updated successfully, but these errors were encountered: