ARROW-10163: [Rust] [DataFusion] Add DictionaryArray coercion support #8463
Conversation
There was a problem hiding this comment.
this is just a refactor, it is not meant to change the semantics
There was a problem hiding this comment.
I updated the parameter names here to make this macro more consistent with its documentation
rust/datafusion/tests/sql.rs
Outdated
There was a problem hiding this comment.
This test demonstrates DictionaryArrays being used in DataFusion
alamb
force-pushed
the
alamb/ARROW-10159-dictionary-array-coercion-take-3
branch
from
8630722 to
434bd01
Compare
jorgecarleitao
left a comment
jorgecarleitao
left a comment
There was a problem hiding this comment.
I went through this and LGTM. Thanks a lot, @alamb . Really sweet and easy to follow.
The comments on the tests helped a lot!
alamb
force-pushed
the
alamb/ARROW-10159-dictionary-array-coercion-take-3
branch
from
434bd01 to
cd319b0
Compare
Thanks @nevi-me -- I responded to @jorgecarleitao and I have rebased this PR against master. I think it is good to go, but I will wait for @jorgecarleitao 's response |
This PR adds: 1. Basic `DictionaryArray` coercion (not cast) support in DataFusion 2. A test in `sql.rs` demonstrating basic operations using DataFusion on `DictionaryArray` arrays Note that the performance operating on `DictionaryArrays` is likely to leave a lot to be desired -- specifically almost any operation will cause the `DictionaryArray` to get unpacked to a normal array reducing most/all of any performance gains. I plan to add additional performance improvements over time -- but I felt getting queries to run was the first important step. Closes #8463 from alamb/alamb/ARROW-10159-dictionary-array-coercion-take-3 Authored-by: alamb <[email protected]> Signed-off-by: Neville Dipale <[email protected]>
3 participants
This PR adds:
DictionaryArraycoercion (not cast) support in DataFusionsql.rsdemonstrating basic operations using DataFusion onDictionaryArrayarraysNote that the performance operating on
DictionaryArraysis likely to leave a lot to be desired -- specifically almost any operation will cause theDictionaryArrayto get unpacked to a normal array reducing most/all of any performance gains.I plan to add additional performance improvements over time -- but I felt getting queries to run was the first important step.