Merge string-view2 branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default)
#11667
Commits on Jul 19, 2024
-
Add a config to force using string view in benchmark (#11514)
* add a knob to force string view in benchmark * fix sql logic test * update doc * fix ci * fix ci only test * Update benchmarks/src/util/options.rs Co-authored-by: Andrew Lamb <[email protected]> * Update datafusion/common/src/config.rs Co-authored-by: Andrew Lamb <[email protected]> * update tests --------- Co-authored-by: Andrew Lamb <[email protected]>
-
Add String view helper functions (#11517)
* add functions * add tests for hash util
-
Add ArrowBytesViewMap and ArrowBytesViewSet (#11515)
* Update `string-view` branch to arrow-rs main (#10966) * Pin to arrow main * Fix clippy with latest arrow * Uncomment test that needs new arrow-rs to work * Update datafusion-cli Cargo.lock * Update Cargo.lock * tapelo * merge * update cast * consistent dep * fix ci * add more tests * make doc happy * update new implementation * fix bug * avoid unused dep * update dep * update * fix cargo check * update doc * pick up the comments change again --------- Co-authored-by: Andrew Lamb <[email protected]>
Commits on Jul 20, 2024
-
Enable
GroupValueBytesViewfor aggregation with StringView types (#……11519) * add functions * Update `string-view` branch to arrow-rs main (#10966) * Pin to arrow main * Fix clippy with latest arrow * Uncomment test that needs new arrow-rs to work * Update datafusion-cli Cargo.lock * Update Cargo.lock * tapelo * merge * update cast * consistent dep * fix ci * avoid unused dep * update dep * update * fix cargo check * better group value view aggregation * update --------- Co-authored-by: Andrew Lamb <[email protected]>
Commits on Jul 22, 2024
-
Initial support for regex_replace on
StringViewArray(#11556)* initial support for string view regex * update tests
-
Add support for Utf8View for date/temporal codepaths (#11518)
* Add StringView support for date_part and make_date funcs * run cargo update in datafusion-cli * cargo fmt --------- Co-authored-by: Andrew Lamb <[email protected]>
Commits on Jul 25, 2024
-
GC
StringViewArrayinCoalesceBatchesStream(#11587)* gc string view when appropriate * make clippy happy * address comments * make doc happy * update style * Add comments and tests for gc_string_view_batch * better herustic * update test * Update datafusion/physical-plan/src/coalesce_batches.rs Co-authored-by: Andrew Lamb <[email protected]> --------- Co-authored-by: Andrew Lamb <[email protected]>
Commits on Jul 26, 2024
-
[Bug] fix bug in return type inference of
utf8_to_int_type(#11662)* fix bug in return type inference * update doc * add tests --------- Co-authored-by: Andrew Lamb <[email protected]>
-
Commits on Jul 27, 2024
-
Increase ByteViewMap block size to 2MB (#11674)
* better default block size * fix related test
-
Change
--string-viewto only apply to parquet formats (#11663)* use inferenced schema, don't load schema again * move config to parquet-only * update * update * better format * format * update
-
Implement native support StringView for character length (#11676)
* native support for character length * Update datafusion/functions/src/unicode/character_length.rs --------- Co-authored-by: Andrew Lamb <[email protected]>