-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable datafusion.execution.parquet.schema_force_string_view
by default
#11682
Open
Labels
enhancement
New feature or request
Comments
This was referenced Jul 30, 2024
Want to share my numbers here:
|
take |
This was referenced Aug 20, 2024
This was referenced Sep 9, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Part of #11752
Is your feature request related to a problem or challenge?
As part of #10918, @XiangpengHao has threaded the use of
StringView
through parquet, arrow-rs and then into DataFusionWhen the
datafusion.execution.parquet.schema_force_string_view
option is enabled, the DataFusion Parquet reader will read all Utf8 columns asStringView
instead, which results in significantly faster performance (details TBD but we will write it down in #11603 )However, when initially merged #11667 this setting will be off by default
This ticket tracks what it would take to turn the setting on by default
Describe the solution you'd like
Change the default value of
datafusion.execution.parquet.schema_force_string_view
to trueDescribe alternatives you've considered
Basically we should enable the flag by default and then run some benchmarks to ensure performance doesn't change by too much
Additional context
No response
The text was updated successfully, but these errors were encountered: