Skip to content

Document config options to use to enable hybridscan and increase batching in cudf exchange #248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
devavret wants to merge 4 commits into
base: main
Choose a base branch
from
+20 −0
Draft

Document config options to use to enable hybridscan and increase batching in cudf exchange #248

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
73ff878
document config options to use to enable hybridscan and increase batc…
devavret Feb 23, 2026
c935c91
document that the configs don't work yet and need to be hardcoded
devavret Feb 24, 2026
f9f7dd7
correct the location of hybrid scan config. Add documentation for cud…
devavret Feb 25, 2026
d86e175
Update presto/docker/config/template/etc_worker/config_native.properties
devavret Mar 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions presto/docker/config/template/etc_worker/catalog/hive.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,6 @@ parquet.reader.chunk-read-limit=0
# Limit (in bytes) on the amount of memory used for reading and decompressing data or 0 if there is no limit
parquet.reader.pass-read-limit=0
cudf.hive.use-buffered-input=false

# Turn on to use hybrid parquet reader. (Separate IO and compute)
#cudf.hive.use-experimental-reader=true
17 changes: 17 additions & 0 deletions presto/docker/config/template/etc_worker/config_native.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,21 @@ cudf.exchange=false
cudf.exchange.server.port=8083
cudf.memory_resource=async

# Turn on to use intra-node exchange optimization.
# NOTE: In cudf exchange 20260212 branch, this is needed for UCX to use nvlink.
#cudf.intra_node_exchange=true

# Turn on to use rebatching to minimum of 100m rows (currently only before aggregations).
#cudf.batch_size_min_threshold=100000000
#cudf.concat_optimization_enabled=true

# =================================================================================
# NOTE: This setting is not yet propagated to the worker properly because only a fixed set of query configs are supported.
# https://github.com/prestodb/presto/blob/a62672886152c8c6b61cf301d246f217d850e357/presto-native-execution/presto_cpp/main/PrestoToVeloxQueryConfig.cpp#L106-L224
Copy link
Copy Markdown

@GregoryKimball GregoryKimball Feb 24, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How can I tell which configs work and which require hardcoding? Will this diff help? prestodb/presto#27164

# As a result, this needs to be hardcoded right now.

# Turn on to use 100M rows per chunk for cudf partitioned output.
#cudf.partitioned_output_batch_rows=100000000
# =================================================================================

async-data-cache-enabled=false
Loading