Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/src/main/sphinx/admin/properties-general.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
- **Type:** {ref}`prop-type-string`
- **Allowed values:** `AUTOMATIC`, `PARTITIONED`, `BROADCAST`
- **Default value:** `AUTOMATIC`
- **Session property:** `join_distribution_type`

The type of distributed join to use. When set to `PARTITIONED`, Trino
uses hash distributed joins. When set to `BROADCAST`, it broadcasts the
Expand All @@ -18,20 +19,19 @@ only need to fit in distributed memory across all nodes. When set to `AUTOMATIC`
Trino makes a cost based decision as to which distribution type is optimal.
It considers switching the left and right inputs to the join. In `AUTOMATIC`
mode, Trino defaults to hash distributed joins if no cost could be computed, such as if
the tables do not have statistics. This can be specified on a per-query basis using
the `join_distribution_type` session property.
the tables do not have statistics.

## `redistribute-writes`

- **Type:** {ref}`prop-type-boolean`
- **Default value:** `true`
- **Session property:** `redistribute_writes`

This property enables redistribution of data before writing. This can
eliminate the performance impact of data skew when writing by hashing it
across nodes in the cluster. It can be disabled, when it is known that the
output data set is not skewed, in order to avoid the overhead of hashing and
redistributing all the data across the network. This can be specified
on a per-query basis using the `redistribute_writes` session property.
redistributing all the data across the network.

## `protocol.v1.alternate-header-name`

Expand Down
28 changes: 14 additions & 14 deletions docs/src/main/sphinx/admin/properties-optimizer.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,21 +4,21 @@

- **Type:** {ref}`prop-type-boolean`
- **Default value:** `false`
- **Session property:** `dictionary_aggregation`

Enables optimization for aggregations on dictionaries. This can also be specified
on a per-query basis using the `dictionary_aggregation` session property.
Enables optimization for aggregations on dictionaries.

## `optimizer.optimize-hash-generation`

- **Type:** {ref}`prop-type-boolean`
- **Default value:** `true`
- **Session property:** `optimize_hash_generation

Compute hash codes for distribution, joins, and aggregations early during execution,
allowing result to be shared between operations later in the query. This can reduce
CPU usage by avoiding computing the same hash multiple times, but at the cost of
additional network transfer for the hashes. In most cases it decreases overall
query processing time. This can also be specified on a per-query basis using the
`optimize_hash_generation` session property.
query processing time.

It is often helpful to disable this property, when using {doc}`/sql/explain` in order
to make the query plan easier to read.
Expand All @@ -44,6 +44,7 @@ create them.
- **Type:** {ref}`prop-type-string`
- **Allowed values:** `AUTOMATIC`, `ALWAYS`, `NONE`
- **Default value:** `AUTOMATIC`
- **Session property:** `mark_distinct_strategy`

The mark distinct strategy to use for distinct aggregations. `NONE` does not use
`MarkDistinct` operator. `ALWAYS` uses `MarkDistinct` for multiple distinct
Expand All @@ -55,13 +56,13 @@ aggregation implementation cannot utilize CPU efficiently.
`optimizer.use-mark-distinct`. If `optimizer.mark-distinct-strategy` is not
set, but `optimizer.use-mark-distinct` is then `optimizer.use-mark-distinct`
is mapped to `optimizer.mark-distinct-strategy` with value `true` mapped to
`AUTOMATIC` and value `false` mapped to `NONE`.The strategy can be specified
on a per-query basis using the `mark_distinct_strategy` session property.
`AUTOMATIC` and value `false` mapped to `NONE`.

## `optimizer.push-aggregation-through-outer-join`

- **Type:** {ref}`prop-type-boolean`
- **Default value:** `true`
- **Session property:** `push_aggregation_through_join`

When an aggregation is above an outer join and all columns from the outer side of the join
are in the grouping clause, the aggregation is pushed below the outer join. This optimization
Expand All @@ -75,36 +76,35 @@ SELECT * FROM item i
WHERE i.i_category = j.i_category);
```

Enabling this optimization can substantially speed up queries by reducing
the amount of data that needs to be processed by the join. However, it may slow down some
queries that have very selective joins. This can also be specified on a per-query basis using
the `push_aggregation_through_join` session property.
Enabling this optimization can substantially speed up queries by reducing the
amount of data that needs to be processed by the join. However, it may slow down
some queries that have very selective joins.

## `optimizer.push-table-write-through-union`

- **Type:** {ref}`prop-type-boolean`
- **Default value:** `true`
- **Session property:** `push_table_write_through_union`

Parallelize writes when using `UNION ALL` in queries that write data. This improves the
speed of writing output tables in `UNION ALL` queries, because these writes do not require
additional synchronization when collecting results. Enabling this optimization can improve
`UNION ALL` speed, when write speed is not yet saturated. However, it may slow down queries
in an already heavily loaded system. This can also be specified on a per-query basis
using the `push_table_write_through_union` session property.
in an already heavily loaded system.

## `optimizer.join-reordering-strategy`

- **Type:** {ref}`prop-type-string`
- **Allowed values:** `AUTOMATIC`, `ELIMINATE_CROSS_JOINS`, `NONE`
- **Default value:** `AUTOMATIC`
- **Session property:** `join_reordering_strategy`

The join reordering strategy to use. `NONE` maintains the order the tables are listed in the
query. `ELIMINATE_CROSS_JOINS` reorders joins to eliminate cross joins, where possible, and
otherwise maintains the original query order. When reordering joins, it also strives to maintain the
original table order as much as possible. `AUTOMATIC` enumerates possible orders, and uses
statistics-based cost estimation to determine the least cost order. If stats are not available, or if
for any reason a cost could not be computed, the `ELIMINATE_CROSS_JOINS` strategy is used. This can
be specified on a per-query basis using the `join_reordering_strategy` session property.
for any reason a cost could not be computed, the `ELIMINATE_CROSS_JOINS` strategy is used.

## `optimizer.max-reordered-joins`

Expand Down
3 changes: 1 addition & 2 deletions docs/src/main/sphinx/admin/properties-spilling.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ These properties control {doc}`spill`.

- **Type:** {ref}`prop-type-boolean`
- **Default value:** `false`
- **Session property:** `spill_enabled`

Try spilling memory to disk to avoid exceeding memory limits for the query.

Expand All @@ -14,8 +15,6 @@ footprint to pass at the cost of slower execution times. Spilling is supported f
aggregations, joins (inner and outer), sorting, and window functions. This property does not
reduce memory usage required for other join types.

This config property can be overridden by the `spill_enabled` session property.

## `spiller-spill-path`

- **Type:** {ref}`prop-type-string`
Expand Down
15 changes: 7 additions & 8 deletions docs/src/main/sphinx/admin/properties-task.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@
- **Type:** {ref}`prop-type-integer`
- **Restrictions:** Must be a power of two
- **Default value:** The number of physical CPUs of the node, with a minimum value of 2 and a maximum of 32
- **Session property:** `task_concurrency`

Default local concurrency for parallel operators, such as joins and aggregations.
This value should be adjusted up or down based on the query concurrency and worker
resource utilization. Lower values are better for clusters that run many queries
concurrently, because the cluster is already utilized by all the running
queries, so adding more concurrency results in slow downs due to context
switching and other overhead. Higher values are better for clusters that only run
one or a few queries at a time. This can also be specified on a per-query basis
using the `task_concurrency` session property.
one or a few queries at a time.

## `task.http-response-threads`

Expand Down Expand Up @@ -102,16 +102,17 @@ the task has remaining splits to process.

## `task.scale-writers.enabled`

- **Description:** {ref}`prop-task-scale-writers`
- **Description:** see details at {ref}`prop-task-scale-writers`

## `task.scale-writers.max-writer-count`

- **Description:** {ref}`prop-task-scale-writers-max-writer-count`
- **Description:** see details at {ref}`prop-task-scale-writers-max-writer-count`

## `task.writer-count`

- **Type:** {ref}`prop-type-integer`
- **Default value:** `1`
- **Session property:** `task_writer_count`

The number of concurrent writer threads per worker per query when
{ref}`preferred partitioning <preferred-write-partitioning>` and
Expand All @@ -126,21 +127,19 @@ utilization. Especially when the engine is inserting into a partitioned table wi
could write to all partitions. This can lead to out of memory error since writing to a partition
allocates a certain amount of memory for buffering.

This can also be specified on a per-query basis using the `task_writer_count` session property.

## `task.partitioned-writer-count`

- **Type:** {ref}`prop-type-integer`
- **Restrictions:** Must be a power of two
- **Default value:** The number of physical CPUs of the node, with a minimum value of 2 and a maximum of 32
- **Session property:** `task_partitioned_writer_count`

The number of concurrent writer threads per worker per query when
{ref}`preferred partitioning <preferred-write-partitioning>` is used. Increasing this value may
increase write speed, especially when a query is not I/O bound and can take advantage of additional
CPU for parallel writes. Some connectors can be bottlenecked on CPU when writing due to compression
or other factors. Setting this too high may cause the cluster to become overloaded due to excessive
resource utilization. This can also be specified on a per-query basis using the
`task_partitioned_writer_count` session property.
resource utilization.

## `task.interrupt-stuck-split-tasks-enabled`

Expand Down
4 changes: 2 additions & 2 deletions docs/src/main/sphinx/admin/properties-write-partitioning.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@

- **Type:** {ref}`prop-type-boolean`
- **Default value:** `true`
- **Session property:** `use_preferred_write_partitioning`

Enable preferred write partitioning. When set to `true`, each partition is
written by a separate writer. For some connectors such as the Hive connector,
only a single new file is written per partition, instead of multiple files.
Partition writer assignments are distributed across worker nodes for parallel
processing. `use-preferred-write-partitioning` can be specified on a per-query
basis using the `use_preferred_write_partitioning` session property.
processing.
35 changes: 13 additions & 22 deletions docs/src/main/sphinx/admin/properties-writer-scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,24 +15,24 @@ the needs of the query.

- **Type:** {ref}`prop-type-boolean`
- **Default value:** `true`
- **Session property:** `scale_writers`

Enable writer scaling by dynamically increasing the number of writer tasks on
the cluster. This can be specified on a per-query basis using the `scale_writers`
session property.
the cluster.

(prop-task-scale-writers)=

## `task.scale-writers.enabled`

- **Type:** {ref}`prop-type-boolean`
- **Default value:** `true`
- **Session property:** `task_scale_writers_enabled`

Enable scaling the number of concurrent writers within a task. The maximum writer
count per task for scaling is `task.scale-writers.max-writer-count`. Additional
writers are added only when the average amount of uncompressed data processed per writer
is above the minimum threshold of `writer-scaling-min-data-processed` and query is bottlenecked on
writing. This can be specified on a per-query basis using the `task_scale_writers_enabled`
session property.
writing.

(prop-task-scale-writers-max-writer-count)=

Expand All @@ -41,31 +41,22 @@ session property.
- **Type:** {ref}`prop-type-integer`
- **Default value:** The number of physical CPUs of the node with a maximum of 32
Comment thread
mosabua marked this conversation as resolved.
Outdated

Maximum number of concurrent writers per task upto which the task can be scaled when
`task.scale-writers.enabled` is set. Increasing this value may improve the
performance of writes when the query is bottlenecked on writing. Setting this too high
may cause the cluster to become overloaded due to excessive resource utilization.
Maximum number of concurrent writers per task up to which the task can be scaled
when `task.scale-writers.enabled` is set. Increasing this value may improve the
performance of writes when the query is bottlenecked on writing. Setting this
too high may cause the cluster to become overloaded due to excessive resource
utilization.

## `writer-min-size`

- **Type:** {ref}`prop-type-data-size`
- **Default value:** `32MB`

The minimum amount of data that must be written by a writer task before
another writer is eligible to be added. Each writer task may have multiple
writers, controlled by `task.writer-count`, thus this value is effectively
divided by the number of writers per task. This can be specified on a
per-query basis using the `writer_min_size` session property.

:::{warning}
This property is deprecated now. Please use `writer-scaling-min-data-processed`.
:::
Deprecated and replaced by {ref}`writer-scaling-min-data-processed`.

(writer-scaling-min-data-processed)=
## `writer-scaling-min-data-processed`

- **Type:** {ref}`prop-type-data-size`
- **Default value:** `100MB`
- **Session property:** `writer_scaling_min_data_processed`

The minimum amount of uncompressed data that must be processed by a writer
before another writer can be added. This can be specified on a
per-query basis using the `writer_scaling_min_data_processed` session property.
before another writer can be added.