diff --git a/presto-docs/src/main/sphinx/admin.rst b/presto-docs/src/main/sphinx/admin.rst index 18d75296725d9..6893bba2b5daa 100644 --- a/presto-docs/src/main/sphinx/admin.rst +++ b/presto-docs/src/main/sphinx/admin.rst @@ -8,6 +8,7 @@ Administration admin/web-interface admin/tuning admin/properties + admin/properties-session admin/spill admin/exchange-materialization admin/cte-materialization diff --git a/presto-docs/src/main/sphinx/admin/properties-session.rst b/presto-docs/src/main/sphinx/admin/properties-session.rst new file mode 100644 index 0000000000000..f3a7a22ad9f3c --- /dev/null +++ b/presto-docs/src/main/sphinx/admin/properties-session.rst @@ -0,0 +1,285 @@ +========================= +Presto Session Properties +========================= + +This section describes session properties that may be used to tune +Presto or alter its behavior when required. + +The following is not a complete list of all session properties +available in Presto, and does not include any connector-specific +catalog properties. + +For information on catalog properties, see the :doc:`connector documentation `. + +For information on configuration properties, see :doc:`properties`. + + +.. contents:: + :local: + :backlinks: none + :depth: 1 + +General Properties +------------------ + +``join_distribution_type`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``string`` +* **Allowed values:** ``AUTOMATIC``, ``PARTITIONED``, ``BROADCAST`` +* **Default value:** ``AUTOMATIC`` + +The type of distributed join to use. When set to ``PARTITIONED``, presto will +use hash distributed joins. When set to ``BROADCAST``, it will broadcast the +right table to all nodes in the cluster that have data from the left table. +Partitioned joins require redistributing both tables using a hash of the join key. +This can be slower (sometimes substantially) than broadcast joins, but allows much +larger joins. In particular broadcast joins will be faster if the right table is +much smaller than the left. However, broadcast joins require that the tables on the right +side of the join after filtering fit in memory on each node, whereas distributed joins +only need to fit in distributed memory across all nodes. When set to ``AUTOMATIC``, +Presto will make a cost based decision as to which distribution type is optimal. +It will also consider switching the left and right inputs to the join. In ``AUTOMATIC`` +mode, Presto will default to hash distributed joins if no cost could be computed, such as if +the tables do not have statistics. + +``redistribute_writes`` +^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``true`` + +This property enables redistribution of data before writing. This can +eliminate the performance impact of data skew when writing by hashing it +across nodes in the cluster. It can be disabled when it is known that the +output data set is not skewed in order to avoid the overhead of hashing and +redistributing all the data across the network. + +``task_writer_count`` +^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``integer`` +* **Default value:** ``1`` + +Default number of local parallel table writer threads per worker. It is required +to be a power of two for a Java query engine. + +``task_partitioned_writer_count`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``integer`` +* **Default value:** ``task_writer_count`` + +Number of local parallel table writer threads per worker for partitioned writes. If not +set, the number set by ``task_writer_count`` will be used. It is required to be a power +of two for a Java query engine. + +Spilling Properties +------------------- + +``spill_enabled`` +^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``false`` + +Try spilling memory to disk to avoid exceeding memory limits for the query. + +Spilling works by offloading memory to disk. This process can allow a query with a large memory +footprint to pass at the cost of slower execution times. Currently, spilling is supported only for +aggregations and joins (inner and outer), so this property will not reduce memory usage required for +window functions, sorting and other join types. + +Be aware that this is an experimental feature and should be used with care. + +``join_spill_enabled`` +^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``true`` + +When ``spill_enabled`` is ``true``, this determines whether Presto will try spilling memory to disk for joins to +avoid exceeding memory limits for the query. + +``aggregation_spill_enabled`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``true`` + +When ``spill_enabled`` is ``true``, this determines whether Presto will try spilling memory to disk for aggregations to +avoid exceeding memory limits for the query. + +``distinct_aggregation_spill_enabled`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``true`` + +When ``aggregation_spill_enabled`` is ``true``, this determines whether Presto will try spilling memory to disk for distinct +aggregations to avoid exceeding memory limits for the query. + +``order_by_aggregation_spill_enabled`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``true`` + +When ``aggregation_spill_enabled`` is ``true``, this determines whether Presto will try spilling memory to disk for order by +aggregations to avoid exceeding memory limits for the query. + +``window_spill_enabled`` +^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``true`` + +When ``spill_enabled`` is ``true``, this determines whether Presto will try spilling memory to disk for window functions to +avoid exceeding memory limits for the query. + +``order_by_spill_enabled`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``true`` + +When ``spill_enabled`` is ``true``, this determines whether Presto will try spilling memory to disk for order by to +avoid exceeding memory limits for the query. + +``aggregation_operator_unspill_memory_limit`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``data size`` +* **Default value:** ``4 MB`` + +Limit for memory used for unspilling a single aggregation operator instance. + +Task Properties +--------------- + +``task_concurrency`` +^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``integer`` +* **Restrictions:** must be a power of two +* **Default value:** ``16`` + +Default local concurrency for parallel operators such as joins and aggregations. +This value should be adjusted up or down based on the query concurrency and worker +resource utilization. Lower values are better for clusters that run many queries +concurrently because the cluster will already be utilized by all the running +queries, so adding more concurrency will result in slow downs due to context +switching and other overhead. Higher values are better for clusters that only run +one or a few queries at a time. + +``task_writer_count`` +^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``integer`` +* **Restrictions:** must be a power of two +* **Default value:** ``1`` + +The number of concurrent writer threads per worker per query. Increasing this value may +increase write speed, especially when a query is not I/O bound and can take advantage +of additional CPU for parallel writes (some connectors can be bottlenecked on CPU when +writing due to compression or other factors). Setting this too high may cause the cluster +to become overloaded due to excessive resource utilization. + +Optimizer Properties +-------------------- + +``dictionary_aggregation`` +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``false`` + +Enables optimization for aggregations on dictionaries. + +``optimize_hash_generation`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``true`` + +Compute hash codes for distribution, joins, and aggregations early during execution, +allowing result to be shared between operations later in the query. This can reduce +CPU usage by avoiding computing the same hash multiple times, but at the cost of +additional network transfer for the hashes. In most cases it will decrease overall +query processing time. + +It is often helpful to disable this property when using :doc:`/sql/explain` in order +to make the query plan easier to read. + +``push_aggregation_through_join`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``true`` + +When an aggregation is above an outer join and all columns from the outer side of the join +are in the grouping clause, the aggregation is pushed below the outer join. This optimization +is particularly useful for correlated scalar subqueries, which get rewritten to an aggregation +over an outer join. For example:: + + SELECT * FROM item i + WHERE i.i_current_price > ( + SELECT AVG(j.i_current_price) FROM item j + WHERE i.i_category = j.i_category); + +Enabling this optimization can substantially speed up queries by reducing +the amount of data that needs to be processed by the join. However, it may slow down some +queries that have very selective joins. + +``push_table_write_through_union`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``true`` + +Parallelize writes when using ``UNION ALL`` in queries that write data. This improves the +speed of writing output tables in ``UNION ALL`` queries because these writes do not require +additional synchronization when collecting results. Enabling this optimization can improve +``UNION ALL`` speed when write speed is not yet saturated. However, it may slow down queries +in an already heavily loaded system. + + +``join_reordering_strategy`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``string`` +* **Allowed values:** ``AUTOMATIC``, ``ELIMINATE_CROSS_JOINS``, ``NONE`` +* **Default value:** ``AUTOMATIC`` + +The join reordering strategy to use. ``NONE`` maintains the order the tables are listed in the +query. ``ELIMINATE_CROSS_JOINS`` reorders joins to eliminate cross joins where possible and +otherwise maintains the original query order. When reordering joins it also strives to maintain the +original table order as much as possible. ``AUTOMATIC`` enumerates possible orders and uses +statistics-based cost estimation to determine the least cost order. If stats are not available or if +for any reason a cost could not be computed, the ``ELIMINATE_CROSS_JOINS`` strategy is used. + +``confidence_based_broadcast`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``false`` + +Enable broadcasting based on the confidence of the statistics that are being used, by +broadcasting the side of a joinNode which has the highest (``HIGH`` or ``FACT``) confidence statistics. +If both sides have the same confidence statistics, then the original behavior will be followed. + +``treat-low-confidence-zero-estimation-as-unknown`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``false`` + +Enable treating ``LOW`` confidence, zero estimations as ``UNKNOWN`` during joins. + +``retry-query-with-history-based-optimization`` +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* **Type:** ``boolean`` +* **Default value:** ``false`` + +Enable retry for failed queries who can potentially be helped by HBO. \ No newline at end of file diff --git a/presto-docs/src/main/sphinx/admin/properties.rst b/presto-docs/src/main/sphinx/admin/properties.rst index 0b4a7ccc8393a..39c8ac2a2434c 100644 --- a/presto-docs/src/main/sphinx/admin/properties.rst +++ b/presto-docs/src/main/sphinx/admin/properties.rst @@ -1,14 +1,17 @@ -==================== -Properties Reference -==================== +=============================== +Presto Configuration Properties +=============================== -This section describes the most important config properties that -may be used to tune Presto or alter its behavior when required. +This section describes configuration properties that may be used to tune +Presto or alter its behavior when required. -The following pages are not a complete list of all configuration and -session properties available in Presto, and do not include any connector-specific -catalog configuration properties. For more information on catalog configuration -properties, refer to the :doc:`connector documentation `. +The following is not a complete list of all configuration properties +available in Presto, and does not include any connector-specific +catalog configuration properties. + +For information on catalog configuration properties, see the :doc:`connector documentation `. + +For information on session properties, see :doc:`properties-session`. .. contents:: :local: @@ -37,8 +40,7 @@ only need to fit in distributed memory across all nodes. When set to ``AUTOMATIC Presto will make a cost based decision as to which distribution type is optimal. It will also consider switching the left and right inputs to the join. In ``AUTOMATIC`` mode, Presto will default to hash distributed joins if no cost could be computed, such as if -the tables do not have statistics. This can also be specified on a per-query basis using -the ``join_distribution_type`` session property. +the tables do not have statistics. ``redistribute-writes`` ^^^^^^^^^^^^^^^^^^^^^^^ @@ -50,27 +52,7 @@ This property enables redistribution of data before writing. This can eliminate the performance impact of data skew when writing by hashing it across nodes in the cluster. It can be disabled when it is known that the output data set is not skewed in order to avoid the overhead of hashing and -redistributing all the data across the network. This can also be specified -on a per-query basis using the ``redistribute_writes`` session property. - -``task_writer_count`` -^^^^^^^^^^^^^^^^^^^^^ - -* **Type:** ``integer`` -* **Default value:** ``1`` - -Default number of local parallel table writer threads per worker. It is required -to be a power of two for a Java query engine. - -``task_partitioned_writer_count`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -* **Type:** ``integer`` -* **Default value:** ``task_writer_count`` - -Number of local parallel table writer threads per worker for partitioned writes. If not -set, the number set by ``task_writer_count`` will be used. It is required to be a power -of two for a Java query engine. +redistributing all the data across the network. ``eager-plan-validation-enabled`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -184,8 +166,6 @@ window functions, sorting and other join types. Be aware that this is an experimental feature and should be used with care. -This config property can be overridden by the ``spill_enabled`` session property. - ``experimental.join-spill-enabled`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -195,8 +175,6 @@ This config property can be overridden by the ``spill_enabled`` session property When ``spill_enabled`` is ``true``, this determines whether Presto will try spilling memory to disk for joins to avoid exceeding memory limits for the query. -This config property can be overridden by the ``join_spill_enabled`` session property. - ``experimental.aggregation-spill-enabled`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -206,8 +184,6 @@ This config property can be overridden by the ``join_spill_enabled`` session pro When ``spill_enabled`` is ``true``, this determines whether Presto will try spilling memory to disk for aggregations to avoid exceeding memory limits for the query. -This config property can be overridden by the ``aggregation_spill_enabled`` session property. - ``experimental.distinct-aggregation-spill-enabled`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -217,8 +193,6 @@ This config property can be overridden by the ``aggregation_spill_enabled`` sess When ``aggregation_spill_enabled`` is ``true``, this determines whether Presto will try spilling memory to disk for distinct aggregations to avoid exceeding memory limits for the query. -This config property can be overridden by the ``distinct_aggregation_spill_enabled`` session property. - ``experimental.order-by-aggregation-spill-enabled`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -228,8 +202,6 @@ This config property can be overridden by the ``distinct_aggregation_spill_enabl When ``aggregation_spill_enabled`` is ``true``, this determines whether Presto will try spilling memory to disk for order by aggregations to avoid exceeding memory limits for the query. -This config property can be overridden by the ``order_by_aggregation_spill_enabled`` session property. - ``experimental.window-spill-enabled`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -239,8 +211,6 @@ This config property can be overridden by the ``order_by_aggregation_spill_enabl When ``spill_enabled`` is ``true``, this determines whether Presto will try spilling memory to disk for window functions to avoid exceeding memory limits for the query. -This config property can be overridden by the ``window_spill_enabled`` session property. - ``experimental.order-by-spill-enabled`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -250,8 +220,6 @@ This config property can be overridden by the ``window_spill_enabled`` session p When ``spill_enabled`` is ``true``, this determines whether Presto will try spilling memory to disk for order by to avoid exceeding memory limits for the query. -This config property can be overridden by the ``order_by_spill_enabled`` session property. - ``experimental.spiller.task-spilling-strategy`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * **Type:** ``string`` @@ -375,7 +343,6 @@ Max spill space to be used by a single query on a single node. * **Default value:** ``4 MB`` Limit for memory used for unspilling a single aggregation operator instance. -This config property can be overridden by the ``aggregation_operator_unspill_memory_limit`` session property ``experimental.spill-compression-enabled`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -513,8 +480,7 @@ resource utilization. Lower values are better for clusters that run many queries concurrently because the cluster will already be utilized by all the running queries, so adding more concurrency will result in slow downs due to context switching and other overhead. Higher values are better for clusters that only run -one or a few queries at a time. This can also be specified on a per-query basis -using the ``task_concurrency`` session property. +one or a few queries at a time. ``task.http-response-threads`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -605,8 +571,7 @@ The number of concurrent writer threads per worker per query. Increasing this va increase write speed, especially when a query is not I/O bound and can take advantage of additional CPU for parallel writes (some connectors can be bottlenecked on CPU when writing due to compression or other factors). Setting this too high may cause the cluster -to become overloaded due to excessive resource utilization. This can also be specified on -a per-query basis using the ``task_writer_count`` session property. +to become overloaded due to excessive resource utilization. ``task.interrupt-runaway-splits-timeout`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -698,8 +663,7 @@ Optimizer Properties * **Type:** ``boolean`` * **Default value:** ``false`` -Enables optimization for aggregations on dictionaries. This can also be specified -on a per-query basis using the ``dictionary_aggregation`` session property. +Enables optimization for aggregations on dictionaries. ``optimizer.optimize-hash-generation`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -711,8 +675,7 @@ Compute hash codes for distribution, joins, and aggregations early during execut allowing result to be shared between operations later in the query. This can reduce CPU usage by avoiding computing the same hash multiple times, but at the cost of additional network transfer for the hashes. In most cases it will decrease overall -query processing time. This can also be specified on a per-query basis using the -``optimize_hash_generation`` session property. +query processing time. It is often helpful to disable this property when using :doc:`/sql/explain` in order to make the query plan easier to read. @@ -761,8 +724,7 @@ over an outer join. For example:: Enabling this optimization can substantially speed up queries by reducing the amount of data that needs to be processed by the join. However, it may slow down some -queries that have very selective joins. This can also be specified on a per-query basis using -the ``push_aggregation_through_join`` session property. +queries that have very selective joins. ``optimizer.push-table-write-through-union`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -774,8 +736,7 @@ Parallelize writes when using ``UNION ALL`` in queries that write data. This imp speed of writing output tables in ``UNION ALL`` queries because these writes do not require additional synchronization when collecting results. Enabling this optimization can improve ``UNION ALL`` speed when write speed is not yet saturated. However, it may slow down queries -in an already heavily loaded system. This can also be specified on a per-query basis -using the ``push_table_write_through_union`` session property. +in an already heavily loaded system. ``optimizer.join-reordering-strategy`` @@ -790,8 +751,7 @@ query. ``ELIMINATE_CROSS_JOINS`` reorders joins to eliminate cross joins where otherwise maintains the original query order. When reordering joins it also strives to maintain the original table order as much as possible. ``AUTOMATIC`` enumerates possible orders and uses statistics-based cost estimation to determine the least cost order. If stats are not available or if -for any reason a cost could not be computed, the ``ELIMINATE_CROSS_JOINS`` strategy is used. This can -also be specified on a per-query basis using the ``join_reordering_strategy`` session property. +for any reason a cost could not be computed, the ``ELIMINATE_CROSS_JOINS`` strategy is used. ``optimizer.max-reordered-joins`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -873,7 +833,7 @@ a query plan. The optimizer may then use these properties to perform various opt Enable broadcasting based on the confidence of the statistics that are being used, by broadcasting the side of a joinNode which has the highest (``HIGH`` or ``FACT``) confidence statistics. If both sides have the same confidence statistics, then the original behavior will be followed. -This can also be specified on a per-query basis using the ``confidence_based_broadcast`` session property. + ``optimizer.treat-low-confidence-zero-estimation-as-unknown`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -881,8 +841,7 @@ This can also be specified on a per-query basis using the ``confidence_based_bro * **Type:** ``boolean`` * **Default value:** ``false`` -Enable treating ``LOW`` confidence, zero estimations as ``UNKNOWN`` during joins. This can also be specified -on a per-query basis using the ``treat-low-confidence-zero-estimation-as-unknown`` session property. +Enable treating ``LOW`` confidence, zero estimations as ``UNKNOWN`` during joins. ``optimizer.retry-query-with-history-based-optimization`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -890,8 +849,7 @@ on a per-query basis using the ``treat-low-confidence-zero-estimation-as-unknown * **Type:** ``boolean`` * **Default value:** ``false`` -Enable retry for failed queries who can potentially be helped by HBO. This can also be specified -on a per-query basis using the ``retry-query-with-history-based-optimization`` session property. +Enable retry for failed queries who can potentially be helped by HBO. Planner Properties ------------------