diff --git a/docs/configuration/index.md b/docs/configuration/index.md index 4924fb478ff1..a04029ff0bfc 100644 --- a/docs/configuration/index.md +++ b/docs/configuration/index.md @@ -1501,7 +1501,7 @@ Processing properties set on the MiddleManager will be passed through to Peons. |`druid.processing.buffer.sizeBytes`|This specifies a buffer size (less than 2GiB) for the storage of intermediate results. The computation engine in both the Historical and Realtime processes will use a scratch buffer of this size to do all of their intermediate computations off-heap. Larger values allow for more aggregations in a single pass over the data while smaller values can require more passes depending on the query that is being executed. [Human-readable format](human-readable-byte.md) is supported.|auto (max 1 GiB)| |`druid.processing.buffer.poolCacheMaxCount`|Processing buffer pool caches the buffers for later use. This is the maximum count that the cache will grow to. Note that pool can create more buffers than it can cache if necessary.|`Integer.MAX_VALUE`| |`druid.processing.formatString`|Realtime and Historical processes use this format string to name their processing threads.|processing-%s| -|`druid.processing.numMergeBuffers`|The number of direct memory buffers available for merging query results. The buffers are sized by `druid.processing.buffer.sizeBytes`. This property is effectively a concurrency limit for queries that require merging buffers. If you are using any queries that require merge buffers (currently, just groupBy v2) then you should have at least two of these.|`max(2, druid.processing.numThreads / 4)`| +|`druid.processing.numMergeBuffers`|The number of direct memory buffers available for merging query results. The buffers are sized by `druid.processing.buffer.sizeBytes`. This property is effectively a concurrency limit for queries that require merging buffers. If you are using any queries that require merge buffers (currently, just groupBy) then you should have at least two of these.|`max(2, druid.processing.numThreads / 4)`| |`druid.processing.numThreads`|The number of processing threads to have available for parallel processing of segments. Our rule of thumb is `num_cores - 1`, which means that even under heavy load there will still be one core available to do background tasks like talking with ZooKeeper and pulling down segments. If only one core is available, this property defaults to the value `1`.|Number of cores - 1 (or 1)| |`druid.processing.fifo`|Enables the processing queue to treat tasks of equal priority in a FIFO manner.|`true`| |`druid.processing.tmpDir`|Path where temporary files created while processing a query should be stored. If specified, this configuration takes priority over the default `java.io.tmpdir` path.|path represented by `java.io.tmpdir`| @@ -1650,7 +1650,7 @@ Druid uses Jetty to serve HTTP requests. |`druid.processing.buffer.sizeBytes`|This specifies a buffer size (less than 2GiB) for the storage of intermediate results. The computation engine in the Indexer processes will use a scratch buffer of this size to do all of their intermediate computations off-heap. Larger values allow for more aggregations in a single pass over the data while smaller values can require more passes depending on the query that is being executed. [Human-readable format](human-readable-byte.md) is supported.|auto (max 1GiB)| |`druid.processing.buffer.poolCacheMaxCount`|processing buffer pool caches the buffers for later use, this is the maximum count cache will grow to. note that pool can create more buffers than it can cache if necessary.|Integer.MAX_VALUE| |`druid.processing.formatString`|Indexer processes use this format string to name their processing threads.|processing-%s| -|`druid.processing.numMergeBuffers`|The number of direct memory buffers available for merging query results. The buffers are sized by `druid.processing.buffer.sizeBytes`. This property is effectively a concurrency limit for queries that require merging buffers. If you are using any queries that require merge buffers (currently, just groupBy v2) then you should have at least two of these.|`max(2, druid.processing.numThreads / 4)`| +|`druid.processing.numMergeBuffers`|The number of direct memory buffers available for merging query results. The buffers are sized by `druid.processing.buffer.sizeBytes`. This property is effectively a concurrency limit for queries that require merging buffers. If you are using any queries that require merge buffers (currently, just groupBy) then you should have at least two of these.|`max(2, druid.processing.numThreads / 4)`| |`druid.processing.numThreads`|The number of processing threads to have available for parallel processing of segments. Our rule of thumb is `num_cores - 1`, which means that even under heavy load there will still be one core available to do background tasks like talking with ZooKeeper and pulling down segments. If only one core is available, this property defaults to the value `1`.|Number of cores - 1 (or 1)| |`druid.processing.fifo`|If the processing queue should treat tasks of equal priority in a FIFO manner|`true`| |`druid.processing.tmpDir`|Path where temporary files created while processing a query should be stored. If specified, this configuration takes priority over the default `java.io.tmpdir` path.|path represented by `java.io.tmpdir`| @@ -1759,7 +1759,7 @@ Druid uses Jetty to serve HTTP requests. |`druid.processing.buffer.sizeBytes`|This specifies a buffer size (less than 2GiB), for the storage of intermediate results. The computation engine in both the Historical and Realtime processes will use a scratch buffer of this size to do all of their intermediate computations off-heap. Larger values allow for more aggregations in a single pass over the data while smaller values can require more passes depending on the query that is being executed. [Human-readable format](human-readable-byte.md) is supported.|auto (max 1GiB)| |`druid.processing.buffer.poolCacheMaxCount`|processing buffer pool caches the buffers for later use, this is the maximum count cache will grow to. note that pool can create more buffers than it can cache if necessary.|Integer.MAX_VALUE| |`druid.processing.formatString`|Realtime and Historical processes use this format string to name their processing threads.|processing-%s| -|`druid.processing.numMergeBuffers`|The number of direct memory buffers available for merging query results. The buffers are sized by `druid.processing.buffer.sizeBytes`. This property is effectively a concurrency limit for queries that require merging buffers. If you are using any queries that require merge buffers (currently, just groupBy v2) then you should have at least two of these.|`max(2, druid.processing.numThreads / 4)`| +|`druid.processing.numMergeBuffers`|The number of direct memory buffers available for merging query results. The buffers are sized by `druid.processing.buffer.sizeBytes`. This property is effectively a concurrency limit for queries that require merging buffers. If you are using any queries that require merge buffers (currently, just groupBy) then you should have at least two of these.|`max(2, druid.processing.numThreads / 4)`| |`druid.processing.numThreads`|The number of processing threads to have available for parallel processing of segments. Our rule of thumb is `num_cores - 1`, which means that even under heavy load there will still be one core available to do background tasks like talking with ZooKeeper and pulling down segments. If only one core is available, this property defaults to the value `1`.|Number of cores - 1 (or 1)| |`druid.processing.fifo`|If the processing queue should treat tasks of equal priority in a FIFO manner|`true`| |`druid.processing.tmpDir`|Path where temporary files created while processing a query should be stored. If specified, this configuration takes priority over the default `java.io.tmpdir` path.|path represented by `java.io.tmpdir`| @@ -1938,7 +1938,7 @@ The broker uses processing configs for nested groupBy queries. |`druid.processing.buffer.sizeBytes`|This specifies a buffer size (less than 2GiB) for the storage of intermediate results. The computation engine in both the Historical and Realtime processes will use a scratch buffer of this size to do all of their intermediate computations off-heap. Larger values allow for more aggregations in a single pass over the data while smaller values can require more passes depending on the query that is being executed. [Human-readable format](human-readable-byte.md) is supported.|auto (max 1GiB)| |`druid.processing.buffer.poolCacheInitialCount`|initializes the number of buffers allocated on the intermediate results pool. Note that pool can create more buffers if necessary.|`0`| |`druid.processing.buffer.poolCacheMaxCount`|processing buffer pool caches the buffers for later use, this is the maximum count cache will grow to. note that pool can create more buffers than it can cache if necessary.|Integer.MAX_VALUE| -|`druid.processing.numMergeBuffers`|The number of direct memory buffers available for merging query results. The buffers are sized by `druid.processing.buffer.sizeBytes`. This property is effectively a concurrency limit for queries that require merging buffers. If you are using any queries that require merge buffers (currently, just groupBy v2) then you should have at least two of these.|`max(2, druid.processing.numThreads / 4)`| +|`druid.processing.numMergeBuffers`|The number of direct memory buffers available for merging query results. The buffers are sized by `druid.processing.buffer.sizeBytes`. This property is effectively a concurrency limit for queries that require merging buffers. If you are using any queries that require merge buffers (currently, just groupBy) then you should have at least two of these.|`max(2, druid.processing.numThreads / 4)`| |`druid.processing.fifo`|If the processing queue should treat tasks of equal priority in a FIFO manner|`true`| |`druid.processing.tmpDir`|Path where temporary files created while processing a query should be stored. If specified, this configuration takes priority over the default `java.io.tmpdir` path.|path represented by `java.io.tmpdir`| |`druid.processing.merge.useParallelMergePool`|Enable automatic parallel merging for Brokers on a dedicated async ForkJoinPool. If `false`, instead merges will be done serially on the `HTTP` thread pool.|`true`| @@ -2026,8 +2026,8 @@ You can optionally only configure caching to be enabled on the Broker by setting See [cache configuration](#cache-configuration) for how to configure cache settings. :::info - Note: Even if cache is enabled, for [groupBy v2](../querying/groupbyquery.md#strategies) queries, segment level cache do not work on Brokers. - See [Differences between v1 and v2](../querying/groupbyquery.md#differences-between-v1-and-v2) and [Query caching](../querying/caching.md) for more information. + Note: Even if cache is enabled, for [groupBy](../querying/groupbyquery.md) queries, segment level cache does not work on Brokers. + See [query caching](../querying/caching.md) for more information. ::: #### Segment Discovery @@ -2202,8 +2202,6 @@ context). If query does have `maxQueuedBytes` in the context, then that value is This section describes the configurations for groupBy queries. You can set the runtime properties in the `runtime.properties` file on Broker, Historical, and MiddleManager processes. You can set the query context parameters through the [query context](../querying/query-context.md). -#### Configurations for groupBy v2 - Supported runtime properties: |Property|Description|Default| @@ -2224,29 +2222,11 @@ Supported query contexts: ### Advanced configurations -#### Common configurations for all groupBy strategies - Supported runtime properties: |Property|Description|Default| |--------|-----------|-------| -|`druid.query.groupBy.defaultStrategy`|Default groupBy query strategy.|v2| |`druid.query.groupBy.singleThreaded`|Merge results using a single thread.|false| - -Supported query contexts: - -|Key|Description| -|---|-----------| -|`groupByStrategy`|Overrides the value of `druid.query.groupBy.defaultStrategy` for this query.| -|`groupByIsSingleThreaded`|Overrides the value of `druid.query.groupBy.singleThreaded` for this query.| - - -#### GroupBy v2 configurations - -Supported runtime properties: - -|Property|Description|Default| -|--------|-----------|-------| |`druid.query.groupBy.bufferGrouperInitialBuckets`|Initial number of buckets in the off-heap hash table used for grouping results. Set to 0 to use a reasonable default (1024).|0| |`druid.query.groupBy.bufferGrouperMaxLoadFactor`|Maximum load factor of the off-heap hash table used for grouping results. When the load factor exceeds this size, the table will be grown or spilled to disk. Set to 0 to use a reasonable default (0.7).|0| |`druid.query.groupBy.forceHashAggregation`|Force to use hash-based aggregation.|false| @@ -2257,6 +2237,7 @@ Supported query contexts: |Key|Description|Default| |---|-----------|-------| +|`groupByIsSingleThreaded`|Overrides the value of `druid.query.groupBy.singleThreaded` for this query.| |`bufferGrouperInitialBuckets`|Overrides the value of `druid.query.groupBy.bufferGrouperInitialBuckets` for this query.|None| |`bufferGrouperMaxLoadFactor`|Overrides the value of `druid.query.groupBy.bufferGrouperMaxLoadFactor` for this query.|None| |`forceHashAggregation`|Overrides the value of `druid.query.groupBy.forceHashAggregation`|None| @@ -2265,24 +2246,6 @@ Supported query contexts: |`sortByDimsFirst`|Sort the results first by dimension values and then by timestamp.|false| |`forceLimitPushDown`|When all fields in the orderby are part of the grouping key, the broker will push limit application down to the Historical processes. When the sorting order uses fields that are not in the grouping key, applying this optimization can result in approximate results with unknown accuracy, so this optimization is disabled by default in that case. Enabling this context flag turns on limit push down for limit/orderbys that contain non-grouping key columns.|false| - -#### GroupBy v1 configurations - -Supported runtime properties: - -|Property|Description|Default| -|--------|-----------|-------| -|`druid.query.groupBy.maxIntermediateRows`|Maximum number of intermediate rows for the per-segment grouping engine. This is a tuning parameter that does not impose a hard limit; rather, it potentially shifts merging work from the per-segment engine to the overall merging index. Queries that exceed this limit will not fail.|50000| -|`druid.query.groupBy.maxResults`|Maximum number of results. Queries that exceed this limit will fail.|500000| - -Supported query contexts: - -|Key|Description|Default| -|---|-----------|-------| -|`maxIntermediateRows`|Ignored by groupBy v2. Can be used to lower the value of `druid.query.groupBy.maxIntermediateRows` for a groupBy v1 query.|None| -|`maxResults`|Ignored by groupBy v2. Can be used to lower the value of `druid.query.groupBy.maxResults` for a groupBy v1 query.|None| -|`useOffheap`|Ignored by groupBy v2, and no longer supported for groupBy v1. Enabling this option with groupBy v1 will result in an error. For off-heap aggregation, switch to groupBy v2, which always operates off-heap.|false| - #### Expression processing configurations |Key|Description|Default| diff --git a/docs/operations/basic-cluster-tuning.md b/docs/operations/basic-cluster-tuning.md index 9d54afed27d5..538ae33d75f2 100644 --- a/docs/operations/basic-cluster-tuning.md +++ b/docs/operations/basic-cluster-tuning.md @@ -326,13 +326,13 @@ The TopN and GroupBy queries use these buffers to store intermediate computed re ### GroupBy merging buffers -If you plan to issue GroupBy V2 queries, `druid.processing.numMergeBuffers` is an important configuration property. +If you plan to issue GroupBy queries, `druid.processing.numMergeBuffers` is an important configuration property. -GroupBy V2 queries use an additional pool of off-heap buffers for merging query results. These buffers have the same size as the processing buffers described above, set by the `druid.processing.buffer.sizeBytes` property. +GroupBy queries use an additional pool of off-heap buffers for merging query results. These buffers have the same size as the processing buffers described above, set by the `druid.processing.buffer.sizeBytes` property. -Non-nested GroupBy V2 queries require 1 merge buffer per query, while a nested GroupBy V2 query requires 2 merge buffers (regardless of the depth of nesting). +Non-nested GroupBy queries require 1 merge buffer per query, while a nested GroupBy query requires 2 merge buffers (regardless of the depth of nesting). -The number of merge buffers determines the number of GroupBy V2 queries that can be processed concurrently. +The number of merge buffers determines the number of GroupBy queries that can be processed concurrently. diff --git a/docs/querying/caching.md b/docs/querying/caching.md index 62cdb30fa25c..a84e3d25eee4 100644 --- a/docs/querying/caching.md +++ b/docs/querying/caching.md @@ -101,12 +101,11 @@ Caching does not solve all types of query performance issues. For each cache typ **Per-segment caching** doesn't work for the following: - queries containing a sub-query in them. However the output of sub-queries may be cached. See [Query execution](./query-execution.md) for more details on sub-queries execution. - queries with joins do not support any caching on the broker. -- GroupBy v2 queries do not support any caching on broker. +- GroupBy queries do not support segment level caching on broker. - queries with `bySegment` set in the query context are not cached on the broker. **Whole-query caching** doesn't work for the following: - queries that involve an inline datasource or a lookup datasource. -- GroupBy v2 queries. - queries with joins. - queries with a union datasource. diff --git a/docs/querying/groupbyquery.md b/docs/querying/groupbyquery.md index 81da0f0e2323..be5e03d56eed 100644 --- a/docs/querying/groupbyquery.md +++ b/docs/querying/groupbyquery.md @@ -240,49 +240,9 @@ The response for the query above would look something like: ## Implementation details -### Strategies - -GroupBy queries can be executed using two different strategies. The default strategy for a cluster is determined by the -"druid.query.groupBy.defaultStrategy" runtime property on the Broker. This can be overridden using "groupByStrategy" in -the query context. If neither the context field nor the property is set, the "v2" strategy will be used. - -- "v2", the default, is designed to offer better performance and memory management. This strategy generates -per-segment results using a fully off-heap map. Data processes merge the per-segment results using a fully off-heap -concurrent facts map combined with an on-heap string dictionary. This may optionally involve spilling to disk. Data -processes return sorted results to the Broker, which merges result streams using an N-way merge. The broker materializes -the results if necessary (e.g. if the query sorts on columns other than its dimensions). Otherwise, it streams results -back as they are merged. - -- "v1", a legacy engine, generates per-segment results on data processes (Historical, realtime, MiddleManager) using a map which -is partially on-heap (dimension keys and the map itself) and partially off-heap (the aggregated values). Data processes then -merge the per-segment results using Druid's indexing mechanism. This merging is multi-threaded by default, but can -optionally be single-threaded. The Broker merges the final result set using Druid's indexing mechanism again. The broker -merging is always single-threaded. Because the Broker merges results using the indexing mechanism, it must materialize -the full result set before returning any results. On both the data processes and the Broker, the merging index is fully -on-heap by default, but it can optionally store aggregated values off-heap. - -### Differences between v1 and v2 - -Query API and results are compatible between the two engines; however, there are some differences from a cluster -configuration perspective: - -- groupBy v1 controls resource usage using a row-based limit (maxResults) whereas groupBy v2 uses bytes-based limits. -In addition, groupBy v1 merges results on-heap, whereas groupBy v2 merges results off-heap. These factors mean that -memory tuning and resource limits behave differently between v1 and v2. In particular, due to this, some queries -that can complete successfully in one engine may exceed resource limits and fail with the other engine. See the -"Memory tuning and resource limits" section for more details. -- groupBy v1 imposes no limit on the number of concurrently running queries, whereas groupBy v2 controls memory usage -by using a finite-sized merge buffer pool. By default, the number of merge buffers is 1/4 the number of processing -threads. You can adjust this as necessary to balance concurrency and memory usage. -- groupBy v1 supports caching on either the Broker or Historical processes, whereas groupBy v2 only supports caching on -Historical processes. -- groupBy v2 supports both array-based aggregation and hash-based aggregation. The array-based aggregation is used only -when the grouping key is a single indexed string column. In array-based aggregation, the dictionary-encoded value is used -as the index, so the aggregated values in the array can be accessed directly without finding buckets based on hashing. - ### Memory tuning and resource limits -When using groupBy v2, four parameters control resource usage and limits: +When using groupBy, four parameters control resource usage and limits: - `druid.processing.buffer.sizeBytes`: size of the off-heap hash table used for aggregation, per query, in bytes. At most `druid.processing.numMergeBuffers` of these will be created at once, which also serves as an upper limit on the @@ -306,7 +266,7 @@ sorted and flushed to disk. Then, both in-memory structures will be cleared out then go on to exceed `maxOnDiskStorage` will fail with a "Resource limit exceeded" error indicating that they ran out of disk space. -With groupBy v2, cluster operators should make sure that the off-heap hash tables and on-heap merging dictionaries +With groupBy, cluster operators should make sure that the off-heap hash tables and on-heap merging dictionaries will not exceed available memory for the maximum possible concurrent query load (given by `druid.processing.numMergeBuffers`). See the [basic cluster tuning guide](../operations/basic-cluster-tuning.md) for more details about direct memory usage, organized by Druid process type. @@ -315,24 +275,18 @@ Brokers do not need merge buffers for basic groupBy queries. Queries with subque Historicals and ingestion tasks need one merge buffer for each groupBy query, unless [parallel combination](groupbyquery.md#parallel-combine) is enabled, in which case they need two merge buffers per query. -When using groupBy v1, all aggregation is done on-heap, and resource limits are done through the parameter -`druid.query.groupBy.maxResults`. This is a cap on the maximum number of results in a result set. Queries that exceed -this limit will fail with a "Resource limit exceeded" error indicating they exceeded their row limit. Cluster -operators should make sure that the on-heap aggregations will not exceed available JVM heap space for the expected -concurrent query load. - -### Performance tuning for groupBy v2 +### Performance tuning for groupBy #### Limit pushdown optimization -Druid pushes down the `limit` spec in groupBy queries to the segments on Historicals wherever possible to early prune unnecessary intermediate results and minimize the amount of data transferred to Brokers. By default, this technique is applied only when all fields in the `orderBy` spec is a subset of the grouping keys. This is because the `limitPushDown` doesn't guarantee the exact results if the `orderBy` spec includes any fields that are not in the grouping keys. However, you can enable this technique even in such cases if you can sacrifice some accuracy for fast query processing like in topN queries. See `forceLimitPushDown` in [advanced groupBy v2 configurations](#groupby-v2-configurations). +Druid pushes down the `limit` spec in groupBy queries to the segments on Historicals wherever possible to early prune unnecessary intermediate results and minimize the amount of data transferred to Brokers. By default, this technique is applied only when all fields in the `orderBy` spec is a subset of the grouping keys. This is because the `limitPushDown` doesn't guarantee the exact results if the `orderBy` spec includes any fields that are not in the grouping keys. However, you can enable this technique even in such cases if you can sacrifice some accuracy for fast query processing like in topN queries. See `forceLimitPushDown` in [advanced configurations](#advanced-configurations). #### Optimizing hash table -The groupBy v2 engine uses an open addressing hash table for aggregation. The hash table is initialized with a given initial bucket number and gradually grows on buffer full. On hash collisions, the linear probing technique is used. +The groupBy engine uses an open addressing hash table for aggregation. The hash table is initialized with a given initial bucket number and gradually grows on buffer full. On hash collisions, the linear probing technique is used. -The default number of initial buckets is 1024 and the default max load factor of the hash table is 0.7. If you can see too many collisions in the hash table, you can adjust these numbers. See `bufferGrouperInitialBuckets` and `bufferGrouperMaxLoadFactor` in [Advanced groupBy v2 configurations](#groupby-v2-configurations). +The default number of initial buckets is 1024 and the default max load factor of the hash table is 0.7. If you can see too many collisions in the hash table, you can adjust these numbers. See `bufferGrouperInitialBuckets` and `bufferGrouperMaxLoadFactor` in [advanced configurations](#advanced-configurations). #### Parallel combine @@ -350,16 +304,16 @@ longer time than timeseries or topN queries, they should release processing thre However, you might care about the performance of some really heavy groupBy queries. Usually, the performance bottleneck of heavy groupBy queries is merging sorted aggregates. In such cases, you can use processing threads for it as well. This is called _parallel combine_. To enable parallel combine, see `numParallelCombineThreads` in -[Advanced groupBy v2 configurations](#groupby-v2-configurations). Note that parallel combine can be enabled only when +[advanced configurations](#advanced-configurations). Note that parallel combine can be enabled only when data is actually spilled (see [Memory tuning and resource limits](#memory-tuning-and-resource-limits)). -Once parallel combine is enabled, the groupBy v2 engine can create a combining tree for merging sorted aggregates. Each +Once parallel combine is enabled, the groupBy engine can create a combining tree for merging sorted aggregates. Each intermediate node of the tree is a thread merging aggregates from the child nodes. The leaf node threads read and merge aggregates from hash tables including spilled ones. Usually, leaf processes are slower than intermediate nodes because they need to read data from disk. As a result, less threads are used for intermediate nodes by default. You can change the -degree of intermediate nodes. See `intermediateCombineDegree` in [Advanced groupBy v2 configurations](#groupby-v2-configurations). +degree of intermediate nodes. See `intermediateCombineDegree` in [advanced configurations](#advanced-configurations). -Please note that each Historical needs two merge buffers to process a groupBy v2 query with parallel combine: one for +Please note that each Historical needs two merge buffers to process a groupBy query with parallel combine: one for computing intermediate aggregates from each segment and another for combining intermediate aggregates in parallel. @@ -377,18 +331,14 @@ results acceptable. ### Nested groupBys -Nested groupBys (dataSource of type "query") are performed differently for "v1" and "v2". The Broker first runs the -inner groupBy query in the usual way. "v1" strategy then materializes the inner query's results on-heap with Druid's -indexing mechanism, and runs the outer query on these materialized results. "v2" strategy runs the outer query on the -inner query's results stream with off-heap fact map and on-heap string dictionary that can spill to disk. Both -strategy perform the outer query on the Broker in a single-threaded fashion. +Nested groupBys (dataSource of type "query") are performed with the Broker first running the inner groupBy query in the +usual way. Next, the outer query is run on the inner query's results stream with off-heap fact map and on-heap string +dictionary that can spill to disk. The outer query is run on the Broker in a single-threaded fashion. ### Configurations This section describes the configurations for groupBy queries. You can set the runtime properties in the `runtime.properties` file on Broker, Historical, and MiddleManager processes. You can set the query context parameters through the [query context](query-context.md). -#### Configurations for groupBy v2 - Supported runtime properties: |Property|Description|Default| @@ -405,30 +355,12 @@ Supported query contexts: ### Advanced configurations -#### Common configurations for all groupBy strategies - Supported runtime properties: |Property|Description|Default| |--------|-----------|-------| -|`druid.query.groupBy.defaultStrategy`|Default groupBy query strategy.|v2| |`druid.query.groupBy.singleThreaded`|Merge results using a single thread.|false| |`druid.query.groupBy.intermediateResultAsMapCompat`|Whether Brokers are able to understand map-based result rows. Setting this to `true` adds some overhead to all groupBy queries. It is required for compatibility with data servers running versions older than 0.16.0, which introduced [array-based result rows](#array-based-result-rows).|false| - -Supported query contexts: - -|Key|Description| -|---|-----------| -|`groupByStrategy`|Overrides the value of `druid.query.groupBy.defaultStrategy` for this query.| -|`groupByIsSingleThreaded`|Overrides the value of `druid.query.groupBy.singleThreaded` for this query.| - - -#### GroupBy v2 configurations - -Supported runtime properties: - -|Property|Description|Default| -|--------|-----------|-------| |`druid.query.groupBy.bufferGrouperInitialBuckets`|Initial number of buckets in the off-heap hash table used for grouping results. Set to 0 to use a reasonable default (1024).|0| |`druid.query.groupBy.bufferGrouperMaxLoadFactor`|Maximum load factor of the off-heap hash table used for grouping results. When the load factor exceeds this size, the table will be grown or spilled to disk. Set to 0 to use a reasonable default (0.7).|0| |`druid.query.groupBy.forceHashAggregation`|Force to use hash-based aggregation.|false| @@ -440,6 +372,7 @@ Supported query contexts: |Key|Description|Default| |---|-----------|-------| +|`groupByIsSingleThreaded`|Overrides the value of `druid.query.groupBy.singleThreaded` for this query.| |`bufferGrouperInitialBuckets`|Overrides the value of `druid.query.groupBy.bufferGrouperInitialBuckets` for this query.|None| |`bufferGrouperMaxLoadFactor`|Overrides the value of `druid.query.groupBy.bufferGrouperMaxLoadFactor` for this query.|None| |`forceHashAggregation`|Overrides the value of `druid.query.groupBy.forceHashAggregation`|None| @@ -452,23 +385,6 @@ Supported query contexts: |`groupByEnableMultiValueUnnesting`|Safety flag to enable/disable the implicit unnesting on multi value column's as part of the grouping key. 'true' indicates multi-value grouping keys are unnested. 'false' returns an error if a multi value column is found as part of the grouping key.|true| -#### GroupBy v1 configurations - -Supported runtime properties: - -|Property|Description|Default| -|--------|-----------|-------| -|`druid.query.groupBy.maxIntermediateRows`|Maximum number of intermediate rows for the per-segment grouping engine. This is a tuning parameter that does not impose a hard limit; rather, it potentially shifts merging work from the per-segment engine to the overall merging index. Queries that exceed this limit will not fail.|50000| -|`druid.query.groupBy.maxResults`|Maximum number of results. Queries that exceed this limit will fail.|500000| - -Supported query contexts: - -|Key|Description|Default| -|---|-----------|-------| -|`maxIntermediateRows`|Ignored by groupBy v2. Can be used to lower the value of `druid.query.groupBy.maxIntermediateRows` for a groupBy v1 query.|None| -|`maxResults`|Ignored by groupBy v2. Can be used to lower the value of `druid.query.groupBy.maxResults` for a groupBy v1 query.|None| -|`useOffheap`|Ignored by groupBy v2, and no longer supported for groupBy v1. Enabling this option with groupBy v1 will result in an error. For off-heap aggregation, switch to groupBy v2, which always operates off-heap.|false| - #### Array based result rows Internally Druid always uses an array based representation of groupBy result rows, but by default this is translated diff --git a/docs/querying/troubleshooting.md b/docs/querying/troubleshooting.md index 042cfb65dfbc..4b9a83d8a291 100644 --- a/docs/querying/troubleshooting.md +++ b/docs/querying/troubleshooting.md @@ -63,6 +63,6 @@ To mitigate query failure due to web server timeout: Set the max idle time in the `druid.server.http.maxIdleTime` property in the `historical/runtime.properties` file. You must restart the Druid cluster for this change to take effect. See [Configuration reference](../configuration/index.md) for more information on configuring the server. -* If the timeout occurs because the data servers have not pushed any results to the Broker, consider optimizing data server performance. Significant slowdown in the data servers may be a result of spilling too much data to disk in [groupBy v2 queries](groupbyquery.md#performance-tuning-for-groupby-v2), large [`IN` filters](filters.md#in-filter) in the query, or an under scaled cluster. Analyze your [Druid query metrics](../operations/metrics.md#query-metrics) to determine the bottleneck. +* If the timeout occurs because the data servers have not pushed any results to the Broker, consider optimizing data server performance. Significant slowdown in the data servers may be a result of spilling too much data to disk in [groupBy queries](groupbyquery.md#performance-tuning-for-groupby), large [`IN` filters](filters.md#in-filter) in the query, or an under scaled cluster. Analyze your [Druid query metrics](../operations/metrics.md#query-metrics) to determine the bottleneck. * If the timeout is caused by Broker backpressure, consider optimizing Broker performance. Check whether the connection is fast enough between the Broker and deep storage. diff --git a/processing/src/main/java/org/apache/druid/query/QueryCapacityExceededException.java b/processing/src/main/java/org/apache/druid/query/QueryCapacityExceededException.java index 694fbb780cb6..82e6ef8eaeef 100644 --- a/processing/src/main/java/org/apache/druid/query/QueryCapacityExceededException.java +++ b/processing/src/main/java/org/apache/druid/query/QueryCapacityExceededException.java @@ -30,7 +30,7 @@ * * *

* As a {@link QueryException} it is expected to be serialied to a json response, but will be mapped to diff --git a/processing/src/main/java/org/apache/druid/query/groupby/GroupByQueryConfig.java b/processing/src/main/java/org/apache/druid/query/groupby/GroupByQueryConfig.java index d514b90b0233..8eec4c027b41 100644 --- a/processing/src/main/java/org/apache/druid/query/groupby/GroupByQueryConfig.java +++ b/processing/src/main/java/org/apache/druid/query/groupby/GroupByQueryConfig.java @@ -45,8 +45,6 @@ public class GroupByQueryConfig public static final String CTX_KEY_ENABLE_MULTI_VALUE_UNNESTING = "groupByEnableMultiValueUnnesting"; public static final String CTX_KEY_BUFFER_GROUPER_MAX_SIZE = "bufferGrouperMaxSize"; private static final String CTX_KEY_IS_SINGLE_THREADED = "groupByIsSingleThreaded"; - private static final String CTX_KEY_MAX_INTERMEDIATE_ROWS = "maxIntermediateRows"; - private static final String CTX_KEY_MAX_RESULTS = "maxResults"; private static final String CTX_KEY_BUFFER_GROUPER_INITIAL_BUCKETS = "bufferGrouperInitialBuckets"; private static final String CTX_KEY_BUFFER_GROUPER_MAX_LOAD_FACTOR = "bufferGrouperMaxLoadFactor"; private static final String CTX_KEY_MAX_ON_DISK_STORAGE = "maxOnDiskStorage"; @@ -328,14 +326,6 @@ public GroupByQueryConfig withOverrides(final GroupByQuery query) final GroupByQueryConfig newConfig = new GroupByQueryConfig(); final QueryContext queryContext = query.context(); newConfig.singleThreaded = queryContext.getBoolean(CTX_KEY_IS_SINGLE_THREADED, isSingleThreaded()); - newConfig.maxIntermediateRows = Math.min( - queryContext.getInt(CTX_KEY_MAX_INTERMEDIATE_ROWS, getMaxIntermediateRows()), - getMaxIntermediateRows() - ); - newConfig.maxResults = Math.min( - queryContext.getInt(CTX_KEY_MAX_RESULTS, getMaxResults()), - getMaxResults() - ); newConfig.bufferGrouperMaxSize = Math.min( queryContext.getInt(CTX_KEY_BUFFER_GROUPER_MAX_SIZE, getBufferGrouperMaxSize()), getBufferGrouperMaxSize()