Distinguish+expand Hot Spotting from Task Backlog

troubleshoot/elasticsearch/hotspotting.md

@@ -23,7 +23,9 @@
 
 ## Detect hot spotting [detect]
 
+### Active [detect-active]
+
 Hot spotting most commonly surfaces as significantly elevated resource utilization (of `disk.percent`, `heap.percent`, or `cpu`) among a subset of nodes as reported via [cat nodes](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-nodes). Individual spikes aren’t necessarily problematic, but if utilization repeatedly spikes or consistently remains high over time (for example longer than 30 seconds), the resource may be experiencing problematic hot spotting.
 
 For example, let’s show case two separate plausible issues using cat nodes:
 
@@ -42,6 +44,38 @@
 
 Here we see two significantly unique utilizations: where the master node is at `cpu: 95` and a hot node is at `disk.used_percent: 90%`. This would indicate hot spotting was occurring on these two nodes, and not necessarily from the same root cause.
 
+### Historical [detect-historical]
+
+A secondary method to notice hot spotting build up is to poll the [node statistics API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-stats) for index-related performance metrics.
+
+```console
+GET _nodes/stats?pretty=true&filter_path=nodes.*.name,nodes.*.roles,nodes.*.indices
+```
+
+This outputs node operational metrics like `query`, `refresh`, and `index`. It allows you to gauge the
+
+* total events attempted per node 
+* node's average processing time per event type
+
+These metrics accumulate from individual node uptime. As example to view some of this output, you can parse this response using [third-party tool JQ](https://jqlang.github.io/jq/):
+
+```bash
+cat nodes_stats.json | jq -rc '.nodes[]|.name as $n|.roles as $r|.indices|to_entries[]|.key as $m|.value|select(.total and .total_time_in_millis)|select(.total>0)|{node:$n, roles:$r, metric:$m, total:.total, avg_millis:(.total_time_in_millis?/.total|round)}'
+```
+
+Multiple major operations being nonperformant across nodes likely suggest under provisioned cluster. If a particular operation type or node stands out, it likely indicates [shard distribution issues](#causes-shards) which you might compare against [indices stats](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-stats).
+
+```console
+GET /_stats?level=shards&human&expand_wildcards=all&ignore_unavailable=true
+```
+
+These metrics accumulate from individual shard history. As an example using this output par to before, you can parse this response with JQ:
+
+```bash
+cat indices_stats.json | jq -rc '.indices|to_entries[]|.key as $i|.value.shards[]|to_entries[]|.key as $sh|.value|.routing.primary as $p|.routing.node[:4] as $n|to_entries[]|.key as $m|.value|select(.total and .total_time_in_millis)|select(.total>0)|{index:$i, shard:$sh, primary:$p, node:$n, metric:$m, total:.total, avg_millis:(.total_time_in_millis/.total|round)}'
+```
+
+
 
 ## Causes [causes]
 
@@ -146,36 +180,4 @@
 
 ### Task loads [causes-tasks]
 
-Shard distribution problems will most-likely surface as task load as seen above in the [cat thread pool](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-thread-pool) example. It is also possible for tasks to hot spot a node either due to individual qualitative expensiveness or overall quantitative traffic loads.
-
-For example, if [cat thread pool](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-thread-pool) reported a high queue on the `warmer` [thread pool](elasticsearch://reference/elasticsearch/configuration-reference/thread-pool-settings.md), you would look-up the effected node’s [hot threads](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-hot-threads). Let’s say it reported `warmer` threads at `100% cpu` related to `GlobalOrdinalsBuilder`. This would let you know to inspect [field data’s global ordinals](elasticsearch://reference/elasticsearch/mapping-reference/eager-global-ordinals.md).
-
-Alternatively, let’s say [cat nodes](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-nodes) shows a hot spotted master node and [cat thread pool](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-thread-pool) shows general queuing across nodes. This would suggest the master node is overwhelmed. To resolve this, first ensure [hardware high availability](../../deploy-manage/production-guidance/availability-and-resilience/resilience-in-small-clusters.md) setup and then look to ephemeral causes. In this example, [the nodes hot threads API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-hot-threads) reports multiple threads in `other` which indicates they’re waiting on or blocked by either garbage collection or I/O.
-
-For either of these example situations, a good way to confirm the problematic tasks is to look at longest running non-continuous (designated `[c]`) tasks via [cat task management](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-tasks). This can be supplemented checking longest running cluster sync tasks via [cat pending tasks](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-pending-tasks). Using a third example,
-
-```console
-GET _cat/tasks?v&s=time:desc&h=type,action,running_time,node,cancellable
-```
-
-This could return:
-
-```console-result
-type   action                running_time  node    cancellable
-direct indices:data/read/eql 10m           node_1  true
-...
-```
-
-This surfaces a problematic [EQL query](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-search). We can gain further insight on it via [the task management API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-tasks),
-
-```console
-GET _tasks?human&detailed
-```
-
-Its response contains a `description` that reports this query:
-
-```eql
-indices[winlogbeat-*,logs-window*], sequence by winlog.computer_name with maxspan=1m\n\n[authentication where host.os.type == "windows" and event.action:"logged-in" and\n event.outcome == "success" and process.name == "svchost.exe" ] by winlog.event_data.TargetLogonId
-```
-
-This lets you know which indices to check (`winlogbeat-*,logs-window*`), as well as the [EQL search](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-eql-search) request body. Most likely this is [SIEM related](/solutions/security.md). You can combine this with [audit logging](../../deploy-manage/security/logging-configuration/enabling-audit-logs.md) as needed to trace the request source.
+Shard distribution problems will most-likely surface as task load as seen above in the [cat thread pool](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-thread-pool) example. It is also possible for tasks to hot spot a node either due to individual qualitative expensiveness or overall quantitative traffic loads, which will surface in [backlogged tasks](/troubleshoot/elasticsearch/task-queue-backlog.md).

troubleshoot/elasticsearch/task-queue-backlog.md

@@ -12,35 +12,36 @@
 % **Product:** Elasticsearch<br> **Deployment type:** Elastic Cloud % Enterprise, Elastic Cloud Hosted, Elastic Cloud on Kubernetes, Elastic
 % Self-Managed <br> **Versions:** All
 
-A backlogged task queue can prevent tasks from completing and lead to an unhealthy cluster state. Contributing factors include resource constraints, a large number of tasks triggered at once, and long-running tasks.
+A backlogged task queue can lead to [rejected requests](/troubleshoot/elasticsearch/rejected-requests.md) or an [unhealthy cluster state](/troubleshoot/elasticsearch/red-yellow-cluster-status.md). Contributing factors include [uneven or resource constrained hardware](/troubleshoot/elasticsearch/hotspotting.md#causes-hardware), a large number of tasks triggered at once, expensive tasks using [high CPU](/troubleshoot/elasticsearch/high-cpu-usage.md) or inducing [high JVM](/troubleshoot/elasticsearch/high-jvm-memory-pressure.md), and long-running tasks.
 
 
 ## Diagnose a backlogged task queue [diagnose-task-queue-backlog] 
 
 To identify the cause of the backlog, try these diagnostic actions.
 
-* [Check the thread pool status](#diagnose-task-queue-thread-pool)
-* [Inspect hot threads on each node](#diagnose-task-queue-hot-thread)
+* [Check thread pool status](#diagnose-task-queue-thread-pool)
+* [Inspect node hot threads](#diagnose-task-queue-hot-thread)
 * [Identify long-running node tasks](#diagnose-task-queue-long-running-node-tasks)
 * [Look for long-running cluster tasks](#diagnose-task-queue-long-running-cluster-tasks)
 
 
-### Check the thread pool status [diagnose-task-queue-thread-pool] 
-
-A [depleted thread pool](high-cpu-usage.md) can result in [rejected requests](rejected-requests.md).
+### Check thread pool status [diagnose-task-queue-thread-pool] 
 
 Use the [cat thread pool API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-thread-pool) to monitor active threads, queued tasks, rejections, and completed tasks:
 
 ```console
 GET /_cat/thread_pool?v&s=t,n&h=type,name,node_name,active,queue,rejected,completed
 ```
 
-* Look for high `active` and `queue` metrics, which indicate potential bottlenecks and opportunities to [reduce CPU usage](high-cpu-usage.md#reduce-cpu-usage).
-* Determine whether thread pool issues are specific to a [data tier](../../manage-data/lifecycle/data-tiers.md).
-* Check whether a specific node’s thread pool is depleting faster than others. This might indicate [hot spotting](#resolve-task-queue-backlog-hotspotting).
+You will be checking for bottlenecks such as:
+
+* Look for continually high `queue` metrics, which indicate long-running tasks or [CPU-expensive tasks](high-cpu-usage.md).
+* Look for bursts of elevated `queue` metrics, which indicate opportunities to spread traffic volume.
+* Determine whether thread pool issues are specific to a [node role](/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md).
+* Check whether a specific node is depleting faster than others within a [data tier](/manage-data/lifecycle/data-tiers.md). This might indicate [hot spotting](/troubleshoot/elasticsearch/hotspotting.md).
 
 
-### Inspect hot threads on each node [diagnose-task-queue-hot-thread] 
+### Inspect node hot threads [diagnose-task-queue-hot-thread] 
 
 If a particular thread pool queue is backed up, periodically poll the [nodes hot threads API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-hot-threads) to gauge the thread’s progression and ensure it has sufficient resources:
 
@@ -76,46 +77,58 @@
 
 Long-running tasks might need to be [canceled](#resolve-task-queue-backlog-stuck-tasks).
 
-See this [this video](https://www.youtube.com/watch?v=lzw6Wla92NY) for a walkthrough of troubleshooting the [task management API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-tasks) output.
+See this [this video](https://www.youtube.com/watch?v=lzw6Wla92NY) for a walkthrough of troubleshooting the [task management API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-tasks) output. 
+
+Refer also to tuning for [searching speed](/deploy-manage/production-guidance/optimize-performance/search-speed.md) and [indexing speed](/deploy-manage/production-guidance/optimize-performance/indexing-speed.md) for more information.
 
 ### Look for long-running cluster tasks [diagnose-task-queue-long-running-cluster-tasks] 
 
-Use the [cluster pending tasks API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-pending-tasks) to identify delays in cluster state synchronization:
+Use the [cat pending tasks API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-pending-tasks) to identify delays in cluster state synchronization:
 
 ```console
-GET /_cluster/pending_tasks
+GET /_cat/pending_tasks?v=true
 ```
 
-Tasks with a high `timeInQueue` value are likely contributing to the backlog and might need to be [canceled](#resolve-task-queue-backlog-stuck-tasks).
+This can expect to fall behind during an [unstable cluster](/troubleshoot/elasticsearch/troubleshooting-unstable-cluster.md), but otherwise usually indicates an unworkable [cluster setting](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-get-settings) override or traffic pattern.
+
+Some common `source` to look out for:
+
+* `ilm-`: [ILM](/manage-data/lifecycle/index-lifecycle-management.md) polls every `10m` by default from setting [`indices.lifecycle.poll_interval`](elasticsearch://reference/elasticsearch/configuration-reference/index-lifecycle-management-settings.md). It kicks off asynchronous tasks executed by the node tasks. If ILM continually reports as a cluster pending task this setting was likely overrode else the cluster likely has misconfigured [indices count to master heap](/deploy-manage/production-guidance/optimize-performance/size-shards.md#shard-count-recommendation).
+* `put-mapping`: {{es}} enables [dynamic mapping](/manage-data/data-store/mapping/dynamic-mapping.md) by default. This or the [update mapping API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-mapping) triggers a mapping update. Correlating cluster log will contain `update_mapping` along with the effected index name.
+* `shard-started`: Indicates [active shard recoveries](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cat-recovery). Overriding [`cluster.routing.allocation.*` settings](elasticsearch://reference/elasticsearch/configuration-reference/cluster-level-shard-allocation-routing-settings.md#cluster-shard-allocation-settings) can cause this to back up.
 
 
 ## Recommendations [resolve-task-queue-backlog] 
 
 After identifying problematic threads and tasks, resolve the issue by increasing resources or canceling tasks.
 
+### Address cpu-intensive tasks [resolve-task-queue-backlog-cpu]
 
-### Increase available resources [resolve-task-queue-backlog-resources] 
+If an individual task is causing a [thread pool `queue`]](#diagnose-task-queue-thread-pool) due to [high CPU usage](high-cpu-usage.md), try [cancelling it](#resolve-task-queue-backlog-stuck-tasks) to then optimize it before retrying.
 
-If tasks are progressing slowly, try [reducing CPU usage](high-cpu-usage.md#reduce-cpu-usage).
+Frequently when this surfaces it's due to:
 
-In some cases, you might need to increase the thread pool size. For example, the `force_merge` thread pool defaults to a single thread. Increasing the size to 2 might help reduce a backlog of force merge requests.
+* creating new or modifying scheduled tasks which run frequently or are wide affecting, such as [ILM](/manage-data/lifecycle/index-lifecycle-management.md) policies or [Rules](/explore-analyze/alerts-cases.md)
+* performing traffic load testing
+* doing extended look backs, especially across [data tiers](/manage-data/lifecycle/data-tiers.md)
+* [searching](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-search) or [bulk](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk) updating a high number of indices within a request
 
 
 ### Cancel stuck tasks [resolve-task-queue-backlog-stuck-tasks] 
 
-If an active task’s [hot thread](#diagnose-task-queue-hot-thread) shows no progress, consider [canceling the task](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-tasks#task-cancellation).
+If an active task’s [hot thread](#diagnose-task-queue-hot-thread) shows no progress, consider [canceling the task](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-tasks#task-cancellation) if it's flagged as `cancellable`.
 
 
 ### Address hot spotting [resolve-task-queue-backlog-hotspotting] 
 
-If a specific node’s thread pool is depleting faster than others, try addressing uneven node resource utilization, also known as hot spotting. For details on actions you can take, such as rebalancing shards, see [Hot spotting](hotspotting.md).
+If a specific node’s thread pool is depleting faster than its [data tier](/manage-data/lifecycle/data-tiers.md) peers, try addressing uneven node resource utilization, also known as hot spotting. For details on actions you can take, such as rebalancing shards, see [hot spotting](hotspotting.md).
 
+### Increase available resources [resolve-task-queue-backlog-resources] 
 
-## Resources [_resources] 
+By default, {{es}} allocates processors equal to the number reported available by the operating system. You can override this behaviour by adjusting the value of [`node.processors`](elasticsearch://reference/elasticsearch/configuration-reference/thread-pool-settings.md#node.processors), but this advanced setting should be configured only after you've performed load testing.
 
-Related symptoms:
+In some cases, you might need to increase the problematic thread pool `size`. For example, it might help to increase a stuck [`force_merge` thread pool](elasticsearch://reference/elasticsearch/configuration-reference/thread-pool-settings.md). If automatically calculated to `1` based on available CPU processors, then increasing to `2` would appear in `elasticsearch.yml` like:
 
-* [High CPU usage](high-cpu-usage.md)
-* [Rejected requests](rejected-requests.md)
-* [Hot spotting](hotspotting.md)
-* [Troubleshooting overview](/troubleshoot/index.md)
+```yaml
+thread_pool.force_merge.size: 2
+```