Expand Circuit Breaker Troubleshooting

troubleshoot/elasticsearch/circuit-breaker-errors.md

@@ -24,7 +24,7 @@
 
 **Error messages**
 
-If a request triggers a circuit breaker, {{es}} returns an error with a `429` HTTP status code.
+A circuit breaker trips when it prevents a request from executing in order to protect the node's stability. When a request triggers a circuit breaker, {{es}} returns an error with a `429` HTTP status code.
 
 ```js
 {
@@ -45,6 +45,20 @@
 Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<transport_request>] would be [num/numGB], which is larger than the limit of [num/numGB], usages [request=0/0b, fielddata=num/numKB, in_flight_requests=num/numGB, accounting=num/numGB]
 ```
 
+**Check circuit breaker statistics**
+
+You can use the [node stats API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-stats) to get statistics about the circuit breaker per node.
+
+```console
+GET _nodes/stats?filter_path=nodes.*.breakers
+```
+
+This will show you:
+- Estimated memory used for the operation.
+- Memmory limit for the circuit breaker.
+- Total number of times the circuit breaker has been triggered and prevented an out of memory error.
+- And an overhead which is a constant that all estimates for the circuit breaker are multiplied with to calculate a final estimate.
+
 **Check JVM memory usage**
 
 If you’ve enabled Stack Monitoring, you can view JVM memory usage in {{kib}}. In the main menu, click **Stack Monitoring**. On the Stack Monitoring **Overview** page, click **Nodes**. The **JVM Heap** column lists the current memory usage for each node.
@@ -79,3 +93,25 @@
 ```console
 POST _cache/clear?fielddata=true
 ```
+
+## Circuit Breaker Types
+
+[Circuit breaker types](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md) are manually defined for known expensive code paths with a sum-up [parent circuit breaker](elasticsearch://reference/elasticsearch/configuration-reference/circuit-breaker-settings.md#parent-circuit-breaker). Breakers need to be resolved per breaker type; common examples:
+
+- `accounting`: Lucene segments and their overhead. Commonly indicates need to [force merge](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-forcemerge) or [fix large shards](elasticsearch://deploy-manage/production-guidance/optimize-performance/size-shards.md).
+- `fielddata`: [field data] and [global ordinals]
+- `eql_sequence`: [eql sequences]
+- `request`: API request bodies. Commonly for [Bulk] sizes too large
+- `inflight_requests`: total API network bytes
+- `script`: [scripts]
+- `regex`: [regex]
+
+**Memory evaluation**
+
+Circuit breakers may either directly evaluate memory usage estimates or indirectly limit operations that are likely to cause excessive memory consumption. For example, the `script` circuit breaker checks memory indirectly by rate-limiting Painless/Mustache script compilations. However, even with circuit breakers in place, nodes can still encounter out-of-memory (OOM) conditions. This can occur, for example, because:
+
+- Circuit breaker relies on point-in-time memory usage estimations.
+- Parallel operations may still heap DOS-attack the node even with `parent` circuit breakers.
+- Certain dynamic operations can quickly consume substantial memory. For example [aggregations](elasticsearch://explore-analyze/query-filter/aggregations.md) and [complex queries](elasticsearch://reference/query-languages/query-dsl/compound-queries).
+Circuit breakers protect JVM heap memory, but if OOM occurs due to non-heap memory; for example JVM for complication (code cache) or thread stacks.
+