This repository has been archived by the owner on Aug 2, 2022. It is now read-only.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes:
Query Engine Memory Circuit Breaker [WIP]
1.Overview
1.1.Problem Statements
In current Query Engine, the query execution could been divided into two major stages, plan stage and execution stage. The plan stage doesn’t load any data, so the memory usage could be ignored. At executin stage, then the physical execution plan is executed on the single node. The physical execution plan is composed by a chain of Iterator. There are two type of iterator, In-memory physical iterator and Elasticserach query. The Elasticsaerch query iterator submit the query request to Elasticsearch cluster then pass the response to the In-memory physical iterator for futher processing. The In-memory physical iterator process the event based on the heap memory. for example, the physical execution plan could be:
InMemSortIterator → InMemAggIterator → InMemAggIterator → ElasticsearchIndexScanIterator.
OutOfMemoryError
issue.1.2.Estimate the per query memory usage is hard
Firstly, we should define what is the query memory usage. As we described in the problem statements section, the plan stage query memory usage could be ignored becase it didn’t process data. Then we should focus on the memory usage during query execution stage. Both the in-memory processiong and Elasticsearch query will use heap memory, then we can define the per query memory usage as the sum of memory usage of in-memory processing and Elasticsearch query. denoted as
**PerQueryMemUsage = SUM(InMemOpMemUsage + ESQueryMemUsage)**
.Secondly, we should define how to calculate the InMemOpMemUsage and ESQueryMemUsage. Let’s talk about them seperately. InMemOpMemUsage could be calculated based on the Lucene’s RamUsageEstimator which could Estimates the size (memory representation) of Java objects. But ESQueryMemUsage is estimzated the Elastisearch internal implementation which is not exposed to client (Todo: need further confirm this with Elasticsearch source code).
Overall, missing the method to calcuate the ESQueryMemUsage make the PerQueryMemUsage estimatioin hard.
1.3.Do we need per query memory usage limit?
To answer this question, we could explore some use cases. One real use cases is in here. By using the per request circuit breaker, the user could fast fail the expensive query without crashing the entire cluster.
As we explain in the previous section, because we can not estimate the PerQueryMemUsage, the limit of per query memory usage is not accurate. Could we avoid this problem? Do we have workround to protect the cluster from crashing?
Let’s go back to the use cases, does it make sense to limit the expensive query which usurally much usefull for data analysis.
Instead of reject expensive query, if we reject query randomly and add backoff retry for memory allocation.
1.1.x.Summary
In this document, we are trying to find the way avoid in memory operator make Elasicseach running into
OutOfMemoryError
when handling the query request.1.2.Tennet
We following the tennets when designing the feature:
1.2.Requirement
1.2.1.Function Requirement
1.2.2.Insight and Operation Requirement
1.3.Out of scope
1.3.1.Physical plan optimizaiton
The goal of physical plan optimizaiton is try to find the plan which could running the query by taking the benfit from Elasticsearch and Lucene’s capability. The optimization doesn’t guarantee there is no in-memory operator exists. In reallity, it is impossible. If thre are in-memory operator in there, what we can do is avoid the in-memory operator make Elasticsearch OOM.
1.3.2.In memory operator optimization
There must be some solution which could improve the memory consumeption for each in-memory operator. For example, we can use file for sorting. But in this document, we don’t cover this topic in detail. Our discussion in here is the worst case sceneral, the better algorithm could make the query engine have better performance, but it didn’t impact the circuit breaker solution in here.
1.4.Measure of success
1.4.1.stability
Under certern pressue test, the test cluster should have no OOM.
1.4.2.error rate linear to load
Under certern load test, the request error rate should linear increase with load.
3.Solution
The following diagram explain the relationship between each component in the system.
[Image: Screen Shot 2020-06-25 at 10.19.49 AM.png]
Resource Monitor
ResourceMonitor only have one interface isHealthy which is used to monitor the resource healthy. The storage engine should has their own implementation for monitor the resource. e.g. ElasticsearchResourceMonitor provide the implementation for Elasticsearch.
Internally, ElasticsearchResourceMonitor only monitor the real memory usage now. The algorithm is
[Image: Screen Shot 2020-06-24 at 9.59.33 AM.png]
Setting
User could use opendistro.ppl.query.memory_limit setting to config the limit for memory usage limitation. The default value 85%.
Appendix
1.Circuit Breaker in Elasticsearch
Elasticsearch has Circuit breaker settings in hierarchy strcuture.
For example, the request circuit breaker allows Elasticsearch to prevent per-request data structures (for example, memory used for calculating aggregations during a request) from exceeding a certain amount of memory. It has indices.breaker.total.limit as the total limit.
From Elasticsearch 7.0.0, it release the https://www.elastic.co/blog/improving-node-resiliency-with-the-real-memory-circuit-breaker, which determining whether the parent breaker should take real memory usage into account.
2.Circuit Breaker in other CrateDB
CrateDB integrate with Elasticsearch’s CircuitBreaker architecture by create it own
CrateCircuitBreakerService
. and exposed similar setting for client.In the nutshell, Crate use Lucene’s RamUsageEstimator to estimate the Row ram usage. Crate create different SizeEstimator based on the column type. Further more, Crate add the optimization on top of it like SamplingSizeEstimator.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.