Set target_throughput to none for NYC taxis nightly benchmarks #4725
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Set
target_throughput
tonone
for nyc_taxis nightly runs. Ref:https://github.com/opensearch-project/opensearch-benchmark-workloads/blob/main/nyc_taxis/test_procedures/default.json#L56-L69
Illustrating the issue with 2 screenshots below:
First, this is p90 CPU usage of the entire test procedure. The section in the middle where the graph is not at 0 is for the
distance_amount_agg
operation while the p90 CPU is at 0% for all other operations of the procedure.Similar story is painted by the
search
threadpool active thread count metric below. These node stats are sampled at 1 second intervals, so we can see from this that after thedistance_amount_agg
we are not really filling up the search threadpool and thus not really generating real load on the cluster.This problem is also evident when we look at the performance numbers themselves as we increase the client count for operations other than the
distance_amount_agg
as we do not see performance decreasing with increased client count for other operations, even in the case of concurrent search enabled which should be consuming all CPU resources at the higher client counts.Issues Resolved
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.