From 5956ed30001940548ba0ae7b395321cad2e19aee Mon Sep 17 00:00:00 2001 From: Bryan Boreham Date: Wed, 7 Jul 2021 10:12:13 +0000 Subject: [PATCH] Reduce gRPC concurrent streams to 10,000 This limit governs the number of server calls that gRPC will run in parallel. Beyond this limit, requests will queue in the caller. While parallelism helps to improve throughput, it is unrealistic that 100,000 operations would be able to proceed at once, considering typical CPU core counts, locking, etc. Even 10,000 is probably too high, but changing the limit by more than one order of magnitude also seemed rash. Signed-off-by: Bryan Boreham --- CHANGELOG.md | 1 + cortex/ingester.libsonnet | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c386591..7863c420 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ * [CHANGE] Removed `CortexCacheRequestErrors` alert. This alert was not working because the legacy Cortex cache client instrumentation doesn't track errors. #346 * [CHANGE] Removed `CortexQuerierCapacityFull` alert. #342 * [CHANGE] Changes blocks storage alerts to group metrics by the configured `cluster_labels` (supporting the deprecated `alert_aggregation_labels`). #351 +* [CHANGE] Reduce gRPC max concurrent streams to 10,000. #355 * [ENHANCEMENT] cortex-mixin: Make `cluster_namespace_deployment:kube_pod_container_resource_requests_{cpu_cores,memory_bytes}:sum` backwards compatible with `kube-state-metrics` v2.0.0. #317 * [ENHANCEMENT] Added documentation text panels and descriptions to reads and writes dashboards. #324 * [ENHANCEMENT] Dashboards: defined container functions for common resources panels: containerDiskWritesPanel, containerDiskReadsPanel, containerDiskSpaceUtilization. #331 diff --git a/cortex/ingester.libsonnet b/cortex/ingester.libsonnet index 34b4d987..e0753d84 100644 --- a/cortex/ingester.libsonnet +++ b/cortex/ingester.libsonnet @@ -31,7 +31,7 @@ 'ingester.max-series-per-query': $._config.limits.max_series_per_query, 'ingester.max-samples-per-query': $._config.limits.max_samples_per_query, 'runtime-config.file': '/etc/cortex/overrides.yaml', - 'server.grpc-max-concurrent-streams': 100000, + 'server.grpc-max-concurrent-streams': 10000, 'server.grpc-max-send-msg-size-bytes': 10 * 1024 * 1024, 'server.grpc-max-recv-msg-size-bytes': 10 * 1024 * 1024, } + (