From 52c62915b4fad717903fa6c4fc0651f7d8c84d11 Mon Sep 17 00:00:00 2001 From: Shimi Bandiel Date: Wed, 29 Jul 2026 13:17:33 -0700 Subject: [PATCH 1/2] docs: explain and surface the prometheus-budget closing point max_concurrency is a per-ready-pod divisor: the prometheus-budget gate closes only once load reaches ready_pods * max_concurrency * (1 - baseline). Nothing said so, and nothing reported the resolved number, so a max_concurrency the pool can never reach left the gate permanently open with every batch request dispatching regardless of live traffic -- silently, which is the exact failure the gate exists to prevent. Log the resolved closing point when the gate is built: "prometheus-budget gate configured" pool=... maxConcurrency=100 baseline=0.05 closesAtLoadPerReadyPod=95 Document max_concurrency as per-pod capacity in the README, the CreateGate doc comment, and the e2e guide, with two ways to pick a value: match the EPP saturation detector's MaxConcurrency so the async gate and EPP agree on when the pool is full, or measure the per-pod peak of vllm:num_requests_running under the load you consider saturated. The guide keeps max_concurrency: "100". It is reachable in the setup the guide actually deploys -- a single Qwen3-0.6B replica driven by the guide's own 200-worker load test, which the guide already records as reaching 200 concurrent -- and it matches the EPP default that the llm-d optimized-baseline recipe leaves unset. Lowering it would contradict the expected values the guide states. What was missing was any indication that the number is per-pod and must be re-derived for a different model, pool size, or workload, so that is what this adds, in the values file and in a new "Size max_concurrency for your pool" section. Fixes #363 Signed-off-by: Shimi Bandiel --- README.md | 36 ++++++++++++++- docs/guides/e2e-deploy.md | 45 ++++++++++++++++++- .../guides/e2e-deploy/llm-d-async-values.yaml | 8 ++++ .../inference/flowcontrol/gate_factory.go | 17 +++++++ 4 files changed, 104 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 7db25490..be56f9dd 100644 --- a/README.md +++ b/README.md @@ -291,10 +291,44 @@ For more fine-grained control, configure gates per queue in your configuration f `inference_pool_ready_pods{name=""}` (EPP metric) and, for the vLLM fallback, the `inference_pool` label on scraped vLLM metrics (added via relabeling from pod labels). - `namespace` (optional): Kubernetes namespace to scope metric queries. Required when multiple namespaces share the same pool name with a shared Prometheus instance. - - `max_concurrency` (optional): Per-endpoint request capacity (`MaxConcurrency` in the [inference scheduler's saturation detector](https://github.com/llm-d/llm-d-inference-scheduler/blob/main/pkg/epp/framework/plugins/flowcontrol/saturationdetector/concurrency/config.go)). Default is `100` (matching the inference scheduler default). + - `max_concurrency` (optional): Per-endpoint request capacity (`MaxConcurrency` in the [inference scheduler's saturation detector](https://github.com/llm-d/llm-d-inference-scheduler/blob/main/pkg/epp/framework/plugins/flowcontrol/saturationdetector/concurrency/config.go)). Default is `100` (matching the inference scheduler default). See [sizing `max_concurrency`](#sizing-max_concurrency) below — this is a **per-pod** number, and setting it above what a pod actually serves makes the gate inert. - `baseline` (optional): Reserved baseline B. The gate closes when D ≤ B. Default is `0.05`. - `fallback` (optional): Fallback budget value (0.0-1.0) returned when all metric sources are unavailable. Default is `0.0` (fail closed). + + **Sizing `max_concurrency`.** Because `max_SYS = ready_pods × max_concurrency`, the gate closes only + once the observed load reaches: + + ``` + ready_pods × max_concurrency × (1 − baseline) + ``` + + At the defaults that is 95 concurrent requests *per ready pod*. A pool that never gets near + that — a large model on a few replicas, for instance — leaves the gate permanently open, so every + batch request dispatches regardless of live traffic. The gate logs its resolved closing point at + startup so you can compare it against reality: + + ``` + "prometheus-budget gate configured" pool=... maxConcurrency=100 baseline=0.05 closesAtLoadPerReadyPod=95 + ``` + + Two ways to pick a value: + + - **Match the EPP.** `max_concurrency` mirrors `MaxConcurrency` in the inference scheduler's + saturation detector. Using the same number keeps the async gate and the EPP's own admission + control in agreement about when the pool is full. If you have not configured the saturation + detector, both defaults are `100`. + - **Measure it.** Drive your pool to the load you consider saturated and read the per-pod peak: + + ```promql + max_over_time( + (sum(vllm:num_requests_running{inference_pool=""}) / on() inference_pool_ready_pods{name=""})[1h:] + ) + ``` + + Set `max_concurrency` to that peak. Values well above it mean the gate never closes; values + well below it mean the gate sheds while the pool still has room. + **Metric prerequisites:** The primary metric source requires llm-d's flow control plugin to be enabled; without it, the gate falls back to vLLM metrics. The fallback filters by `inference_pool` label, which vLLM does not emit natively: configure Prometheus relabeling to propagate it from model server pod labels diff --git a/docs/guides/e2e-deploy.md b/docs/guides/e2e-deploy.md index da9090c9..878f0d93 100644 --- a/docs/guides/e2e-deploy.md +++ b/docs/guides/e2e-deploy.md @@ -165,7 +165,9 @@ helm install llm-d-async ${ASYNC_REPO}/charts/llm-d-async/ \ The values file (`docs/guides/e2e-deploy/llm-d-async-values.yaml`) configures: - Image: `ghcr.io/llm-d/llm-d-async:938cd44` - Queue: Redis sorted-set with `redis.url` set directly (chart creates the Secret), configured via `queuesConfig` -- Gate: `prometheus-budget` with pool=`optimized-baseline`, max_concurrency=100, baseline=0.05 (per-queue) +- Gate: `prometheus-budget` with pool=`optimized-baseline`, max_concurrency=100, baseline=0.05 (per-queue). + `max_concurrency` is a **per ready pod** capacity — see [Size `max_concurrency` for your pool](#size-max_concurrency-for-your-pool) + before reusing this value on your own model - Prometheus URL pointing to the cluster's `llmd-kube-prometheus-stack-prometheus` service > **Multi-namespace deployments:** If the cluster has multiple inference pools @@ -192,6 +194,47 @@ The values file (`docs/guides/e2e-deploy/llm-d-async-values.yaml`) configures: - `grafana.dashboards.enabled: true` — provisions a Grafana dashboard (via sidecar) with request rate, outcome breakdown, success/retry gauges, and latency percentiles +### Size `max_concurrency` for your pool + +`max_concurrency` is the request capacity of **one ready pod**, not of the pool. The gate +computes `max_SYS = ready_pods × max_concurrency` and closes only once the observed load +reaches: + +``` +ready_pods × max_concurrency × (1 - baseline) +``` + +With the values above that is `1 × 100 × 0.95` = **95 concurrent requests** against the single +Qwen3-0.6B replica this guide deploys. That is reachable here — the saturation test below drives +200 concurrent requests, and 100 also matches the default `MaxConcurrency` of the EPP's saturation +detector, so the async gate and the EPP agree on when the pool is full. + +It is not automatically reachable anywhere else. Point this configuration at a larger model on a +few replicas and a realistic workload may peak in the single digits per pod, far below the closing +point — in which case the gate never closes and every batch request dispatches regardless of live +traffic, which is the exact failure the gate exists to prevent. + +The processor logs its resolved closing point when it builds the gate, so check it against reality: + +```bash +kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=llm-d-async | grep "prometheus-budget gate configured" +# "prometheus-budget gate configured" pool=optimized-baseline maxConcurrency=100 baseline=0.05 closesAtLoadPerReadyPod=95 +``` + +To derive the value for your own model and hardware, drive the pool to the load you consider +saturated and read the per-pod peak: + +```bash +kubectl run --rm -i prom-peak --image=curlimages/curl --restart=Never -n ${NAMESPACE} -- \ + curl -s --data-urlencode \ + 'query=max_over_time((sum(vllm:num_requests_running{inference_pool="optimized-baseline"}) / on() inference_pool_ready_pods{name="optimized-baseline"})[1h:])' \ + 'http://llmd-kube-prometheus-stack-prometheus.llm-d-monitoring.svc.cluster.local:9090/api/v1/query' +``` + +Set `max_concurrency` to that peak. Well above it and the gate never closes; well below it and the +gate sheds while the pool still has room. If you change it, update the `* 100` divisor in the +verification queries below to match. + ## Verify ### Check pods and gate status diff --git a/docs/guides/e2e-deploy/llm-d-async-values.yaml b/docs/guides/e2e-deploy/llm-d-async-values.yaml index c30fca9e..4757366f 100644 --- a/docs/guides/e2e-deploy/llm-d-async-values.yaml +++ b/docs/guides/e2e-deploy/llm-d-async-values.yaml @@ -24,6 +24,14 @@ ap: gate_type: "prometheus-budget" gate_params: pool: "optimized-baseline" + # Per *ready pod* request capacity, not a pool-wide total: the gate + # closes once load reaches ready_pods * max_concurrency * (1 - baseline), + # i.e. 95 concurrent requests on the single Qwen3-0.6B replica this + # guide deploys. 100 matches the EPP saturation detector's default and + # is reachable by the guide's own load test (hey, 200 workers). + # Re-derive it for your own model and hardware -- a value your pool + # never reaches leaves the gate permanently open. See the + # "Size max_concurrency for your pool" section of the guide. max_concurrency: "100" baseline: "0.05" podMonitor: diff --git a/pkg/async/inference/flowcontrol/gate_factory.go b/pkg/async/inference/flowcontrol/gate_factory.go index 066dfcb2..a98362c1 100644 --- a/pkg/async/inference/flowcontrol/gate_factory.go +++ b/pkg/async/inference/flowcontrol/gate_factory.go @@ -26,6 +26,7 @@ import ( redisgate "github.com/llm-d/llm-d-async/pkg/redis" promapi "github.com/prometheus/client_golang/api" goredis "github.com/redis/go-redis/v9" + "sigs.k8s.io/controller-runtime/pkg/log" ) // DefaultCacheTTL is the default TTL for cached Prometheus metric sources. @@ -87,6 +88,10 @@ func (f *GateFactory) Close() error { // Gate closes when D ≤ B (baseline); returns D − B when open, so callers compute // N = max_SYS × (D − B). Params: pool (required), // max_concurrency (default 100), baseline (default 0.05), fallback (default 0.0) +// max_concurrency is per ready pod, not for the pool as a whole: the gate closes +// once the observed load reaches max_concurrency × (1 − baseline) per ready pod. +// Set it too high for the pool's real capacity and the gate never closes; the +// resolved closing point is logged at gate creation so this is visible. // - "prometheus-query": Evaluates an arbitrary user-supplied PromQL expression as the dispatch // budget. The expression must resolve to an instant vector with a single sample whose value // is in [0, 1]. Unlike prometheus-saturation and prometheus-budget, this gate does not @@ -280,6 +285,18 @@ func (f *GateFactory) CreateGate(cfg pipeline.GateConfig) (pipeline.Gate, error) cachedSource(primary, f.cacheTTL), cachedSource(secondary, f.cacheTTL), ) + + // Report the resolved closing point. max_concurrency is a per-ready-pod + // divisor, so a value the pool can never reach leaves the gate + // permanently open with nothing in the logs or metrics to say so. + // Surfacing it here makes a mis-sized max_concurrency visible at startup. + log.Log.WithName("gate-factory").Info("prometheus-budget gate configured", + "pool", pool, + "maxConcurrency", maxConcurrency, + "baseline", baseline, + "closesAtLoadPerReadyPod", maxConcurrency*(1-baseline), + ) + return NewBudgetDispatchGate(ms, baseline, fallback). WithOwner(cfg.Owner). WithInferencePool(pool), nil From 9cc87da2f3813551edd0dd6163bbd7cd02f0f3b8 Mon Sep 17 00:00:00 2001 From: Shimi Bandiel Date: Wed, 29 Jul 2026 13:18:23 -0700 Subject: [PATCH 2/2] docs: add release note for #374 Signed-off-by: Shimi Bandiel --- release-notes.d/unreleased/374.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 release-notes.d/unreleased/374.md diff --git a/release-notes.d/unreleased/374.md b/release-notes.d/unreleased/374.md new file mode 100644 index 00000000..d9cc50ce --- /dev/null +++ b/release-notes.d/unreleased/374.md @@ -0,0 +1,14 @@ +--- +pr: 374 +url: https://github.com/llm-d/llm-d-async/pull/374 +author: shimib +date: 2026-07-29 +--- + +The `prometheus-budget` gate now logs its resolved closing point when it is +created (`"prometheus-budget gate configured" ... closesAtLoadPerReadyPod=95`), +making a mis-sized `max_concurrency` visible immediately instead of leaving the +gate silently open forever. `max_concurrency` is documented as a **per ready +pod** capacity in the README and the e2e deploy guide, with guidance on deriving +it from the EPP saturation detector's `MaxConcurrency` or from the observed +per-pod peak of `vllm:num_requests_running`.