docs: explain and surface the prometheus-budget closing point

README.md

@@ -291,10 +291,44 @@ For more fine-grained control, configure gates per queue in your configuration f
     `inference_pool_ready_pods{name="<pool>"}` (EPP metric) and, for the vLLM fallback,
     the `inference_pool` label on scraped vLLM metrics (added via relabeling from pod labels).
   - `namespace` (optional): Kubernetes namespace to scope metric queries. Required when multiple namespaces share the same pool name with a shared Prometheus instance.
-  - `max_concurrency` (optional): Per-endpoint request capacity (`MaxConcurrency` in the [inference scheduler's saturation detector](https://github.com/llm-d/llm-d-inference-scheduler/blob/main/pkg/epp/framework/plugins/flowcontrol/saturationdetector/concurrency/config.go)). Default is `100` (matching the inference scheduler default).
+  - `max_concurrency` (optional): Per-endpoint request capacity (`MaxConcurrency` in the [inference scheduler's saturation detector](https://github.com/llm-d/llm-d-inference-scheduler/blob/main/pkg/epp/framework/plugins/flowcontrol/saturationdetector/concurrency/config.go)). Default is `100` (matching the inference scheduler default). See [sizing `max_concurrency`](#sizing-max_concurrency) below — this is a **per-pod** number, and setting it above what a pod actually serves makes the gate inert.
   - `baseline` (optional): Reserved baseline B. The gate closes when D ≤ B. Default is `0.05`.
   - `fallback` (optional): Fallback budget value (0.0-1.0) returned when all metric sources are unavailable. Default is `0.0` (fail closed).
 
+  <a id="sizing-max_concurrency"></a>
+  **Sizing `max_concurrency`.** Because `max_SYS = ready_pods × max_concurrency`, the gate closes only
+  once the observed load reaches:
+
+  ```
+  ready_pods × max_concurrency × (1 − baseline)
+  ```
+
+  At the defaults that is 95 concurrent requests *per ready pod*. A pool that never gets near
+  that — a large model on a few replicas, for instance — leaves the gate permanently open, so every
+  batch request dispatches regardless of live traffic. The gate logs its resolved closing point at
+  startup so you can compare it against reality:
+
+  ```
+  "prometheus-budget gate configured" pool=... maxConcurrency=100 baseline=0.05 closesAtLoadPerReadyPod=95
+  ```
+
+  Two ways to pick a value:
+
+  - **Match the EPP.** `max_concurrency` mirrors `MaxConcurrency` in the inference scheduler's
+    saturation detector. Using the same number keeps the async gate and the EPP's own admission
+    control in agreement about when the pool is full. If you have not configured the saturation
+    detector, both defaults are `100`.
+  - **Measure it.** Drive your pool to the load you consider saturated and read the per-pod peak:
+
+    ```promql
+    max_over_time(
+      (sum(vllm:num_requests_running{inference_pool="<pool>"}) / on() inference_pool_ready_pods{name="<pool>"})[1h:]
+    )
+    ```
+
+    Set `max_concurrency` to that peak. Values well above it mean the gate never closes; values
+    well below it mean the gate sheds while the pool still has room.
+
   **Metric prerequisites:** The primary metric source requires llm-d's flow control plugin to be
   enabled; without it, the gate falls back to vLLM metrics. The fallback filters by `inference_pool` label,
   which vLLM does not emit natively: configure Prometheus relabeling to propagate it from model server pod labels

docs/guides/e2e-deploy.md

@@ -165,7 +165,9 @@ helm install llm-d-async ${ASYNC_REPO}/charts/llm-d-async/ \
 The values file (`docs/guides/e2e-deploy/llm-d-async-values.yaml`) configures:
 - Image: `ghcr.io/llm-d/llm-d-async:938cd44`
 - Queue: Redis sorted-set with `redis.url` set directly (chart creates the Secret), configured via `queuesConfig`
-- Gate: `prometheus-budget` with pool=`optimized-baseline`, max_concurrency=100, baseline=0.05 (per-queue)
+- Gate: `prometheus-budget` with pool=`optimized-baseline`, max_concurrency=100, baseline=0.05 (per-queue).
+  `max_concurrency` is a **per ready pod** capacity — see [Size `max_concurrency` for your pool](#size-max_concurrency-for-your-pool)
+  before reusing this value on your own model
 - Prometheus URL pointing to the cluster's `llmd-kube-prometheus-stack-prometheus` service
 
 > **Multi-namespace deployments:** If the cluster has multiple inference pools
@@ -192,6 +194,47 @@ The values file (`docs/guides/e2e-deploy/llm-d-async-values.yaml`) configures:
 - `grafana.dashboards.enabled: true` — provisions a Grafana dashboard (via sidecar) with
   request rate, outcome breakdown, success/retry gauges, and latency percentiles
 
+### Size `max_concurrency` for your pool
+
+`max_concurrency` is the request capacity of **one ready pod**, not of the pool. The gate
+computes `max_SYS = ready_pods × max_concurrency` and closes only once the observed load
+reaches:
+
+```
+ready_pods × max_concurrency × (1 - baseline)
+```
+
+With the values above that is `1 × 100 × 0.95` = **95 concurrent requests** against the single
+Qwen3-0.6B replica this guide deploys. That is reachable here — the saturation test below drives
+200 concurrent requests, and 100 also matches the default `MaxConcurrency` of the EPP's saturation
+detector, so the async gate and the EPP agree on when the pool is full.
+
+It is not automatically reachable anywhere else. Point this configuration at a larger model on a
+few replicas and a realistic workload may peak in the single digits per pod, far below the closing
+point — in which case the gate never closes and every batch request dispatches regardless of live
+traffic, which is the exact failure the gate exists to prevent.
+
+The processor logs its resolved closing point when it builds the gate, so check it against reality:
+
+```bash
+kubectl logs -n ${NAMESPACE} -l app.kubernetes.io/name=llm-d-async | grep "prometheus-budget gate configured"
+# "prometheus-budget gate configured" pool=optimized-baseline maxConcurrency=100 baseline=0.05 closesAtLoadPerReadyPod=95
+```
+
+To derive the value for your own model and hardware, drive the pool to the load you consider
+saturated and read the per-pod peak:
+
+```bash
+kubectl run --rm -i prom-peak --image=curlimages/curl --restart=Never -n ${NAMESPACE} -- \
+    curl -s --data-urlencode \
+    'query=max_over_time((sum(vllm:num_requests_running{inference_pool="optimized-baseline"}) / on() inference_pool_ready_pods{name="optimized-baseline"})[1h:])' \
+    'http://llmd-kube-prometheus-stack-prometheus.llm-d-monitoring.svc.cluster.local:9090/api/v1/query'
+```
+
+Set `max_concurrency` to that peak. Well above it and the gate never closes; well below it and the
+gate sheds while the pool still has room. If you change it, update the `* 100` divisor in the
+verification queries below to match.
+
 ## Verify
 
 ### Check pods and gate status

docs/guides/e2e-deploy/llm-d-async-values.yaml

@@ -24,6 +24,14 @@ ap:
         gate_type: "prometheus-budget"
         gate_params:
           pool: "optimized-baseline"
+          # Per *ready pod* request capacity, not a pool-wide total: the gate
+          # closes once load reaches ready_pods * max_concurrency * (1 - baseline),
+          # i.e. 95 concurrent requests on the single Qwen3-0.6B replica this
+          # guide deploys. 100 matches the EPP saturation detector's default and
+          # is reachable by the guide's own load test (hey, 200 workers).
+          # Re-derive it for your own model and hardware -- a value your pool
+          # never reaches leaves the gate permanently open. See the
+          # "Size max_concurrency for your pool" section of the guide.
           max_concurrency: "100"
           baseline: "0.05"
   podMonitor:

pkg/async/inference/flowcontrol/gate_factory.go

@@ -26,6 +26,7 @@ import (
 	redisgate "github.com/llm-d/llm-d-async/pkg/redis"
 	promapi "github.com/prometheus/client_golang/api"
 	goredis "github.com/redis/go-redis/v9"
+	"sigs.k8s.io/controller-runtime/pkg/log"
 )
 
 // DefaultCacheTTL is the default TTL for cached Prometheus metric sources.
@@ -87,6 +88,10 @@ func (f *GateFactory) Close() error {
 //     Gate closes when D ≤ B (baseline); returns D − B when open, so callers compute
 //     N = max_SYS × (D − B). Params: pool (required),
 //     max_concurrency (default 100), baseline (default 0.05), fallback (default 0.0)
+//     max_concurrency is per ready pod, not for the pool as a whole: the gate closes
+//     once the observed load reaches max_concurrency × (1 − baseline) per ready pod.
+//     Set it too high for the pool's real capacity and the gate never closes; the
+//     resolved closing point is logged at gate creation so this is visible.
 //   - "prometheus-query": Evaluates an arbitrary user-supplied PromQL expression as the dispatch
 //     budget. The expression must resolve to an instant vector with a single sample whose value
 //     is in [0, 1]. Unlike prometheus-saturation and prometheus-budget, this gate does not
@@ -280,6 +285,18 @@ func (f *GateFactory) CreateGate(cfg pipeline.GateConfig) (pipeline.Gate, error)
 			cachedSource(primary, f.cacheTTL),
 			cachedSource(secondary, f.cacheTTL),
 		)
+
+		// Report the resolved closing point. max_concurrency is a per-ready-pod
+		// divisor, so a value the pool can never reach leaves the gate
+		// permanently open with nothing in the logs or metrics to say so.
+		// Surfacing it here makes a mis-sized max_concurrency visible at startup.
+		log.Log.WithName("gate-factory").Info("prometheus-budget gate configured",
+			"pool", pool,
+			"maxConcurrency", maxConcurrency,
+			"baseline", baseline,
+			"closesAtLoadPerReadyPod", maxConcurrency*(1-baseline),
+		)
+
 		return NewBudgetDispatchGate(ms, baseline, fallback).
 			WithOwner(cfg.Owner).
 			WithInferencePool(pool), nil

release-notes.d/unreleased/374.md

@@ -0,0 +1,14 @@
+---
+pr: 374
+url: https://github.com/llm-d/llm-d-async/pull/374
+author: shimib
+date: 2026-07-29
+---
+
+The `prometheus-budget` gate now logs its resolved closing point when it is
+created (`"prometheus-budget gate configured" ... closesAtLoadPerReadyPod=95`),
+making a mis-sized `max_concurrency` visible immediately instead of leaving the
+gate silently open forever. `max_concurrency` is documented as a **per ready
+pod** capacity in the README and the e2e deploy guide, with guidance on deriving
+it from the EPP saturation detector's `MaxConcurrency` or from the observed
+per-pod peak of `vllm:num_requests_running`.