fix(flowcontrol): add a stock-EPP metric source to the budget cascade

[shimib]

Fixes #361

Problem

The prometheus-budget gate's primary metric source queries inference_extension_flow_control_queue_size. That metric only exists when EPP runs the flow control plugin — and the shipped llm-d-router-endpoint-picker does not enable it and offers no flag to turn it on. On a documented llm-d install, source 0 never resolves.

The cascade then falls straight through to the vLLM source, which filters on an inference_pool label vLLM does not emit natively; it requires model server pods to carry the label plus Prometheus metric relabeling to propagate it. If that is not configured either, every gate evaluation silently returns the configured fallback budget — indistinguishable from a real reading of the same number.

Change

Insert a middle tier built on a metric a stock EPP always exports:

1 - (avg by(name)(inference_pool_per_pod_queue_size{name="my-pool"}) / max_concurrency)

Cascade order is now:

#	Metric	Budget	Available when
0	inference_extension_flow_control_queue_size	D = 1 − (queue_size / max_SYS)	EPP runs the flow control plugin
1	inference_pool_per_pod_queue_size	D = 1 − (mean per-pod queue depth / max_concurrency)	Always — part of EPP's base metric set
2	vllm:num_requests_running	D = 1 − (running_requests / max_SYS)	vLLM metrics carry an inference_pool label

Flow control stays at index 0, so installs that do enable the plugin are unaffected.

Why this metric, and why avg

Averaging over pods is what makes max_SYS = ready_pods × max_concurrency reduce to max_concurrency, so no inference_pool_ready_pods join is needed. That is not just brevity: EPP's metrics refresh returns early when the pool has no pods, freezing inference_pool_ready_pods and inference_pool_average_queue_size at their last values — a drained pool would read as idle capacity and open the gate onto nothing. inference_pool_per_pod_queue_size comes from a scrape-time collector instead, so it stops reporting entirely; the query yields no samples and the cascade moves on.

avg is sum/count, so the result is invariant to several EPP replicas reporting the same pods. EPP labels its inference_pool_* series with name, not inference_pool.

Observability

So this failure mode is visible next time rather than silent:

• Resolved PromQL is logged at startup for every Prometheus gate (prometheus-budget metric source, one line per cascade tier, and prometheus-saturation metric source). Queries are built from gate_params, so previously the only way to see what was actually asked of Prometheus was to read the source. prometheus-query is deliberately not logged — its query is user-supplied verbatim.
• CascadeMetricSource starts at index -1 and logs the first source that resolves (metric source resolved). With a zero start, a cascade served by the primary from the outset looked like "no change" and never reported which source the budget came from.
• New llm_d_async_async_gate_metric_source_available gauge — 1 when the last evaluation got a usable reading, 0 when it fell back (query error, no samples, NaN/Inf). This separates a fallback budget from a genuine reading of the same number; most importantly, a fallback of 0 from a truly saturated pool.

Readiness gating (holding pod readiness until a source returns a sample) was considered and deliberately left out of scope.

Tests

• TestPoolQueueSizePromQL — exact expression, required params, namespace, and an assertion that the expression depends on neither of the two gauges that freeze at zero pods.
• TestGateFactory_BudgetGateCascadeSources — the cascade is three sources in the documented order.
• TestGateFactory_BudgetGateLogsResolvedQueries / TestGateFactory_SaturationGateLogsResolvedQuery — startup logging, via funcr.
• TestMetricDispatchGate_RecordsMetricSourceAvailable / ..._MetricSourceAvailableFlipsBack — the new gauge across usable/error/empty/NaN, and recovery.
• TestCascadeMetricSource_UsesPrimary extended, plus TestCascadeMetricSource_AllUnavailableLeavesIndexUnresolved.

E2E blast radius: test/e2e/helm/budget.yaml sets max_concurrency: "1". In the cascade Describe (EPP without flow control, sim at 0 waiting) source 1 yields D = 1 - 0/1 = 1, so the existing assertion still holds; in the second Describe EPP runs with flow control, so source 0 always answers and source 1 is never consulted. Only comments and one test name needed updating.

Docs updated: README.md, docs/dispatch-budget.md, docs/guides/e2e-deploy.md (verification steps now cover source 1 and the new gauge).