The three gate series carry three different pool_name values, so none of them join
The same running processor, one queue, one pool, emits:
llm_d_async_async_gate_metric_value{pool_name="optimized-baseline"}
llm_d_async_async_gate_metric_threshold{pool_name="optimized-baseline"}
llm_d_async_async_dispatch_budget{pool_name="",queue_id="request-sortedset",...}
llm_d_async_async_broker_backlog{pool_name="default",queue_id="request-sortedset",...}
llm_d_async_async_pool_worker_limit{pool_name="default"}
Three different values of pool_name for one pool. The gate gauges use the gate's own
poolLabel (the InferencePool name), while SetDispatchBudget derives it locally:
// pkg/redis/sortedset_impl.go:425-429
poolName := ""
if cfg, ok := r.configMap[queueID]; ok {
poolName = cfg.WorkerPoolID // empty in a stock single-pool install
}
metrics.SetDispatchBudget(budget, queueID, queueName, poolName)
Impact: no PromQL can join the gate's decision (dispatch_budget) to the inputs that produced
it (gate_metric_value / gate_metric_threshold) on pool_name, which is the obvious way to
build a "why is the gate closed?" panel. You have to know to drop the label and join on
queue_name, which the gate gauges do not carry. Cosmetic in a single-pool cluster; it stops
being cosmetic with more than one pool, where the empty label makes dispatch_budget
unattributable.
Suggested fix: give all five series the pool label from one source — the gate already knows
it. Falling back to "default" rather than "" when WorkerPoolID is unset would at least match
the sibling gauges on the same code path.
Evidence: curl against the processor's :9090/metrics, this run.
Found while standing up an agentic-workload test environment over llm-d-async on GKE (llm-d-async main, llm-d main + PR #2099). Repro scripts and the full findings ledger live outside this repo; happy to attach them. Finding ID F-018.
The three gate series carry three different
pool_namevalues, so none of them joinThe same running processor, one queue, one pool, emits:
Three different values of
pool_namefor one pool. The gate gauges use the gate's ownpoolLabel(the InferencePool name), whileSetDispatchBudgetderives it locally:Impact: no PromQL can join the gate's decision (
dispatch_budget) to the inputs that producedit (
gate_metric_value/gate_metric_threshold) onpool_name, which is the obvious way tobuild a "why is the gate closed?" panel. You have to know to drop the label and join on
queue_name, which the gate gauges do not carry. Cosmetic in a single-pool cluster; it stopsbeing cosmetic with more than one pool, where the empty label makes
dispatch_budgetunattributable.
Suggested fix: give all five series the pool label from one source — the gate already knows
it. Falling back to
"default"rather than""whenWorkerPoolIDis unset would at least matchthe sibling gauges on the same code path.
Evidence:
curlagainst the processor's:9090/metrics, this run.Found while standing up an agentic-workload test environment over
llm-d-asyncon GKE (llm-d-asyncmain, llm-dmain+ PR #2099). Repro scripts and the full findings ledger live outside this repo; happy to attach them. Finding IDF-018.