Skip to content

The three gate series carry three different pool_name values, so none of them join #369

Description

@shimib

The three gate series carry three different pool_name values, so none of them join

The same running processor, one queue, one pool, emits:

llm_d_async_async_gate_metric_value{pool_name="optimized-baseline"}
llm_d_async_async_gate_metric_threshold{pool_name="optimized-baseline"}
llm_d_async_async_dispatch_budget{pool_name="",queue_id="request-sortedset",...}
llm_d_async_async_broker_backlog{pool_name="default",queue_id="request-sortedset",...}
llm_d_async_async_pool_worker_limit{pool_name="default"}

Three different values of pool_name for one pool. The gate gauges use the gate's own
poolLabel (the InferencePool name), while SetDispatchBudget derives it locally:

// pkg/redis/sortedset_impl.go:425-429
poolName := ""
if cfg, ok := r.configMap[queueID]; ok {
    poolName = cfg.WorkerPoolID          // empty in a stock single-pool install
}
metrics.SetDispatchBudget(budget, queueID, queueName, poolName)

Impact: no PromQL can join the gate's decision (dispatch_budget) to the inputs that produced
it (gate_metric_value / gate_metric_threshold) on pool_name, which is the obvious way to
build a "why is the gate closed?" panel. You have to know to drop the label and join on
queue_name, which the gate gauges do not carry. Cosmetic in a single-pool cluster; it stops
being cosmetic with more than one pool, where the empty label makes dispatch_budget
unattributable.

Suggested fix: give all five series the pool label from one source — the gate already knows
it. Falling back to "default" rather than "" when WorkerPoolID is unset would at least match
the sibling gauges on the same code path.

Evidence: curl against the processor's :9090/metrics, this run.


Found while standing up an agentic-workload test environment over llm-d-async on GKE (llm-d-async main, llm-d main + PR #2099). Repro scripts and the full findings ledger live outside this repo; happy to attach them. Finding ID F-018.

Metadata

Metadata

Assignees

Labels

documentationImprovements or additions to documentation

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions