feat(kv-router): stamp request identity on selector scoring rows - #12370
Merged
Conversation
nv-yna
temporarily deployed
to
external_collaborator
July 29, 2026 17:50 — with
GitHub Actions
Inactive
nv-yna
temporarily deployed
to
external_collaborator
July 29, 2026 17:50 — with
GitHub Actions
Inactive
Contributor
Contributor
WalkthroughChangesSelector logging correlation
Estimated code review effort: 2 (Simple) | ~10 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
|
🎯 Code Coverage (details) 🔗 Commit SHA: f19f5d3 | Docs | Datadog PR Page | Give us feedback! |
selector.rs emits one `tracing::debug!` per candidate carrying the whole cost breakdown, but the row has no identity, so offline tooling can only group candidates into decisions by line adjacency. Adjacency is not sound here. These rows are emitted from the `SchedulerQueueActor` task, which scheduling/queue.rs:409 spawns via `tokio::spawn(actor.run(admission_rx))` with no `.instrument(...)`, so the caller's request span is not in scope and the logging layer cannot attach `x_request_id`/`trace_id`. That is why `[ROUTING] Best`, emitted on the caller task at llm/src/kv_router/push_router/selection.rs:183, carries `x_request_id` while these rows do not. There is also one actor per scheduler, and a disaggregated deployment runs two, so prefill-pool and decode-pool decisions interleave into the same log. Measured on a 1.24 GB frontend-log sample from a 6 prefill + 1 decode deployment: candidate-to-winner agreement 72.46% overall, 99.82% on 24-candidate prefill decisions but 42.98% on 8-candidate decode decisions, and 14.40% of winners with no preceding candidate rows. Decode-pool analysis is not currently possible from these logs. `request.mode.request_id()` is already reachable at the emission site and returns the same value `[ROUTING] Best` logs as `request_id`, so it makes the row self-joining to the client record in one hop, with no scheduler state and no new identifier scheme. `worker_type` separates the two pools without a join, which matters because they score differently: the decode path runs with `overlap_score_credit = 0` and `track_prefill_tokens = false` (prefill_router/mod.rs build_decode_router_override). Both fields are evaluated inside the macro, so they cost nothing when DEBUG is disabled. When enabled, `request_id` is a borrowed `&str` and `worker_type` a `&'static str`; nothing is allocated in selector.rs. No scoring behaviour changes, and no function signatures change. Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
nv-yna
force-pushed
the
yna/kv-router-decision-id
branch
from
July 29, 2026 19:04
ebb8bfd to
f92280d
Compare
nv-yna
temporarily deployed
to
external_collaborator
July 29, 2026 19:04 — with
GitHub Actions
Inactive
Contributor
|
/ok to test f92280d |
Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
nv-yna
temporarily deployed
to
external_collaborator
July 29, 2026 20:38 — with
GitHub Actions
Inactive
Contributor
|
/ok to test f19f5d3 |
jthomson04
approved these changes
Jul 29, 2026
nv-yna
enabled auto-merge (squash)
July 29, 2026 21:22
jh-nv
approved these changes
Jul 29, 2026
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview:
selector.rsemits onetracing::debug!per candidate carrying the whole cost breakdown —overlap_blocks,raw_prefill_blocks,overlap_credit_blocks,overlap_credit_decay,prefill_load_scale,adjusted_prefill_blocks,decode_blocks, and the resulting logit. It is the only place the router explains why it picked a worker. But the row carries no identity, so anything reading these logs can only group candidates into decisions by line adjacency.This PR stamps
request_idandworker_typeon those rows, andrequest_idon the winner rows, making them self-joining. The selector change is +15/−0 and logging-only. The branch also repairs two pre-existing README links to the disaggregated-serving architecture page; their old redirect resolves to HTTP 404 and was failing the repository-wide docs-link check.Details:
Adjacency is not sound here, for two structural reasons:
SchedulerQueueActortask, whichscheduling/queue.rs:409spawns viatokio::spawn(actor.run(admission_rx))with no.instrument(...). The caller's request span is therefore not in scope, and the logging layer cannot attachx_request_id/trace_id. This is exactly why[ROUTING] Best— emitted on the caller's task atllm/src/kv_router/push_router/selection.rs:183— does carryx_request_idwhile these rows do not.Measured on a 1.24 GB frontend-log sample from a 6-prefill + 1-decode deployment:
Decode-pool analysis is not currently possible from these logs, and the smaller a pool's candidate set, the more adjacency corrupts it.
The two fields:
request_id—request.mode.request_id()is already reachable at the emission site and returns the same value[ROUTING] Bestlogs. That makes the candidate row self-joining to the client record in one hop, rather than requiring a bridge through the winner row (which is absent for 14.40% of decisions).worker_type— separates the two pools with no join at all. This matters beyond bookkeeping: the pools score differently.prefill_router/mod.rs'sbuild_decode_router_overrideforcesoverlap_score_credit = 0.0andtrack_prefill_tokens = falseon the decode path, so decode candidates are load-only and haveraw_prefill_blocks == 0by construction. Aggregating the two pools mixes two different scoring regimes.Alternatives considered:
router_replica_sync), so aggregated logs from two frontends would share ids. It also puts mutable global state in a componentscheduling/CLAUDE.mddeclares side-effect free.AdmissionCommand::Enqueue. This is the framework-correct fix — it would givex_request_id,trace_id,span_idandparent_idon every event inside selection, permanently. It is a larger, cross-crate change to the actor message type, so it belongs in its own PR. This one is the minimal change that makes the existing rows usable.Cost: both fields are evaluated inside the macro, so they cost nothing when DEBUG is disabled. When enabled,
request_idis a borrowed&strandworker_typea&'static str— nothing is allocated inselector.rs.No scoring behaviour changes, and no function signatures change.
Where should the reviewer start?
lib/kv-router/src/scheduling/selector.rscontains the code change;README.mdcontains only the two URL corrections described above. Start with two places inselector.rs:worker_logit(the twotracing::debug!candidate rows) — confirm the added fields are inside the macro invocation, so they are not evaluated when DEBUG is off, and thatrequest.mode.request_id()introduces no allocation.select_worker(the threetracing::info!winner rows: pinned, decode, general) —request_idis bound once and reused; confirm it matches what[ROUTING] Bestlogs so the two rows join.Worth a skeptical eye: the
unwrap_or("-")fallback.ScheduleMode::QueryOnlypermits aNonerequest id (prefill_router/query.rspassesNone), so those rows are deliberately labelled rather than dropped — an explicit sentinel beats today's silent misattribution by adjacency, but say so if you'd prefer a different sentinel or an omitted field.Related Issues
🚫 This PR is NOT linked to an issue:
Validation
cargo test -p dynamo-kv-router --lib— 849 passed, 0 failed, 2 ignoredcargo clippy -p dynamo-kv-router --no-deps --all-targets -- -D warnings— cleancargo fmt -p dynamo-kv-router— appliedv0.24.2with the workflow flags againstREADME.md— 102 links checked, 0 errorspre-commit run --all-files— all hooks passed