Skip to content

fix(kv-router): stamp request identity on decode-affinity scoring row - #12955

Merged
nv-yna merged 1 commit into
ai-dynamo:mainfrom
nv-yna:yna/dyn-3804-stamp-decode-affinity-row
Aug 10, 2026
Merged

fix(kv-router): stamp request identity on decode-affinity scoring row#12955
nv-yna merged 1 commit into
ai-dynamo:mainfrom
nv-yna:yna/dyn-3804-stamp-decode-affinity-row

Conversation

@nv-yna

@nv-yna nv-yna commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Summary

DefaultWorkerScorer::worker_logit has three tracing::debug! sites that emit one candidate scoring row each. Two of them carry request_id and worker_type so the candidates of a single routing decision can be grouped by id in one hop. The third — the decode-affinity branch — carries neither, and it returns before reaching the other two.

The result is that any decision taking the decode-affinity branch emits candidate rows without request identity, so grouping them back into a decision requires the line-adjacency heuristic that stamping was added to replace.

This is not occasional. The branch predicate reads the configured overlap_score_credit (default 1.0), not any measured overlap:

if self.worker_type == "decode"
    && !context.track_prefill_tokens
    && weights.overlap_score_credit > 0.0

so on an affected deployment every decision takes the branch and 100% of candidate rows lose their identity fields — even when the measured overlap is zero.

Affected configurations

Deployment worker_type track_prefill_tokens overlap_score_credit Affected
Aggregated + --no-router-track-prefill-tokens decode false (flag) 1.0 (default) Yes — all decisions
Aggregated, default flags decode true 1.0 No
Disaggregated, decode pool decode false (forced) 0.0 (forced) No
Disaggregated, prefill pool prefill true 1.0 No

Aggregated deployments route through worker_type = "decode" unconditionally, so the aggregated case only needs the user-facing --no-router-track-prefill-tokens flag to be affected. The disaggregated decode pool is spared because build_decode_router_override forces overlap_score_credit: Some(0.0), making the third condition false.

Origin

Two PRs landed the same day touching this function. #12370 (15:24 PT) added the identity fields to the two rows that existed then. #11720 (19:39 PT) added this third row from a branch that predated the stamping. git merge-base --is-ancestor confirms #12370 was already in #11720's ancestry, so this is a parallel-development merge miss rather than an intentional change — the two PRs touched disjoint lines, so git saw no conflict.

Fix

Add the two fields so all three candidate rows share one shape. context.request_id and self.worker_type are already in scope in this branch (it already reads self.worker_type and context.track_prefill_tokens in its own predicate), both are borrowed &str with no allocation, and both are evaluated inside the macro so they cost nothing when DEBUG is disabled.

Severity is observability only. Routing behaviour, metrics and API surfaces are unchanged, and these rows are DEBUG-level, so a default INFO deployment sees no change.

Follow-ups not in this PR

  • No in-repo test asserts these fields on selector rows, which is why CI did not catch the original gap. tests/utils/router_logs.py already parses request_id out of router logs, so the assertion has a natural home.
  • The three emit sites duplicate the identity fields by hand, which is what let them drift apart. Factoring the emit into one helper would make the next scoring branch inherit the fields automatically.

Validation

  • cargo check -p dynamo-kv-router — clean
  • cargo clippy -p dynamo-kv-router — no new warnings
  • cargo fmt --all -- --check — clean

Behavioural verification was done on the 1.4.0 line, where the same defect is present, by running an aggregated deployment (dynamo.frontend --router-mode kv + dynamo.mocker --num-workers 3, Qwen3-0.6B) twice with only --no-router-track-prefill-tokens differing:

  • control (flag off): 24 of 24 candidate rows carried both fields
  • subject (flag on): 24 of 24 candidate rows carried neither, and overlap_credit_blocks: 0.000 confirmed the branch fired with no measured overlap

The control arm establishes that the log directives, deployment and field parsers all work, so the subject result is attributable to the branch and not to test setup.

An internal Linear ticket also tracks this (DYN-3804).


Open in Devin Review

Summary by CodeRabbit

  • Bug Fixes
    • Enhanced decode-worker scoring logs with request IDs and worker types for more complete diagnostic information.

The decode-affinity branch in `DefaultWorkerScorer::worker_logit` emits its own
candidate scoring row and returns early, so it never reaches the two rows that
carry `request_id` and `worker_type`. ai-dynamo#12370 stamped those two rows; ai-dynamo#11720
added this third row in parallel and merged without the fields, so any decision
taking this branch emits candidates in the pre-ai-dynamo#12370 shape.

The predicate keys off the configured `overlap_score_credit` (default 1.0)
rather than any measured overlap, so on an affected deployment every decision
takes the branch and every candidate row loses its identity. Grouping the
candidates of one routing decision then falls back to line adjacency, which is
unreliable when decisions interleave.

Add the two fields so all three candidate rows share one shape. Both values are
already in scope in this branch and are evaluated inside the macro, so they cost
nothing when DEBUG is disabled.

Observability only: routing behaviour, metrics and API surfaces are unchanged,
and these rows are DEBUG-level.

Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
@nv-yna
nv-yna requested a review from a team as a code owner August 10, 2026 18:29
@nv-yna
nv-yna temporarily deployed to external_collaborator August 10, 2026 18:29 — with GitHub Actions Inactive
@nv-yna
nv-yna temporarily deployed to external_collaborator August 10, 2026 18:29 — with GitHub Actions Inactive
@github-actions github-actions Bot added the fix label Aug 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

👋 Hi nv-yna! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

@github-actions github-actions Bot added the external-contribution Pull request is from an external contributor label Aug 10, 2026

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

@coderabbitai

coderabbitai Bot commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 695f7b7b-7d9d-4a6f-9aaf-1be7a9a9ed21

📥 Commits

Reviewing files that changed from the base of the PR and between c0b5e55 and e6c76ef.

📒 Files selected for processing (1)
  • lib/kv-router/src/scheduling/selector/default.rs

Walkthrough

The decode-worker scoring debug log now includes request_id and worker_type fields.

Changes

Decode scoring logging

Layer / File(s) Summary
Add decode scoring metadata
lib/kv-router/src/scheduling/selector/default.rs
The decode scoring debug event records request_id and worker_type before the existing formula details.

Estimated code review effort: 1 (Trivial) | ~2 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description gives detailed scope, impact, affected configurations, fix, and validation, but it omits the required Related Issues section and reviewer-start guidance. Add the required Related Issues section and confirm either the linked issue number or that no related issue exists; also identify where reviewers should start.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the primary change: adding request identity to decode-affinity scoring logs.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Comment @coderabbitai help to get the list of available commands.

@datadog-official

datadog-official Bot commented Aug 10, 2026

Copy link
Copy Markdown

Pipelines

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 41.92% (-0.39%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e6c76ef | Docs | Datadog PR Page | Give us feedback!

@nv-yna
nv-yna merged commit cdc9226 into ai-dynamo:main Aug 10, 2026
191 of 193 checks passed
pvijayakrish pushed a commit that referenced this pull request Aug 10, 2026
… scoring row onto release/1.4.0 (#12955) (#12956)

Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
Co-authored-by: Yuewei Na <nv-yna@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contribution Pull request is from an external contributor fix size/XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants