feat(router): add worker cache-reuse telemetry (F2-F4)

[chanh]

Summary

Adds always-on cache-reuse counters for each KV-routing attempt: raw cached prefix on the best eligible worker (F2), raw cached prefix on the selected worker/rank (F3), and worker-reported reused tokens (F4, formerly F5). Routing decisions and tier weights are unchanged.

Start review in request_guard.rs, metrics.rs, and the vLLM generation/prefill response paths in handlers.py.

Metrics and compatibility

All names below have the dynamo_component_router_ prefix and carry phase (aggregated, prefill, or decode) and model labels.

Measurement	Metric
Received input tokens (F0)	input_sequence_tokens_sum (existing histogram sum)
Best eligible cached prefix (F2)	kv_best_eligible_cached_prefix_tokens_total
Selected worker/rank cached prefix (F3)	kv_selected_cached_prefix_tokens_total
Worker-reported reused tokens (F4)	kv_worker_reused_tokens_total

• Router estimates are recorded at selection. Reuse is counted once on receipt of the first valid, prompt-matched worker report, including zero. A later error or cancellation does not undo it.
• Missing or invalid reports contribute zero. The reuse series is initialized at selection even for backends without a producer. No outcome or lookup metric is added.
• The optional engine_data.kv_cache_hit object contains only prompt_tokens and reused_tokens. Unknown fields are accepted, so optional tier/lookup data can be added later. The payload is new in this PR; no shipped wire key changes.
• Both vLLM generation and dedicated prefill responses report RequestOutput.num_cached_tokens. We retain a separate per-attempt report because disaggregated decode overwrites completion usage with prefill usage for client accounting.
• Older workers contribute zero reported reuse; older frontends ignore the new payload. SGLang and TensorRT-LLM producers are not added here.
• F0's histogram excludes query-only selections and gains matching phase/model labels. Process-wide queries must sum across those labels; the histogram still includes non-KV traffic.
• F2–F4 need no enable flag. F1 history remains separate and opt-in in feat(router): add opt-in F1 cache reuse history #14889. Lookup telemetry is deferred until a backend can supply it.

Interpretation

• Compare within the same model and phase. Prefill/decode are separate attempts; migration adds attempts that can include already-generated tokens.
• The estimates are raw token counts, unlike tier-weighted router_kv_hit_rate. Values are uncapped.
• Multimodal routing tokens can differ from input token IDs counted by F0; mismatched worker prompt lengths contribute zero.
• Worker reuse means reported reuse, not independently verified successful reuse. Accurate backend reporting is required; backend counts may retain initial matches after failed transfers.
• Selection and reporting occur at different times, so short windows can cover different attempts.
• Frontend cached-token metrics and their estimate fallback are unchanged.

Related issues

F1 history: #14889. Incorporates the request-metrics structure from Neelay's reference draft #15173.

Validation

• 13 focused Rust tests pass on this revision: report parsing/extensions, immediate once-only guard accounting, cancellation, zero-series export, registration, and phase/model labels.
• Clippy passes for dynamo-llm, all targets, with --no-default-features -- -D warnings.
• Nine AST-isolated Python source checks pass using the actual vLLM 0.29.0 RequestOutput class: five payload cases, three dedicated-prefill cases, and one generation/KV-transfer case. Surrounding backend dependencies are mocked; the full vLLM pytest suite was not run locally. GPU integration checks are described below.
• Rust formatting, Python syntax, pinned Black/isort/Ruff, codespell, generated metric constants, and git diff --check pass. Direct Flake8 passes with existing W503 findings excluded. The pre-commit wrapper could not fetch its isort environment because GitHub git transport returned HTTP 503.
• Docs lint and Fern configuration pass. Fern broken-links still reports the same unrelated DeepSeek recipe link as the preceding revision.
• Upstream Pre Merge passes on 51b78ad5b5, including Rust tests, Clippy, pre-commit, and docs checks. This is not a claim that the full backend/GPU matrix ran.
• Additional GPU integration validation used this head combined with the separate F1 history PR (feat(router): add opt-in F1 cache reuse history #14889), not the standalone PR tree: a two-worker vLLM smoke test completed eight requests, with received tokens = 7,528 and each of history/best-eligible/selected/reported-reuse = 6,496.
• A separate two-worker shadow deployment exercised mirrored traffic and the end-to-end dashboard. Its older vLLM worker required a compatibility backport, so this is integration evidence, not exact-tree upstream backend coverage. Synthetic repeated-prefix requests reported 4,352 cached tokens out of 6,330 prompt tokens on both warm requests. No production payloads are included here.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a0f5d0-27d3-7061-93ea-bd5b40484c3f.docs.buildwithfern.com/dynamo/dev

[coderabbitai]

Walkthrough

The KV router now propagates eligible-worker cache counts, tracks bounded canonical cache history, records cache-loss metrics through request lifecycles, and attaches validated cache counters to final vLLM generation chunks.

Changes

KV cache-loss accounting

Layer / File(s)	Summary
Propagate eligible-worker cache counts lib/kv-router/src/..., lib/llm/src/kv_router.rs, lib/llm/src/kv_router/routing_host/...	Scheduling and routing results now include max_cached_tokens for admitted and advisory selections.
Bounded cache-history model lib/llm/src/kv_router/minimal_cache_loss.rs	The router stores bounded canonical sequence hashes, performs prefix lookup, tracks generated outputs, and records route observations.
Initialize history and metrics lib/llm/src/kv_router/metrics.rs, lib/llm/src/kv_router/routing_host.rs	KV hosts initialize shared cache history and publish cache-loss counters and retention gauges.
Track request outcomes and finalization lib/llm/src/kv_router/routing_host/kv.rs, lib/llm/src/kv_router/routing_host/request_guard.rs, lib/llm/src/kv_router/routing_host/tests.rs	Request guards collect streamed tokens, validate worker outcomes, record metrics, and finalize history on completion and cleanup paths.
Attach vLLM cache statistics components/src/dynamo/vllm/handlers.py, components/src/dynamo/vllm/tests/test_vllm_prompt_embeds.py	Final generation chunks include complete cache counters when available and otherwise mark cache-loss data incomplete.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🟡 Moderate · up to a66ca

The PR adds bounded cache-loss telemetry, but the current implementation still performs the feature’s memory and hashing work by default and serializes prompt hashing on a shared routing lock; incomplete worker payloads and capacity-limited multi-output history can also produce missing or nondeterministic telemetry. These are concrete runtime and correctness issues, so merge should wait for fixes or explicit owner acceptance.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 24.19% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 62 functions across 16 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the router change and the added worker cache-reuse telemetry. It is concise and aligned with the primary F2-F4 changes, although it does not mention the broader cache-loss…
Description check	✅ Passed	The description is detailed and covers the change scope, metrics, compatibility, interpretation, review starting points, related references, and validation results. The Related Issues section does not…

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[nnshah1]

Two labels these series need: phase and model. RouterRequestMetrics::from_component creates one shared instance per process (a static OnceLock), and router_id is the DRT instance_id of whichever component initialised first. Routing hosts are created per model and per pool, but they all share that one instance — so a frontend hosting the prefill and decode routers, or several models, records all of their attempts into the same series. The catalog's "compare within a pool" is therefore only true across processes today.

• phase = aggregated | prefill | decode, from tracker.phase() — the same label and values the frontend already exports on stage_duration_seconds{stage,phase}. Not worker_type: an aggregated router reports "decode" (routing_load.rs:36), which would mislabel a prefill-reuse metric in the most common deployment, and widening that vocabulary changes what existing per-worker gauges mean.
• model, from PreprocessedRequest.model — a closed set (the served models), with router-side precedent in the queue-pending and selection-policy gauges.

Number of series: × phases present (1 aggregated, 2 disaggregated) × served models — the same shape as the existing overlap_blocks_lost{worker_type}. @jh-nv — flagging for a series-count check.

— Neelay + 🤖

[nnshah1]

Suggestion for a follow-up (or here if cheap): record F4/F5 in all routing modes, not just KV. F2/F3 genuinely need the KV index — non-KV modes never hash the prompt or query it, so "best eligible cached prefix" has no meaning there. But F4/F5 come from the worker and don't depend on how it was chosen; the guard sees the final chunk in every mode, yet today RoundRobin, Random, PowerOfTwoChoices, LeastLoaded and the others build guards with no route estimate, so nothing is recorded. Separating the two is about 15 lines (prompt length and a tracked flag on the builtin guard; F2/F3 simply absent rather than zero).

Why it's worth it: F5 under round-robin versus F5 under KV routing is the direct measurement of what KV routing buys — the most useful comparison this family could enable, and impossible today. It changes F5's population relative to F3's, so raising it rather than asking for it. @jh-nv

— Neelay + 🤖

[nnshah1]

Filed as separate issues so they don't grow this PR: SGLang per-request cache-hit report (#15174), TRT-LLM per-request cache-hit report (#15175), and per-model / per-pool separation for the other router per-attempt metrics (#15176).

— Neelay + 🤖

[chanh]

In reply to your comment:

• Keeping counters — in fae8561. The funnel compares total tokens across stages; it does not need per-request distributions or percentiles such as p95. Counters provide what we need with fewer time series and less recording overhead.
• Accepted the structural suggestion: the counters now live directly on RouterRequestMetrics; histogram support is not needed for that simplification.

[chanh]

In reply to your comment:

• Accepted — addressed in fae8561. Added phase (aggregated, prefill, decode) and model to F2–F5 and the worker-outcome counter.
• Also applied to F0: the existing input-sequence histogram gets the same labels so the denominator can be scoped consistently. It still includes non-KV routing, so comparisons must use matching KV-routing traffic.
• Phase stays consistent: the guard captures the attempt's original phase and retains it through completion.

[chanh]

In reply to your comment:

• Agreed — separate follow-up. Keeping F4/F5 limited to KV-routing attempts in this PR. Reporting them for other routing modes would enable a useful comparison, but changes their population relative to F2/F3 and deserves its own change.

[chanh]

In reply to your comment:

• Agreed — keeping these as separate follow-ups. Thanks for filing feat(sglang): report per-request KV-cache hit tokens for the router's reuse metrics #15174, feat(trtllm): report per-request KV-cache hit tokens for the router's reuse metrics #15175, and fix(router): per-model and per-pool separation for the router's per-attempt metrics #15176. This PR documents the current backend coverage and scopes the new cache metrics plus their input-token denominator by phase/model; it does not expand the other router metrics.

[chanh]

/ok to test 519c001

[nnshah1]

Full pass over the current head (57cef28e3) — built it, ran clippy with -D warnings and the PR's Rust + Python tests locally (all 23 pass; clippy fails only on new_kv). Substantive points are inline; the small items are grouped here as one thread.

[before merge — names and tests; the rest are suggestions]

Tests

• metrics.rs:1680-1682 — assert!(!output.contains("cache_loss")), "funnel", "stage=" assert the absence of strings that don't appear anywhere in the code. Delete.
• request_guard.rs:1041-1042 — the comment on counter_overflow_marks_the_observation_incomplete describes revision history; git has it. Delete.
• kv_cache_hit_metrics_export_counters_with_matching_input_labels is two tests (kv counters; input_sequence_tokens labels) in one 50-line body — split or trim.
• 22 tests for ~250 lines of logic. Overlaps: worker_outcomes_can_exceed_router_observations and worker_values_may_exceed_router_estimate_and_prompt_length are the same assertion with different numbers; successful_stream_emits_the_latched_hit / stream_error_discards_a_buffered_hit restate what the e2e complete / cancelled tests prove through the real guard; raw_overlap_is_specific_to_the_selected_dp_rank tests three-map addition. Suggested keep set: parse rejection, overflow, the three e2e attempt tests, completion_keeps_selection_phase, one selector max test, and test_kv_cache_hit_engine_data_marks_missing_worker_counters_incomplete.

Naming

• "worker-stage" is the old name for what is now the kv cache-hit report: protocols.rs:1093, types.rs:156, new_with_worker_stage_tracking / track_worker_stages (selector/mod.rs:158-162), mod worker_stage_tests and three test names.
• new_with_worker_stage_tracking exists for one test (mod.rs:672); that test can set request.mode = ScheduleMode::Tracked { .. } like the test above it. Delete the constructor.
• selector/mod.rs:141-145 — the comment says "preserves F2", which is defined nowhere in the code. One line does it: "max over the same eligible set row() already visits; no second traversal."
• KV_PHASE_LABEL's doc comment says it matches the stage metrics' label; those use a literal "phase". Either promote to prometheus_names::labels::PHASE and use it in both places, or drop the comment.

Shapes

• observe_kv_worker_hit(&self, phase, model, (lookup, reused): (Option<u64>, u64)) destructures a tuple in the parameter list, and kv_worker_hit_for returns Option<(Option<u64>, u64)>. Pass and return WorkerReport — the enum exists for exactly this. (Moot for the inner Option if the lookup deferral is taken.)
• kv_tokens_counter uses unwrap_or_else(|_| panic!(..)); every sibling uses .expect(..).
• kv.rs:416-428 — if !is_query_only { ..zip..map.. } else { None } → (!is_query_only).then(|| ..).flatten(), or gate kv_route once where is_query_only is decided.

— Neelay + 🤖

[nnshah1]

[blocking] mergeable_state is dirty: lib/kv-router/src/scheduling/selector/mod.rs conflicts with main's f804218070 (#14785, "move default worker selection into builtin plugin"), which landed after your Merge upstream main (57cef28e3). Two hunks: MaterializedSelectionInput now has a cache_snapshot: CacheSnapshot<'a> field and new(request, block_size) lost the weights parameter (~:134-165), and the #[inline(always)] on row() collides with the max_raw_cached_tokens() accessor (~:189-196). Reproduce locally with git merge-tree --write-tree origin/main HEAD.

The Cell<Option<usize>> accumulator itself is unaffected — it just needs re-homing on the new struct shape. If the selected_raw_cached_tokens → selection_result() suggestion is taken, it's worth doing both in the same conflict-resolution pass since they touch the same lines.

Separately, CI on this head is red only on rust-clippy (too_many_arguments on new_kv, see the inline comment); everything else in Pre Merge is green.

— Neelay + 🤖

[chanh]

In reply to the grouped review, addressed in e0cc7a9:

• Removed obsolete negative assertions, revision-history comments, duplicate helper tests, and the one-test selector constructor; replaced worker-stage naming and the undefined F2 comment.
• Split metric export/input-label checks. Kept the overflow, missing-versus-zero, uncapped-value, real guard lifecycle, and selected-DP-rank coverage because those pin distinct behavior.
• Pass WorkerReport directly rather than nested tuples, use expect for registration, and gate query-only route tracking once.
• The actual worker adapter now uses stock vLLM fields only; the separate lookup thread explains the remaining scope choice.

Also addressed the merge-conflict comment: merged main at 4948984 and preserved its CacheSnapshot/builtin-policy refactor while retaining raw-token telemetry.

Validation: 22 focused Rust tests and five AST-isolated Python cases pass; clippy passes for both affected Rust crates with all targets and --no-default-features. Formatting, docs lint, and Fern configuration pass. Fern broken-links still reports the unrelated recipe link already present on main. Full GPU/backend CI needs a maintainer to approve this head.

[nnshah1]

Approving on e0cc7a94d. Both review rounds are addressed: kv_* naming with phase/model labels, local_/external_* wire keys, Option<KvHitTracking> on the guard, selected_raw_cached_tokens produced by selection_result, one shared metrics build path, PreprocessedRequest::phase(), and the vLLM adapter reading only RequestOutput.num_cached_tokens with the catalog stating no stock backend populates the lookup series yet. Merge conflict with #14785 resolved; clippy and pre-merge are green on this head.

— Neelay + 🤖

[chanh]

Ready for another review of 51b78ad5b5.

@ai-dynamo/dynamo-backend-vllm-codeowners @ai-dynamo/dynamo-rl-codeowners @ai-dynamo/dynamo-docs-codeowners — GitHub still lists these codeowner review requests as outstanding. Could you review the current head?

Since that approval: deferred lookup/tier/outcome telemetry; reduced the worker payload to prompt_tokens/reused_tokens; record reported reuse once when received (including zero), with missing data contributing zero; added dedicated-prefill reporting. F4 now means reported reuse. Review-thread replies and the description reflect this scope.

Current-head Pre Merge passes, including Rust tests and Clippy. The description now also records the two-worker GPU smoke and shadow/dashboard checks, explicitly distinguishing the combined F1 integration build and compatibility backport from standalone PR validation. Full upstream backend/GPU coverage remains separate.

The previous approval was on e0cc7a94d5; please review 51b78ad5b5 rather than relying on that approval.

[chanh]

Updated the funnel docs in b5212f5: a plain-English F0–F4 table with metric names, enablement, and the longer-window timing note. Removed implementation details from the page; the PR description retains the review context. F1 uses the renamed metric in #14889. Docs lint and Fern configuration pass; the local broken-links check still reports the unrelated DeepSeek recipe link.

[chanh]

Refreshed against main in a5fe0fe and resolved the scheduler/request-guard conflicts. Preserved the new projected-load and output-block tracking paths, plus the existing F2–F4 semantics and simplified docs. Formatting and diff checks pass locally; full CI is running: https://github.com/ai-dynamo/dynamo/actions/runs/36813596696

[chanh]

Fixed the full-CI mypy failure in 465ce8f: explicitly typed the heterogeneous response dictionary so the nested cache-reuse report is accepted. This is a one-line typing-only change. The isolated mypy reproduction passes; fresh full CI: https://github.com/ai-dynamo/dynamo/actions/runs/36816830163

[devin-ai-integration]

✅ Dynamo PR CI passed — run 36816830163 (attempt 1) on 465ce8f60b

✅ 52 passed · ❌ 0 failed · ⏹️ 0 cancelled · ⏭️ 26 skipped (not needed for this change)

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Each cell counts that group's GitHub Actions jobs by result (✅ passed, ❌ failed, ⏰ timed out, ⏹️ cancelled); ⏭️ = all skipped, ➖ = no such job.

Framework	Build	1-GPU amd64	1-GPU arm64	Multi-GPU amd64	Deploy	Snapshot
vLLM	✅ 6	✅ 1	✅ 1	✅ 1	✅ 4	✅ 4
SGLang	✅ 3	✅ 1	✅ 1	✅ 1	⏭️	⏭️
TRT-LLM	✅ 3	✅ 1	✅ 1	✅ 1	⏭️	⏭️

Other	Jobs
changed-files	✅ 1
deploy-operator	✅ 1
dynamo-runtime	✅ 8
frontend	✅ 3
frontend (amd64)	✅ 1
frontend (arm64)	✅ 1
Operator	✅ 1
planner	✅ 5
triton-runtime	✅ 2

⏭️ 7 other components not run (skipped by change detection or an upstream result)

allure-report, DGDR Deploy Test, dynamo-sidecar, Helm Chart Tests, Operator Integration, Power Agent, sidecar-runtime

_{Posted automatically by Devin for run 36816830163. Updated on every full-CI run of this PR.}

[PeaBrane]

Superseded by #15443, which carries this PR's two commits unchanged (authorship preserved via cherry-pick -x) plus SGLang and TensorRT-LLM producers, and drops the report from disaggregated decode workers, whose engines count transferred prefill KV as cached. Please hold this one so the two don't merge separately. Review feedback for the router consumer is welcome on #15443.

[chanh]

Closing as superseded by #15443, which incorporates this PR’s router cache-reuse telemetry with authorship preserved and consolidates the backend reporting changes. Further review and integration will continue there. The replacement PR is still open; this closure does not indicate it has merged.