feat(router): expose host/disk cache hit weights via CLI and env by Change72 · Pull Request #10157 · ai-dynamo/dynamo

Change72 · 2026-05-31T05:43:36Z

Overview:

Expose host_cache_hit_weight and disk_cache_hit_weight through the
dynamo.frontend / dynamo.router CLI and matching DYN_ROUTER_* env vars.

These fields already exist on the pyo3 KvRouterConfig binding and are used
by the selector when scoring cached blocks. The missing piece is the
operator-facing CLI/env surface: today deployments can tune
--router-kv-overlap-score-credit and --router-prefill-load-scale, but the
lower-tier cache-hit weights are only reachable by constructing
KvRouterConfig from Python directly.

This is strictly additive. The new defaults match the existing binding
defaults (0.75 for host, 0.25 for disk), so behavior is unchanged unless an
operator opts in via the new flags or env vars.

Details:

New KvRouterArgGroup options:

flag	env var	default
`--router-host-cache-hit-weight`	`DYN_ROUTER_HOST_CACHE_HIT_WEIGHT`	`0.75`
`--router-disk-cache-hit-weight`	`DYN_ROUTER_DISK_CACHE_HIT_WEIGHT`	`0.25`

Both fields are added to _KV_ROUTER_FIELDS and KvRouterConfigBase, so
kv_router_kwargs() forwards them into KvRouterConfig(**kwargs) through the
same path used by prefill_load_scale. The load_aware preset preserves both
values.

The practical use case is lower-tier KV cache exposed by a KV connector, such
as vLLM CPU offload or a disk-backed tier. These weights control whether a
worker with a full host/disk-tier prefix match should beat another worker with
a smaller device-local match.

Test coverage (added in test_kv_router_args.py):

CLI, env var, and default parsing for both new weights.
kv_router_kwargs() forwarding for both fields.
--load-aware preservation of both weights.

Docs (docs/components/router/router-configuration.md):

New flag descriptions in the Routing Behavior list.
Tuning-guidelines paragraph for lower-tier weights.

Validation:

pytest components/src/dynamo/common/tests/configuration/test_kv_router_args.py
passes locally (41/41, 8 new + 33 pre-existing).
pre-commit run passes on the changed Python and docs files.

Where should the reviewer start?

components/src/dynamo/common/configuration/groups/kv_router_args.py
-- CLI/env plumbing and kv_router_kwargs() forwarding.
components/src/dynamo/common/tests/configuration/test_kv_router_args.py
-- focused coverage for CLI, env vars, defaults, forwarding, and
--load-aware.
docs/components/router/router-configuration.md
-- user-facing flag descriptions and tuning guidance.

No selector, Rust config, or pyo3 signature changes are needed because those
already accept and consume these kwargs.

Related Issues:

N/A. The contribution guide flags changes ≥100 lines or new features for
pre-issue discussion. The source diff here is ~31 lines exposing
pre-existing config fields (the rest is tests and docs), and there is no new
feature surface beyond what the pyo3 binding already supports -- happy to
file an issue retroactively if maintainers prefer.

Summary by CodeRabbit

New Features
- Added --router-host-cache-hit-weight and --router-disk-cache-hit-weight CLI parameters to configure cache hit credit weighting in KV router (defaults: 0.75 for host, 0.25 for disk). Supports environment variable configuration.
Documentation
- Updated router configuration guide with descriptions of new cache hit weight parameters and tuning guidelines.

copy-pr-bot · 2026-05-31T05:43:39Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-05-31T05:43:44Z

🌿 Fern Docs Preview: https://nvidia-preview-0c251088-b512-4210-8e0f-c4edcecd7e05.docs.buildwithfern.com/dynamo/dev

coderabbitai · 2026-05-31T05:47:56Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d07d872a-03c0-497b-80f9-c222535e097a

📥 Commits

Reviewing files that changed from the base of the PR and between e55ab43 and 45cb241.

📒 Files selected for processing (3)

components/src/dynamo/common/configuration/groups/kv_router_args.py
components/src/dynamo/common/tests/configuration/test_kv_router_args.py
docs/components/router/router-configuration.md

Walkthrough

This PR introduces two new KV router configuration parameters—host_cache_hit_weight and disk_cache_hit_weight—to control cache hit credit multipliers. The changes add schema fields, CLI argument binding with environment variable support, comprehensive validation tests, and documentation.

Changes

KV Router Cache Hit Weight Feature

Layer / File(s)	Summary
Config Field Definition `components/src/dynamo/common/configuration/groups/kv_router_args.py`	Add `host_cache_hit_weight` and `disk_cache_hit_weight` float fields to `KvRouterConfigBase` and register them in `_KV_ROUTER_FIELDS` so they are included in `kv_router_kwargs()` output.
CLI Argument Registration `components/src/dynamo/common/configuration/groups/kv_router_args.py`	Extend `KvRouterArgGroup.add_arguments` with `--router-host-cache-hit-weight` and `--router-disk-cache-hit-weight` options, environment variable support (`DYN_ROUTER_HOST_CACHE_HIT_WEIGHT`, `DYN_ROUTER_DISK_CACHE_HIT_WEIGHT`), and defaults (0.75, 0.25).
CLI, Environment, and Default Value Tests `components/src/dynamo/common/tests/configuration/test_kv_router_args.py`	Six unit tests validating CLI argument parsing, environment variable binding, and default values matching the config schema.
End-to-End and Regression Tests `components/src/dynamo/common/tests/configuration/test_kv_router_args.py`	Integration test confirming cache hit weights flow through `kv_router_kwargs()`; regression test confirming `--load-aware` preset preserves these weights while overriding `overlap_score_credit` to 0.0.
User-Facing Documentation `docs/components/router/router-configuration.md`	Document the new CLI flags with defaults and purposes; update `--load-aware` preset description to list the flags it preserves; add tuning guidance for lower-tier prefix cache scenarios.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately and concisely summarizes the main change: exposing cache hit weight configuration parameters via CLI and environment variables.
Description check	✅ Passed	The description includes all required template sections with detailed information about the changes, test coverage, and where reviewers should focus.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

The pyo3 `KvRouterConfig` binding already accepts `host_cache_hit_weight` and `disk_cache_hit_weight` as constructor kwargs, and the selector reads them at every routing decision (`compute_logit_cached_blocks` in `selector.rs`), but neither is reachable from `dynamo.frontend` / `dynamo.router` CLI or via the matching `DYN_ROUTER_*` env var. Without these, deployments running a backend that exposes a lower-tier KV cache (for example, vLLM's `OffloadingConnector` for CPU offload) cannot tune how strongly the router prefers a worker holding the prefix in CPU vs disk vs device memory short of building `KvRouterConfig` from Python directly. Add two flags symmetric to `--router-prefill-load-scale`: --router-host-cache-hit-weight / DYN_ROUTER_HOST_CACHE_HIT_WEIGHT --router-disk-cache-hit-weight / DYN_ROUTER_DISK_CACHE_HIT_WEIGHT Defaults match the existing pyo3 defaults (0.75 / 0.25), so behavior is unchanged for deployments that do not set the new flag or env var. The `load_aware` preset preserves both values, mirroring how it already preserves `prefill_load_scale`. No Rust changes: pure CLI plumbing through the existing `_KV_ROUTER_FIELDS` / `kv_router_kwargs()` pattern that `prefill_load_scale` already follows. Tests: - Eight new cases in `test_kv_router_args.py` cover CLI flag, env var, default, `kv_router_kwargs()` flow-through, and load-aware preset preservation for both fields. Docs: - `docs/components/router/router-configuration.md` documents the two new flags in the Routing Behavior list and adds a tuning-guidelines paragraph for lower-tier weights. Signed-off-by: Change72 <cguo51@asu.edu>

Signed-off-by: Change72 <cguo51@asu.edu>

PeaBrane · 2026-06-02T17:56:20Z

/ok to test f96a405

…dynamo#10157) Signed-off-by: Change72 <cguo51@asu.edu> Signed-off-by: Muqi Li <muqi1029@gmail.com>

) Signed-off-by: Change72 <cguo51@asu.edu>

@Change72

The column was mixing two incompatible quantities. Most cells came from the release bodies, but the bodies themselves are not uniform: v1.1.0's figure counted only external contributors, from a list its own wording calls partial ("first-time external contributors ... include"). GitHub's New Contributors lists answer a third question -- first-ever merged PR over the compare range -- and put the pre-v1.0.0 releases at 11 where their announcements say 20 and 14. Settle on the bodies, which is what every other column in this table already uses, and require the figure to be release-wide and complete. v1.1.0 fails both tests and becomes a dash rather than an undercount. v1.3.0 moves 23 -> 24. The published body states 24 and enumerates 24; 23 came from a pre-release draft. @Change72 was raised as a name that should not be on that list, on the grounds that PR #10157 predates the v1.2.1 tag. It does by date, but release/1.2.x was cut from main on 2026-05-12 and #10157 merged 2026-06-02, so it is absent from v1.2.1 and first ships in v1.3.0 -- inside the compare range, which makes them a genuine first-timer here. Left in place. Footnote the empty cells on the rendered page too. They are permanent now rather than pending, so a reader deserves to know why they are empty and that first-timers are counted release-wide. Signed-off-by: Dan Gil <dagil@nvidia.com>

@Change72

The column was mixing two incompatible quantities. Most cells came from the release bodies, but the bodies themselves are not uniform: v1.1.0's figure counted only external contributors, from a list its own wording calls partial ("first-time external contributors ... include"). GitHub's New Contributors lists answer a third question -- first-ever merged PR over the compare range -- and put the pre-v1.0.0 releases at 11 where their announcements say 20 and 14. Settle on the bodies, which is what every other column in this table already uses, and require the figure to be release-wide and complete. v1.1.0 fails both tests and becomes a dash rather than an undercount. v1.3.0 moves 23 -> 24. The published body states 24 and enumerates 24; 23 came from a pre-release draft. @Change72 was raised as a name that should not be on that list, on the grounds that PR #10157 predates the v1.2.1 tag. It does by date, but release/1.2.x was cut from main on 2026-05-12 and #10157 merged 2026-06-02, so it is absent from v1.2.1 and first ships in v1.3.0 -- inside the compare range, which makes them a genuine first-timer here. Left in place. Footnote the empty cells on the rendered page too. They are permanent now rather than pending, so a reader deserves to know why they are empty and that first-timers are counted release-wide. Signed-off-by: Dan Gil <dagil@nvidia.com>

@Change72

The column was mixing two incompatible quantities. Most cells came from the release bodies, but the bodies themselves are not uniform: v1.1.0's figure counted only external contributors, from a list its own wording calls partial ("first-time external contributors ... include"). GitHub's New Contributors lists answer a third question -- first-ever merged PR over the compare range -- and put the pre-v1.0.0 releases at 11 where their announcements say 20 and 14. Settle on the bodies, which is what every other column in this table already uses, and require the figure to be release-wide and complete. v1.1.0 fails both tests and becomes a dash rather than an undercount. v1.3.0 moves 23 -> 24. The published body states 24 and enumerates 24; 23 came from a pre-release draft. @Change72 was raised as a name that should not be on that list, on the grounds that PR #10157 predates the v1.2.1 tag. It does by date, but release/1.2.x was cut from main on 2026-05-12 and #10157 merged 2026-06-02, so it is absent from v1.2.1 and first ships in v1.3.0 -- inside the compare range, which makes them a genuine first-timer here. Left in place. Footnote the empty cells on the rendered page too. They are permanent now rather than pending, so a reader deserves to know why they are empty and that first-timers are counted release-wide. Signed-off-by: Dan Gil <dagil@nvidia.com>

Change72 requested review from a team as code owners May 31, 2026 05:43

pull-request-size Bot added the size/L label May 31, 2026

github-actions Bot added feat external-contribution Pull request is from an external contributor documentation Improvements or additions to documentation labels May 31, 2026

dynamo-ops approved these changes May 31, 2026

View reviewed changes

PeaBrane reviewed Jun 2, 2026

View reviewed changes

Comment thread components/src/dynamo/common/tests/configuration/test_kv_router_args.py Outdated

PeaBrane reviewed Jun 2, 2026

View reviewed changes

Comment thread components/src/dynamo/common/tests/configuration/test_kv_router_args.py

PeaBrane approved these changes Jun 2, 2026

View reviewed changes

jthomson04 approved these changes Jun 2, 2026

View reviewed changes

pull-request-size Bot added size/M and removed size/L labels Jun 2, 2026

Change72 added 2 commits June 2, 2026 10:40

Delete tautological tests

1dc3a80

Signed-off-by: Change72 <cguo51@asu.edu>

Change72 force-pushed the feat/expose-cache-hit-weights-cli branch from 144bfdc to 8828237 Compare June 2, 2026 17:42

Change72 requested review from a team as code owners June 2, 2026 17:42

Change72 requested a review from a team June 2, 2026 17:42

pull-request-size Bot removed the size/M label Jun 2, 2026

Change72 force-pushed the feat/expose-cache-hit-weights-cli branch from 8828237 to 1dc3a80 Compare June 2, 2026 17:44

pull-request-size Bot added the size/M label Jun 2, 2026

Merge branch 'main' into feat/expose-cache-hit-weights-cli

f96a405

copy-pr-bot Bot temporarily deployed to GITLAB June 2, 2026 17:56 Inactive

copy-pr-bot Bot temporarily deployed to GITLAB June 2, 2026 18:01 Inactive

ziqifan617 approved these changes Jun 2, 2026

View reviewed changes

ziqifan617 merged commit 5da7cda into ai-dynamo:main Jun 2, 2026
100 checks passed

Muqi1029 pushed a commit to Muqi1029/dynamo that referenced this pull request Jun 3, 2026

feat(router): expose host/disk cache hit weights via CLI and env (ai-…

16a37df

…dynamo#10157) Signed-off-by: Change72 <cguo51@asu.edu> Signed-off-by: Muqi Li <muqi1029@gmail.com>

tmonty12 pushed a commit that referenced this pull request Jun 8, 2026

feat(router): expose host/disk cache hit weights via CLI and env (#10157

659be17

) Signed-off-by: Change72 <cguo51@asu.edu>

dagil-nvidia mentioned this pull request Jul 28, 2026

docs: compatibility corrections — first-timer definition, SGLang KV routing, driver-floor table #12241

Closed

Navigation Menu

Uh oh!

feat(router): expose host/disk cache hit weights via CLI and env - #10157

feat(router): expose host/disk cache hit weights via CLI and env#10157
ziqifan617 merged 3 commits into
ai-dynamo:mainfrom
Change72:feat/expose-cache-hit-weights-cli

Change72 commented May 31, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

copy-pr-bot Bot commented May 31, 2026

Uh oh!

github-actions Bot commented May 31, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented May 31, 2026

❌ Failed checks (1 warning)

Uh oh!

Uh oh!

Uh oh!

PeaBrane commented Jun 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

Change72 commented May 31, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview:

Details:

Where should the reviewer start?

Related Issues:

Summary by CodeRabbit

Uh oh!

copy-pr-bot Bot commented May 31, 2026

Uh oh!

github-actions Bot commented May 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented May 31, 2026

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (1 warning)

Uh oh!

Uh oh!

Uh oh!

PeaBrane commented Jun 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Change72 commented May 31, 2026 •

edited by coderabbitai Bot

Loading

github-actions Bot commented May 31, 2026 •

edited

Loading