feat(kv-router): Route vLLM CPU KV events to HostPinned and count lower-tier applies by Change72 · Pull Request #10368 · ai-dynamo/dynamo

Change72 · 2026-06-05T19:38:12Z

What

This PR makes Dynamo's KV router consume and observe vLLM native CPU-offload KV events.

Changes:

Treat vLLM's medium="CPU" as an alias for the existing HostPinned tier, in the shared
StorageTier::from_kv_medium normalization layer (so the alias also applies to the
kv_consolidator consumer — verified consistent: CPU → HostPinned).
Preserve existing CPU_PINNED and CPU_TIER1 behavior.
Add wire tests for placeholder CPU payloads and full self-describing CPU payloads.
Count lower-tier Stored / Removed / Cleared applies in kv_cache_events_applied.
Wire the metrics handle into lazily-created lower-tier indexers used by the dynamo-llm router
assembly path.

Where should the reviewer start?

lib/kv-router/src/protocols.rs — the medium="CPU" -> StorageTier::HostPinned alias.
lib/kv-router/src/zmq_wire/tests.rs — focused wire tests for placeholder and full CPU
BlockStored payloads.
lib/kv-router/src/indexer/lower_tier.rs — lower-tier worker now increments
kv_cache_events_applied using the same (event_type, status) labels as the primary indexer.
lib/kv-router/src/indexer/lower_tier_indexers.rs,
lib/kv-router/src/indexer/local.rs, and lib/llm/src/kv_router/indexer/mod.rs — metrics
handles are threaded into lazily-created lower-tier indexers.

Why

vLLM's native OffloadingConnector publishes CPU-tier KV events with medium="CPU". Dynamo
previously did not classify that string as HostPinned, so the event could not be routed to the
lower-tier indexer as intended.

After adding the alias, the lower-tier path also needed observability. The first metrics patch made
LowerTierIndexer::worker increment the same kv_cache_events_applied counter as the primary
device-tier indexer, but the dynamo-llm router assembly path still constructed lower-tier
indexers without a metrics handle. The final wiring patch passes the shared metrics handle through
LocalKvIndexer and LowerTierIndexers::new_with_metrics, including lazily-created
HostPinned/Disk/External indexers.

This is why there are two metrics commits:

Apply-path support: count lower-tier events when a lower-tier indexer has metrics.
Production wiring: make sure production lower-tier indexers actually receive metrics.

The metrics change is intentionally additive to the existing counter rather than adding a new
storage_tier label. The existing kv_cache_events_applied{event_type,status} metric represents
the total number of indexer-applied events. Before this PR, lower-tier applies were missing from
that total; after this PR, device and lower-tier events contribute to the same counter. Adding a
tier label would be a useful follow-up if operators need per-tier event accounting, but it would be
a metric-schema change across both primary and lower-tier indexers and is outside this PR.

Relationship to vLLM #43468

This PR is the Dynamo side of the self-describing CPU-event path. It expects CPU BlockStored
events to carry enough payload for Dynamo to reconstruct local block hashes. The vLLM-side PR
provides that payload for native OffloadingConnector, including chunk mode.

For chunked offload, vLLM intentionally emits plain fan-out. Overlapping chunks may repeat the same
per-block hash on the wire. Dynamo's standard worker publisher path already runs
EventDedupFilter, which ref-counts duplicate per-worker/tier hash announcements before they reach
lower-tier indexing.

This PR does not rely on a remove_blocks_impl skip-absent-hashes change.

Tests

Test commands run:

cargo test -p dynamo-kv-router --lib       # 593 passed, 0 failed
cargo test -p dynamo-llm --lib kv_router   # 164 passed, 0 failed, 2 ignored

Focused coverage:

cpu_event_with_placeholder_payload_is_dropped_safely
cpu_event_with_full_payload_is_indexable

End-to-end validation

Validated with vLLM PR #43468 on a real single-GPU L4 stack:

model: Qwen/Qwen3-0.6B
vLLM block size: 16
offloaded chunk size: 48 (factor=3)
CPU pool: 128 MiB, intentionally small to force real CPU LRU evictions
explicit ZMQ KV events enabled
self_describing_kv_events=true
worker publisher EventDedupFilter in the path

Wire capture and router metrics reconciled exactly:

wire stored: 685 total = 331 GPU + 354 CPU
wire removed: 24 CPU removes
router kv_cache_events_applied: stored ok = 685, removed ok = 24
CPU placeholders: 0
lower-tier warnings / BlockNotFound: 0

Related Issues

Relates to vLLM native CPU offload KV-event integration:

vLLM PR: [feature][kv_offload] Self-describing KV events for OffloadingConnector vllm-project/vllm#43468

No Dynamo issue is currently linked.

copy-pr-bot · 2026-06-05T19:38:16Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-06-05T19:38:21Z

👋 Hi Change72! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

StorageTier::from_kv_medium now treats the bare string "CPU" as an alias for the canonical "CPU_PINNED" name. This lets KV-event publishers that emit medium="CPU" (notably vLLM's OffloadingConnector via CPULoadStoreSpec.medium()) route into the HostPinned tier on the router side, rather than silently falling back to the GPU/Device tier through from_kv_medium_or_default's default branch. Add three wire-contract tests in zmq_wire::tests: - cpu_medium_alias_routes_to_host_pinned_tier: pins the alias mapping and end-to-end tier classification via convert_event. - cpu_event_with_placeholder_payload_is_dropped_safely: defensive contract; underspecified BlockStored payloads (block_size=0, empty token_ids) produce zero indexable blocks and bump the unpublished- block warning counter, so the router never inserts garbage entries. - cpu_event_with_full_payload_is_indexable: happy path; fully-populated CPU BlockStored decodes to a HostPinned PlacementEvent with one indexable block per block_hashes entry. Existing 457 tests in dynamo-kv-router continue to pass. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Change72 <cguo51@asu.edu> Signed-off-by: Change72 <changg@nvidia.com>

The primary (Device) indexer increments `kv_cache_events_applied` on every event it applies, but `LowerTierIndexer::worker` previously took its `metrics` parameter as `_metrics` and dropped it on the floor. As a result, HostPinned/Disk/External traffic was invisible on `/metrics` even when CPU offload was firing and the routing pipeline was using it. Prebind the same `PreBoundEventCounters` the primary tier uses and call `.inc(kind, result)` after each Stored/Removed/Cleared apply on both `WorkerTask::Event` and `WorkerTask::EventWithAck` paths. Same counter, no new label introduced, so dashboards keep working as-is and the counter now reflects total events applied across all tiers. The trait parameter is intentionally kept as `_metrics` to minimize the diff surface against the existing `SyncIndexer::worker` signature; the underscore prefix is just the Rust convention to silence the "unused parameter" warning and using it from the body remains valid. `#[allow(clippy::used_underscore_binding)]` documents the choice for future readers. Verified end-to-end with `vllm/benchmarks/multi_turn/benchmark_serving_multi_turn.py` on L4 + Qwen3-0.6B + OffloadingConnector: the counter now ticks for HostPinned events as expected and the same workload reports ~2x more applied stored events compared to before this fix. cargo test -p dynamo-kv-router --lib: 476 / 476 pass. Signed-off-by: Change72 <changg@nvidia.com>

5b7725f made LowerTierIndexer record kv_cache_events_applied when given a metrics handle, but the construction sites never passed one: both LocalKvIndexer's lazily created per-tier indexers and the LowerTierIndexers registry built their ThreadPoolIndexer with the metrics-less constructor, so HostPinned/Disk/External traffic was applied correctly yet stayed invisible on /metrics (only Device-tier counts showed up). Add LowerTierIndexers::new_with_metrics and keep a metrics handle on LocalKvIndexer, wiring it through at all four production assembly sites. The metrics-less new() remains for tests and documents that production assembly should pass metrics. Verified on a real single-GPU dynamo serve + vLLM (OffloadingConnector, chunked offload, small CPU pool) run: with the wiring, kv_cache_events_applied matches the captured wire exactly -- stored ok = 685 (331 GPU + 354 CPU) and removed ok = 24 (real CPU evictions), previously 332 / 0. cargo test -p dynamo-kv-router --lib: 545 / 545 pass. cargo test -p dynamo-llm --lib kv_router: 166 / 166 pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Change72 <changg@nvidia.com>

Signed-off-by: Change72 <changg@nvidia.com>

devin-ai-integration

Devin Review found 1 potential issue.

coderabbitai · 2026-06-17T01:46:34Z

Walkthrough

KvIndexerMetrics is now propagated into all lower-tier indexer construction paths (LowerTierIndexers::new_with_metrics, LocalKvIndexer, and all four Indexer::new branches). The LowerTierIndexer worker loop activates the previously no-op metrics parameter to emit per-event counters. The "CPU" wire string is added as a StorageTier::HostPinned alias, with two new tests covering CPU BlockStored event conversion.

Changes

Lower-tier indexer metrics propagation and CPU medium mapping

Layer / File(s)	Summary
CPU medium wire string mapping and tests `lib/kv-router/src/protocols.rs`, `lib/kv-router/src/zmq_wire/tests.rs`	`StorageTier::from_kv_medium` now maps `"CPU"` to `HostPinned`. Two new tests cover `convert_event` for CPU `BlockStored` with a placeholder payload (dropped with warning) and a full payload (produces correct `HostPinned` placement, zero warnings).
LowerTierIndexers metrics field and LowerTierIndexer worker activation `lib/kv-router/src/indexer/lower_tier_indexers.rs`, `lib/kv-router/src/indexer/lower_tier.rs`	`LowerTierIndexers` gains `metrics: Option<Arc<KvIndexerMetrics>>` and a `new_with_metrics` constructor; `get_or_create` switches to `ThreadPoolIndexer::new_with_metrics`. In the `LowerTierIndexer` worker loop, the previously unused `metrics` option now pre-binds a counter and increments it per `WorkerTask::Event`/`EventWithAck` using `EventKind` and the `apply_event` result.
Metrics wiring through LocalKvIndexer and Indexer::new `lib/kv-router/src/indexer/local.rs`, `lib/llm/src/kv_router/indexer/mod.rs`	`LocalKvIndexer` stores the `Arc<KvIndexerMetrics>` handle and passes `Some(self.metrics.clone())` when lazily creating per-`StorageTier` `ThreadPoolIndexer`s. All four `Indexer::new` branches (use_kv_events × thread count) switch lower-tier construction to `LowerTierIndexers::new_with_metrics` and clone metrics into primary indexer constructors.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~15 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main changes: routing vLLM CPU KV events to HostPinned and adding metrics counting for lower-tier applies.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The pull request description is comprehensive and follows the template structure with all required sections completed.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Signed-off-by: Change72 <changg@nvidia.com>

jthomson04 · 2026-06-22T21:10:55Z

/ok to test 17e2672

Trace the exact code paths for both fixes: - GAP-1: SGLang StorageMedium.CPU event on HiCache offload - GAP-2: PR ai-dynamo#10368 from_kv_medium() CPU mapping Co-Authored-By: Claude <noreply@anthropic.com>

…er-tier applies (ai-dynamo#10368) Signed-off-by: Change72 <cguo51@asu.edu> Signed-off-by: Change72 <changg@nvidia.com> Co-authored-by: Claude <noreply@anthropic.com> (cherry picked from commit d8a94ef)

- Narrow the normalizer bypass to hash-only lower tiers (Disk / External) so CPU / HostPinned offload events keep flowing through salted cache-namespace propagation (no ai-dynamo#10368 regression). - Gate both consolidator ingresses on locality as well: drop REMOTE / unknown-locality events, not just non-device media, so a device-medium event tagged REMOTE is never tracked as a local G1 block. Adds a RawKvEvent::locality() accessor and extends the two consolidator G1-only tests with REMOTE / unknown cases. Signed-off-by: Change72 <changg@nvidia.com>

pull-request-size Bot added the size/L label Jun 5, 2026

github-actions Bot added external-contribution Pull request is from an external contributor router Relates to routing, KV-aware routing, etc. labels Jun 5, 2026

Change72 and others added 3 commits June 16, 2026 17:24

Change72 force-pushed the feat/kv-router-cpu-medium-alias branch from db0ec35 to 1d4e7b0 Compare June 17, 2026 00:25

test(kv-router): update CPU event tests for image token arg

4c4bba4

Signed-off-by: Change72 <changg@nvidia.com>

Change72 changed the title ~~[Feat]kv router cpu medium alias~~ [kv-router] Route vLLM CPU KV events to HostPinned and count lower-tier applies Jun 17, 2026

Change72 changed the title ~~[kv-router] Route vLLM CPU KV events to HostPinned and count lower-tier applies~~ feat(kv-router) Route vLLM CPU KV events to HostPinned and count lower-tier applies Jun 17, 2026

Change72 changed the title ~~feat(kv-router) Route vLLM CPU KV events to HostPinned and count lower-tier applies~~ feat(kv-router): Route vLLM CPU KV events to HostPinned and count lower-tier applies Jun 17, 2026

github-actions Bot added the feat label Jun 17, 2026

Change72 marked this pull request as ready for review June 17, 2026 01:42

Change72 requested a review from a team June 17, 2026 01:42

devin-ai-integration Bot reviewed Jun 17, 2026

View reviewed changes

Comment thread lib/kv-router/src/indexer/lower_tier.rs

Change72 mentioned this pull request Jun 22, 2026

[feature][kv_offload] Self-describing KV events for OffloadingConnector vllm-project/vllm#43468

Merged

jthomson04 reviewed Jun 22, 2026

View reviewed changes

Comment thread lib/kv-router/src/indexer/lower_tier_indexers.rs

docs(kv-router): clarify lower-tier metrics constructor

17e2672

Signed-off-by: Change72 <changg@nvidia.com>

Change72 requested a review from jthomson04 June 22, 2026 17:48

copy-pr-bot Bot temporarily deployed to GITLAB June 22, 2026 21:11 Inactive

copy-pr-bot Bot temporarily deployed to GITLAB June 22, 2026 22:31 Inactive

ziqifan617 approved these changes Jun 25, 2026

View reviewed changes

jthomson04 merged commit d8a94ef into ai-dynamo:main Jun 25, 2026
94 checks passed

Change72 deleted the feat/kv-router-cpu-medium-alias branch June 26, 2026 16:32

Change72 mentioned this pull request Jul 27, 2026

feat(kv-router): route vLLM STORAGE KV events to the Disk tier with locality gating #11571

Merged

Change72 mentioned this pull request Jul 30, 2026

fix(kv-router): fail closed on unrecognized KV event media #12456

Merged

Navigation Menu

Uh oh!

feat(kv-router): Route vLLM CPU KV events to HostPinned and count lower-tier applies - #10368

feat(kv-router): Route vLLM CPU KV events to HostPinned and count lower-tier applies#10368
jthomson04 merged 5 commits into
ai-dynamo:mainfrom
Change72:feat/kv-router-cpu-medium-alias

Change72 commented Jun 5, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jun 5, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

jthomson04 commented Jun 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Change72 commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Where should the reviewer start?

Why

Relationship to vLLM #43468

Tests

End-to-end validation

Related Issues

Uh oh!

copy-pr-bot Bot commented Jun 5, 2026

Uh oh!

github-actions Bot commented Jun 5, 2026

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Uh oh!

Uh oh!

jthomson04 commented Jun 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Change72 commented Jun 5, 2026 •

edited

Loading

coderabbitai Bot commented Jun 17, 2026 •

edited

Loading