[surajssd]

Overview:

Summary

Add experimental, opt-in --shared-cache-type mooncake-store routing for vLLM's MooncakeStoreConnector. The router can credit reusable prefixes held in the shared store after GPU eviction, rather than treating every GPU miss as cold. Hints require validated worker metadata, eligible request and event provenance, and evidence that every required cache object is present.

Details:

Routing and residency

• Learn local request-hash to external engine-hash relationships from primary GPU Stored events before radix-tree insertion can reject an unknown parent. Reuse the precomputed request hashes rather than reproducing vLLM's Python hashing protocol.
• Track full SHA-256 digests, exact exported rank/group prefixes, and independent CPU/disk residency. Return a hint only when every object required at a candidate restart endpoint is present. Support validated full-attention, sliding-window, and Mamba-align layouts without assuming that every earlier endpoint is reusable.
• Bound identity and residency state without ever disabling the hint at a cap. Learned hash edges are kept in least-recently-used order and evict the coldest link at the cap; a request lookup promotes every link it walks. A digest identity lives only while one of its objects is resident and is cleared with residency. Reaching a residency cap discards the evidence and rebuilds it from later events. Invalidate residency on sequence gaps, malformed relevant events, and surfaced stream failures. Prevent old contract generations from changing successor state. The cache disables itself only when its event subscriber stops. Silent reconnects and lost final removal events remain best-effort staleness limits.
• Capture positive text-only eligibility before request and event normalization discards metadata. Missing or unsupported provenance withholds hints without rejecting ordinary GPU events. Publisher coalescing does not merge stores with different eligibility; they flush as separate chained events, so an eligible prefix keeps its provenance. Dump replay restores per-block eligibility from a radix-tree provenance map that is opt-in: the frontend enables it only when --shared-cache-type is not none, and the worker-side local indexer always enables it because its tree dump is the consumer and it cannot see router configuration. The wire decoders keep the distinction between MessagePack binary and text values.

Worker contracts and lifecycle

• Add an explicit worker extension and bounded dynamo_mooncake_store_descriptor RPC. Export initialized connector facts through ModelRuntimeConfig.set_engine_specific, validate rank coverage and descriptor agreement, and support composition with the existing FPM worker extension.
• Pin the adapter to vLLM revision 1085b64425a9e6f5ca52876ad32e55fda5665f4e. Require one data-parallel rank per endpoint, prefill context parallelism (PCP) of one, and decode context parallelism (DCP) of one. Tensor and pipeline parallelism use the exact exported object prefixes.
• Validate contracts against admitted workers without changing model deployment card checksum or admission rules. Keep one passive cache per router, start its background tasks only after construction succeeds, and cancel and await both tasks during shutdown. The Mooncake cache is created only when the worker-selection policy declares WorkerInputs::CACHE and the KV event subscription is required, following the builtin-policy refactor on main.
• Replace the scheduler-owned strong request-lease observer with a weak observer. This breaks a reference cycle exposed by the failed-constructor regression while retaining live admission, progress, and completion callbacks.
• Preserve GPU overlap and cached-token accounting, scheduler semantics, and HiCache behavior. Unsupported optional contracts disable hints, not serving. Invalid feature configuration is rejected explicitly. Add configuration, deployment documentation, and generated ownership rules. Most changes outside these components initialize the new event-provenance field.

Validation

Validation ran on the complete implementation at the current head, not on each intermediate commit.

Check	Result
cargo test -p dynamo-kv-router	1075 passed, 2 ignored
cargo test -p kvbm-consolidator --no-fail-fast	51 passed, 1 ignored
cargo test -p dynamo-llm --no-default-features --lib	2802 passed, 2 ignored; 3 macOS environment failures (gRPC metrics-port bind, HTTP 413 body decode, discovery alias claim) that also fail or flake without this change and pass in isolation
Pinned offline coordinator oracle	All 12 cases passed through both local indexers
Reviewer probe from the eviction thread (edge cap reached, then the same contract republished)	disabled=false, hits preserved; the cache no longer switches itself off
Focused Python worker tests (test_mooncake_store_runtime.py, test_mooncake_store_initialized.py, test_runtime_metadata.py) against the native binding	59 passed; 4 initialized-connector cases skipped
Python router configuration tests against the native binding	83 passed
Rust bindings (maturin develop, macOS, dynamo-llm default features off)	Built and installed
cargo fmt --check, cargo clippy --tests on dynamo-kv-router and dynamo-llm, pre-commit run on changed files	Passed; no new warnings
Pre-merge CI on the current head	All 26 lightweight checks pass, including rust-tests and rust-clippy on all four matrix directories and the docs, Fern, codeowners, and DCO checks. Full CI (PR workflow) was approved at this head and is in progress; its Preview or publish docs job failed on a Fern asset-upload error covering fonts and images this PR does not touch (the same failure hit another PR the previous day and passed on rerun).

The kvbm-consolidator integration harness now encodes injected events as named maps, the same way both production publishers do. The positional encoder only lined up with the positional decoder by accident, and adding the trailing shared_cache_eligible field to BlockStored exposed that.

Acceptance remains unverified for:

• The four initialized-connector cases, which require torch and the exact pinned vLLM installation.
• Live Mooncake transfer and GPU acceptance, including tensor parallelism of at least two with full-attention plus sliding-window and full-attention plus Mamba, reuse after GPU eviction, and a miss when a required parallel object is absent.

The Fern CLI was not run locally; the Fern configuration and broken-link checks pass in CI. The whole Rust workspace test suite runs in the rust-tests (.) job and passes. The whole Python suite was not run locally.

Measurements

The three-group, tensor-parallel-two debug benchmark averaged 13.19 ms for 4,096 blocks and 56.12 ms for 16,384 blocks over 32 runs per size, up from 8.42 ms and 39.28 ms before learned edges moved to a least-recently-used map. Each walked link is now promoted in the LRU order, and the lookup takes the state lock mutably to do so. These are not release-profile latency results. Measured backing-map allocations at the fixed caps were approximately 100.5 MiB live and 119 MiB during growth on aarch64 (rustc 1.96), of which the LRU edge map accounts for about 2 MiB over a plain hash map; these figures exclude other runtime state and are not process RSS.

Where should the reviewer start?

1. lib/llm/src/kv_router/shared_cache/mooncake_store_contract.rs and lib/llm/src/kv_router/shared_cache/vllm_mooncake_store.rs define supported layouts, contract validation, residency, and reusable-prefix lookup.
2. components/src/dynamo/vllm/mooncake_store_runtime.py defines the pinned worker contract and export path. components/src/dynamo/vllm/tests/generate_mooncake_oracle.py supplies the offline coordinator reference cases.
3. lib/llm/src/discovery/model_manager.rs and lib/llm/src/kv_router/indexer/mod.rs integrate admitted membership and primary event observation. lib/kv-router/src/services/selection/core/run.rs issues the request-time check_blocks query, with eligibility threaded from lib/llm/src/kv_router/routing_host/kv_selection.rs.
4. lib/kv-router/src/zmq_wire/tests.rs, lib/llm/src/kv_router/shared_cache/vllm_mooncake_store/tests.rs, and lib/llm/src/discovery/model_manager/mooncake_tests.rs cover provenance, hybrid readiness, contract replacement, and lifetime regressions.

Related Issues

🔗 This PR is linked to an issue:

• Relates to [FEATURE]: Shared-cache tier for the vLLM MooncakeStoreConnector (--shared-cache-type mooncake-store) #14842. This implements a supported subset of the requested feature. The issue's DCP4 deployment is outside the current scope, which requires DCP=1.

Summary by CodeRabbit

• New Features

  • Added experimental Mooncake Store shared-cache routing for compatible vLLM deployments.
  • Added runtime metadata publication, contract validation, cache residency tracking, and fallback behavior.
  • Added request eligibility checks to protect unsupported multimodal, Eagle, and other incompatible workloads.
  • Added support for configuring mooncake-store through router and frontend settings.

• Bug Fixes

  • Improved shared-cache provenance handling for stored KV-cache events and replayed data.

• Documentation

  • Added setup requirements, compatibility limitations, configuration guidance, and troubleshooting details.

• Tests

  • Expanded coverage for configuration, routing, runtime integration, cache behavior, and failure handling.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi surajssd! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[surajssd]

@dmitry-tokarev-nv PTAL :-)

[dmitry-tokarev-nv]

Approved at 7d9d629ab. I found no new defect.

My earlier approval named e5c2714ad6, which is no longer the head. The author force-pushed six times after it. I read the new tree as a new change rather than as a delta. All three of my earlier P2 items are fixed, and I ran code on each one.

Part of this round is self-review. I gave the direction for the batching.rs fix on the CodeRabbit thread, and 7c37a65b82 follows it. I mutation-tested that line on its own and posted the result there.

What changed since the approved tree, by patch identity

I ran git patch-id --stable over every commit on both trees. The first nine commits carry the same content as the tree I approved. Only their commit identities moved, because of the rebase. Five commits are new:

commit	subject	what it answers
88c8c072ae	evict Mooncake Store identity instead of disabling hints	my P2 on the learned-edge and digest caps
13f831d95f	record shared-cache provenance only where a dump consumer needs it	my P2 on the radix-tree map
7c37a65b82	keep mixed shared-cache eligibility in separate store events	the CodeRabbit thread on batching.rs
58a144cad7	read Mooncake runtime metadata through runtime_data	my P2 on the failing Python test
7d9d629aba	encode injected consolidator batches as named events	a test helper that the new wire field broke

Those five commits touch nine files, 381 added and 125 removed lines.

The merge base also moved, from d07fcea4b8 to a67b954e74, which is 43 commits of main. Eighteen of the files this pull request touches also changed in that range. I merged the current main tip b40d169e0ad myself with git merge-tree --write-tree. It produced tree d33a2c7ecb with no conflict. Only one file this pull request touches also changed between a67b954e74 and that tip: docs/fern/pages/reference/components/frontend-configuration.mdx. No conclusion here rests on that file.

What I ran at this head, and what I did not test

On x86_64 Linux with the pinned toolchain 1.96.1:

command	result
cargo test -p dynamo-kv-router --lib	1073 passed, 0 failed, 2 ignored
cargo test -p dynamo-llm --no-default-features --lib kv_router::	411 passed, 0 failed, 1 ignored
cargo test -p kvbm-consolidator	51 passed, 0 failed, 1 ignored, across nine binaries

I reverted lib/kvbm-consolidator/tests/common/mod.rs to the state before 7d9d629aba and ran it again. The dedup binary then failed 3 of 3 with invalid type: boolean false, expected u64. So that commit carries real weight, and the tests pin it.

I ran the three CODEOWNERS gates from .github/workflows/codeowners.yml on the checked-out tree. The strict full-tree gate exited 0. emit_codeowners.py rewrote CODEOWNERS and CONTRIBUTORS.md, both checksums were unchanged, and git diff --exit-code exited 0. So the generated file matches its source. The matcher unit tests failed 16 of 157 on my machine. They fail the same 16 on the base commit a67b954e74. Each failure is a git commit that exits 128 in a temporary repository. That is my environment, not this change.

Every name the new documentation uses exists in the code. SharedCacheType::MooncakeStore carries #[serde(rename = "mooncake-store")] at lib/kv-router/src/scheduling/config.rs:392. The Python --shared-cache-type choices are ["none", "hicache", "mooncake-store"]. DYN_MOONCAKE_KV_EVENTS_ENDPOINT is read at lib/llm/src/discovery/model_manager.rs:2442, and MooncakeStoreWorkerExtension is at components/src/dynamo/vllm/mooncake_store_runtime.py:395. Every new Python test file carries pytestmark with unit or integration, plus vllm, gpu_0 and pre_merge.

I did not test these three:

1. Live Mooncake transfer, GPU acceptance, and the four initialized-connector cases. Those need a GPU and the pinned vLLM build.
2. The full components/src/dynamo/vllm/tests/test_runtime_metadata.py run. My available bindings are older than this branch. I measured the accessor contract it depends on instead, and posted that on the thread.
3. generate_mooncake_oracle.py --check. It needs the pinned vLLM source tree. Nothing in the repository runs it, so oracle.json can only go stale against vLLM, not against the Rust reader. The green pinned_coordinator_oracle_through_normalized_local_indexers test covers the reader.

One hypothesis I killed before it became a finding

ExtraKeyItem gained a Binary variant, and it is first in an untagged enum. I expected a MessagePack text value to match it before Hash(String). If that happens, extra_keys_are_text_only rejects every request that carries a cache salt or a LoRA name, and the hint never fires.

It does not happen. shared_cache_eligibility_requires_known_text_keys_in_maps_and_tuples runs 27 cases through both the named-map and the positional encodings. Six of them expect eligibility to be true with text extra keys. Those cases pass in the 1073-test run, so text matches Hash. Retracted.

I also read the positional decoder. lib/kv-router/src/zmq_wire/deserialize.rs never reads shared_cache_eligible from a positional payload. It computes the value from lora_id, is_eagle, block_mm_infos and extra_keys. So the new field cannot shift the slots, and a real vLLM producer that does not know the field is unaffected.

[dmitry-tokarev-nv]

/ok to test 3c74eeb

[dmitry-tokarev-nv]

Approved at ff1b4ac9df. I found no new defect.

My last approval named 7d9d629ab, and the head no longer contains that commit. The author rebased onto 4d60b59662 and added one commit. I compared the two trees commit by commit and file by file, and I ran code on each change.

What changed since the approved tree.

git merge-base --is-ancestor fails in both directions, so this push rewrote history. git patch-id --stable matches 10 of the 14 old commits. The other four changed only to follow main. Across the 108 files, the added and removed lines of this PR are identical in 103. Five files differ:

file	change
lib/llm/src/discovery/model_manager.rs	The Mooncake Store arm now calls the new KvEventSourceRequirement::derive(role, config, policy.inputs()) from main.
lib/llm/src/kv_router.rs	The shared-cache test router uses the Aggregated role and a membership watch. The builtin policy does not declare CACHE for a plain decode router.
lib/llm/src/kv_router/indexer/ingress.rs	An import merge.
docs/fern/pages/reference/components/frontend-configuration.mdx	The --load-aware note now follows the new text from main.
components/src/dynamo/vllm/mooncake_store_runtime.py	The new mypy commit ff1b4ac9df.

A probe of the rebased gate.

I added a throwaway probe test to lib/llm/src/kv_router.rs. I ran it on x86_64 Linux at ff1b4ac9df. The probe used a mooncake-store configuration that passes validation. In all 40 cases of role, conditional disaggregation, and declared inputs, should_subscribe() equals the CACHE declaration of the policy. So if the policy declares CACHE and the model does not use Eagle, the arm builds the cache. As a control, the two configurations that validation rejects (use_remote_indexer=true, use_kv_events=false) break that equality.

With the builtin policy, the rebased gate matches the subscription rule of the approved tree for prefill, aggregated, and decode routers. This is true with and without conditional disaggregation.

Before the probe, cargo test -p dynamo-llm --no-default-features --lib -- kv_router:: discovery:: gave 655 passed, 0 failed, 1 ignored.

The mypy commit.

I ran the pinned mypy==1.18.2 on mooncake_store_runtime.py with the MYPYPATH of the dynamo-runtime / mypy job. On the parent 3c74eeb864, it reports the same 12 errors as that job, line for line. On ff1b4ac9df, it reports none. The second step of that job, mypy -p dynamo, did not run, because the first step failed. It is clean on the base and on the head, and a planted stub error makes it fail.

The commit message says "No behavior changes". I loaded the module from both commits and ran the rewritten branches. _require raises UnsupportedContract with the same message, so two of the three rewrites act the same. The third changes one case. If vllm.__file__ is None, the worker reply changes from {'unsupported': 'TypeError'} to {'unsupported': 'vLLM package has no source location'}. In both cases, publish_mooncake_store_runtime returns False and publishes nothing. Only the text of its warning differs.

The new vllm_config: Any annotation adds no name to dir(). vLLM reads dir() to find conflicts with a worker extension. test_mooncake_store_runtime.py passes 40 of 40 on both commits.

CI, and a merge with the current main.

The Pre Merge job rust-tests (.) ran at this head, on Merge ff1b4ac9df into cde7c9a8c5. It printed 169 test result lines, and none failed. The tests of this PR appear by name, for example 19 in vllm_mooncake_store::tests and 15 in model_manager::mooncake_tests. The full PR workflow last ran on 3c74eeb864, not on this head.

I merged main at 7472c23abf with git merge-tree --write-tree, and it merged without conflict. The three lock files at that tip are identical to those at cde7c9a8c5, which Pre Merge built with --locked. On the merged tree, emit_codeowners.py rewrites CODEOWNERS with no change, and the strict full-tree gate passes. When I removed the shared_cache/ entry of this PR from areas.yaml, the drift gate failed. So the gate can fail.

What I did not test.

The limits from my last approval still apply. I ran no live Mooncake transfer and no GPU acceptance. The four initialized-connector cases need the pinned vLLM build, and they skip in the vllm-runtime lane.

The batching fix e30c60dd39 follows the direction that I gave on the CodeRabbit thread, so that part is self-review. Its patch is identical to the patch that I approved.