[surajssd]

Overview

Part 2a of the shared-cache eligibility work for #14842. Preserve an explicit eligibility decision through event publication and buffered recovery so later consumers can use it.

Summary

Carry shared_cache_eligible on RouterEvent and PlacementEvent. The wire key is sce. Missing metadata defaults to false, serialization omits false values, and a true value adds five bytes in named MessagePack. KvCacheStoreData stays unchanged.

Key Changes

The patch adds 39 nonblank production lines, excluding tests and comments. The full change is nine files, 156 additions, and 33 deletions.

• protocols.rs: add envelope metadata with legacy defaults and compact serialization.
• batching.rs: keep different eligibility values in separate coalesced events and preserve metadata through publication.
• indexer/tests.rs: verify buffered recovery preserves the flag while synthesized dumps remain ineligible by default.

Validation

After rebasing onto main at f01ff2b075, the macOS router library suite passes 1,232 tests, with three ignored. The LLM publisher suite passes all 127 tests with default features disabled.

shared_cache_eligibility_defaults_false_and_roundtrips verifies legacy defaults, JSON and named MessagePack round trips, omission of false, and the five-byte cost of true. shared_cache_eligibility_survives_coalescing_publication_and_buffering verifies same-value coalescing, separation of different values, and preservation through publication and the recovery buffer.

Local strict Clippy did not pass: the installed Rust 1.99.1 reports warnings in unchanged code, including deprecated fetch_update and async_trait must-use diagnostics. The repository pins Rust 1.96.1. This rebase adds no feature changes.

Issues and Ongoing Work

• Part of [FEATURE]: Shared-cache tier for the vLLM MooncakeStoreConnector (--shared-cache-type mooncake-store) #14842; split from feat(kv-router): add Mooncake Store shared-cache routing #14965.
• feat(kv-router): classify shared-cache event eligibility (2b) #16431 adds raw ZMQ classification. feat(kv-router): preserve opt-in dump eligibility (2c) #16432 adds opt-in tree-dump provenance. Both depend only on this part.
• feat(kv-router): expose Mooncake pin/unpin commands for custom policies #17093 exposes Mooncake pin/unpin commands for custom policies; this PR only carries eligibility metadata.
• Production consumer activation remains deferred to the shared-cache consumer.

Pull Request Checklist

• I used Dynamo's /pr-checklist skill.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi surajssd! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review.

Walkthrough

The change adds shared-cache eligibility metadata to stored events and derives it from wire-event metadata. Indexers track and restore eligibility through block operations and dumps. Publisher batching preserves eligibility boundaries. Mooncake replay preparation stores router events in boxes.

Changes

Shared-cache eligibility

Layer / File(s)	Summary
Eligibility contract and wire parsing lib/kv-router/src/protocols.rs, lib/kv-router/src/zmq_wire/*, lib/kvbm-consolidator/tests/common/mod.rs, lib/kv-router/src/zmq_wire/tests.rs	Stored-event data and wire events carry shared_cache_eligible. Deserialization evaluates provenance, Eagle, LoRA, multimodal, and extra-key metadata. Tests cover map and tuple encodings, malformed metadata, and round trips. Stored-event constructors and fixtures across bindings, benchmarks, consolidator, relay, and mocker paths now initialize eligibility to false.
Indexer provenance tracking lib/kv-router/src/indexer/*, lib/kv-router/src/test_utils.rs, lib/llm/src/kv_router/indexer/*	KvIndexerBuilder can enable provenance tracking. RadixTree records and clears block provenance, then restores eligibility when it dumps events. Indexer setup enables tracking in configured paths. Indexer dump code and test fixtures set eligibility explicitly.
Publisher batching lib/llm/src/kv_router/publisher/*	Stored events coalesce only when their eligibility values match and their parent hashes link. Tests cover all four eligibility pairings and check parent links and serialized values.

Mooncake event payload ownership

Layer / File(s)	Summary
Box events through replay preparation lib/bench/kv_router/mooncake_open_loop.rs, lib/bench/tests/mooncake_trace.rs	MooncakeOperationPayload::Event stores Box<RouterEvent>. Event preparation boxes events; replay clones through as_ref(), and dispatch preparation dereferences the box to create an owned event.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: Minimal · up to 44fa4

The supplied evidence identifies no change that needs to be fixed before merging. Complete the normal checks before release.

Pre-merge checks: 4

Check name	Status	Explanation
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.
Title check		The title clearly and concisely describes the main change: carrying shared-cache eligibility through KV-router events.
Description check		The description includes all required sections, explains the purpose and key changes, documents validation, lists ongoing work, and includes the repository checklist. It is mostly complete, although s…

Full details: Description check

Explanation

The description includes all required sections, explains the purpose and key changes, documents validation, lists ongoing work, and includes the repository checklist. It is mostly complete, although some scope details do not fully match the supplied changeset.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Tip

🎯 Customize your risk assessments

Guide how CodeRabbit assesses merge risk and security architecture risk. Highlight sensitive changes and explain your security boundaries using Merge Risk Instructions and Security Risk Instructions in your review settings. Security architecture risk assessments require the Advanced plan or above.

✅ Automatic approvals, with your rules

Let CodeRabbit approve pull requests once required threads are resolved, pre-merge checks pass, and your risk limits are met. Enable Automatic approvals in review settings, with overrides for teams or file paths.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6cfc1b5d-7463-40e4-bba5-b559e01a2455

📥 Commits

Reviewing files that changed from the base of the PR and between a76e12e and 44fa4d9.

📒 Files selected for processing (61)

• lib/bench/kv_router/common/dc_ckf_parity.rs
• lib/bench/kv_router/common/dc_ckf_shared.rs
• lib/bench/kv_router/dc_ckf_consumer_bench.rs
• lib/bench/kv_router/mooncake_open_loop.rs
• lib/bench/tests/mooncake_trace.rs
• lib/bindings/c/src/lib.rs
• lib/bindings/python/rust/llm/kv.rs
• lib/kv-router/benches/indexer_delegate.rs
• lib/kv-router/benches/selection_core.rs
• lib/kv-router/src/indexer/approximate_lru.rs
• lib/kv-router/src/indexer/branch_sharded.rs
• lib/kv-router/src/indexer/compressed_radix.rs
• lib/kv-router/src/indexer/concurrent_radix_tree.rs
• lib/kv-router/src/indexer/cuckoo/adapter.rs
• lib/kv-router/src/indexer/cuckoo/dc.rs
• lib/kv-router/src/indexer/cuckoo/publication.rs
• lib/kv-router/src/indexer/kv_indexer.rs
• lib/kv-router/src/indexer/local.rs
• lib/kv-router/src/indexer/lower_tier.rs
• lib/kv-router/src/indexer/lower_tier_indexers.rs
• lib/kv-router/src/indexer/positional.rs
• lib/kv-router/src/indexer/radix_tree.rs
• lib/kv-router/src/indexer/shard_handle.rs
• lib/kv-router/src/indexer/tests.rs
• lib/kv-router/src/indexer/thread_pool.rs
• lib/kv-router/src/protocols.rs
• lib/kv-router/src/services/indexer/backend.rs
• lib/kv-router/src/test_utils.rs
• lib/kv-router/src/zmq_wire/convert.rs
• lib/kv-router/src/zmq_wire/deserialize.rs
• lib/kv-router/src/zmq_wire/extra_keys.rs
• lib/kv-router/src/zmq_wire/tests.rs
• lib/kv-router/src/zmq_wire/types.rs
• lib/kv-router/tests/standalone_indexer_http.rs
• lib/kvbm-consolidator/src/ingress/zmq_subscriber.rs
• lib/kvbm-consolidator/tests/common/mod.rs
• lib/kvbm-consolidator/tests/dedup.rs
• lib/kvbm-consolidator/tests/e2e.rs
• lib/kvbm-consolidator/tests/kvbm_bridge.rs
• lib/kvbm-consolidator/tests/lifecycle.rs
• lib/kvbm-consolidator/tests/output_contract.rs
• lib/kvbm-consolidator/tests/zmq_ingress.rs
• lib/llm/benches/kv_router_bench.rs
• lib/llm/src/block_manager/kv_consolidator/subscriber.rs
• lib/llm/src/kv_dc_relay/actor.rs
• lib/llm/src/kv_dc_relay/pool_registry.rs
• lib/llm/src/kv_dc_relay/publication/codec.rs
• lib/llm/src/kv_dc_relay/publication/hub.rs
• lib/llm/src/kv_dc_relay/resolution.rs
• lib/llm/src/kv_dc_relay/wan/grpc/conformance.rs
• lib/llm/src/kv_router/indexer/mod.rs
• lib/llm/src/kv_router/indexer/recovery/state_agent.rs
• lib/llm/src/kv_router/indexer/recovery/worker_query.rs
• lib/llm/src/kv_router/indexer/recovery/worker_query_state.rs
• lib/llm/src/kv_router/publisher/batching.rs
• lib/llm/src/kv_router/publisher/state_agent.rs
• lib/llm/src/kv_router/publisher/tests.rs
• lib/mocker/src/engine_observations.rs
• lib/mocker/src/replay/offline/extensions/kv_router/mod.rs
• lib/mocker/src/replay/online/router.rs
• lib/mocker/src/services/zmq_events.rs

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

[jthomson04]

/ok to test 7a15c49

[devin-ai-integration]

✅ Dynamo PR CI passed — run 38956099883 (attempt 1) on 6fa671de30

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

_{Posted automatically by Devin for run 38956099883. Updated on every full-CI run of this PR.}

[devin-ai-integration]

External PR triage (2026-10-13)

Blocker: trusted CI has not run on the current head ea629c689 — the last /ok to test (@jthomson04, 2026-10-07) covered an older head (7a15c4995); new commits landed on 2026-10-12, so the approval is stale. The previous run's only failures were two image builds that hit job time limits (infra, not this change).

Next action (maintainer): @jthomson04 (approved CI on 2026-10-07) or @ai-dynamo/dynamo-router-codeowners @ai-dynamo/dynamo-kv-memory-codeowners @ai-dynamo/dynamo-performance-codeowners — after review, a vetter comments /ok to test ea629c68938099f4693c5c8805a91cf059b2bece so the full Dynamo PR CI runs on this exact SHA.

Supply-chain audit (head ea629c689): ✅ Looks safe to approve. 60 Rust files (kv-router indexer/zmq wire, kv publisher, mocker, kvbm-consolidator, benches/tests); no workflow/CI or container edits, no new download URLs or scripts, no secret/credential access, no binary blobs; no Cargo.toml/Cargo.lock changes.

_{Automated triage of external contributions; this comment is edited in place on each run. Written by Devin}

[jthomson04]

Thanks for splitting this out of #14965. Before it goes in, I'd like to trim it to what the Mooncake tier will actually read:

1. Nothing in tree consumes shared_cache_eligible yet, but the cost is on by default. LocalKvIndexer::new enables provenance on every vLLM worker, because text-only stores decode as eligible. In a release RadixTree micro-bench with 524k resident blocks, that's +63% heap (27.1 → 44.1 MiB) and +65% apply time per event. Please move the radix-tree provenance and the LocalKvIndexer opt-in into the PR that adds the consumer, or drop dump provenance and let replays be ineligible, as CRTC already does.
2. The frontend gate in kv_router/indexer/mod.rs does nothing useful. The default router_event_threads=4 uses CRTC, which ignores it, and your 9599bcf notes that no observer reads a frontend dump. Please drop it.
3. ExtraKeyItem::Binary changes GPU-path acceptance. On main, a vLLM batch with a non-UTF-8 bin extra key (the prompt-embeds sha256) fails to decode and is dropped; here it's indexed. vLLM reports [0]*n token IDs for those requests, so unrelated requests would now share local hashes. That needs its own decision and PR.
4. Add skip_serializing_if = "std::ops::Not::not". False is already the serde default, and today every store event from every backend pays 23 bytes for the key (+12.6% on a 1-block store).
5. Fan-out: since false is fail-closed, forcing every struct literal to spell it out adds no safety. Putting the bit on RouterEvent/PlacementEvent next to session_id (default + skip + a builder, compared in the coalescer the way session_id is) avoids about 50 files of churn.
6. Tests: deleting both provenance.remove calls in apply_removed keeps every test green. Please assert the map is empty after removes.

The rest checks out: the coalescer split, the dump eligibility runs, cleanup on clear/remove_worker, and wire compatibility in both directions.

[jthomson04]

Thanks for splitting this. Everything from my last review is covered: the flag now sits next to session_id with a default, skip-if-false and a builder; the coalescer compares it; tree dumps stay ineligible; and the provenance map moved to #16432 as opt-in. Perf: nothing in production sets it to true yet, so it adds no bytes on the wire and one bool compare in the coalescer. The ExtraKeyItem::Binary concern moved to #16431 unchanged; I'll follow up there. Three non-blocking notes inline.

[jthomson04]

/ok to test 6fa671d

[jthomson04]

I'm not sure we need this bit, and this applies to 2b and 2c too. As I read 2b, sce means "tokens_hash covered every extra key vLLM hashed for this block". It's only false because the decoder drops keys it doesn't map: untagged salts, native MM hashes, and future keys.

Could the decoder fold those unmapped keys into that block's hash seed instead? No request could then reach an edge learned from such a store, which is the same effect as sce = false. That would remove the field and coalescer split here, shrink 2b to the fold, and drop 2c's provenance map, since dumps would already carry the right hashes. The request-side gate stays in the frontend.

It also stops the GPU tree crediting those stores to plain requests. That changes GPU routing, so it should be its own PR. Is there a case the bit catches that this wouldn't?

[jh-nv]

@surajssd are you still working on this? We would like to push on this feature. Thanks for the contribution!

[surajssd]

@surajssd are you still working on this? We would like to push on this feature. Thanks for the contribution!

Yep