fix(kv-router): fail closed on unrecognized KV event media

[Change72]

Summary

dynamo was fail-open on KV-event medium strings it doesn't recognize: convert_event fell back to the Device tier via from_kv_medium_or_default, so an unrecognized medium was indexed on the G1 primary radix tree — and separately flowed through the salted cache-namespace normalizer state and was tracked as G1 by both consolidator ingresses. This PR makes all three paths fail closed.

Concrete trigger: vLLM v0.26.0 (dynamo's current pin) emits FS / OBJ media (present since vLLM #47923; unified to STORAGE only in 0.27+ via #48123). dynamo doesn't recognize FS / OBJ, so today they hit all three holes. This is the follow-up promised in #11571:

Hardening dynamo's handling of unrecognized media (fail-closed instead of defaulting to Device) is a separate follow-up PR — it is a pre-existing latent issue on main, independent of this change.

Behavior

medium	before → after
absent / GPU / CPU / STORAGE / other recognized	unchanged
unrecognized (e.g. v0.26.0 FS / OBJ, future strings)	router: fell back to Device → indexed on G1 → dropped in conversion (rate-limited warn), now a backstop; normalizer: polluted salted-namespace state, and a later conversion drop burned a next_event_id (false engine-drop metric) → rejected in preprocess (unknown_medium filter reason); no event id burned; consolidator: tracked as G1 → dropped

Changes

• convert_event: drop unrecognized media with a rate-limited warning (was a silent Device fallback). Removes the now-unused StorageTier::from_kv_medium_or_default. This drop is kept as a defensive backstop for direct callers that bypass preprocess.
• preprocess_with_reason: reject unrecognized media with a new unknown_medium filter reason instead of letting them reach conversion. Filtering here — exactly like the feat(kv-router): route vLLM STORAGE KV events to the Disk tier with locality gating #11571 locality gate — means the LLM ZMQ listener records an intentional filter rather than accepting the event, incrementing next_event_id, and only dropping it in conversion; that id gap is what the event processor misreads as an engine drop (engines_dropped_events). Recognized lower-tier media (STORAGE → Disk/External) still bypass normalizer state; Device and HostPinned events (GPU, CPU offload feat(kv-router): Route vLLM CPU KV events to HostPinned and count lower-tier applies #10368) still flow through the normalizer so their salted namespaces keep propagating.
• Both consolidator ingresses (kvbm-consolidator, LLM kv_consolidator): gate unrecognized media out of the G1-only tracker.

The normalizer and consolidator conditions are intentionally different (normalizer keeps CPU on-path for salted propagation; the consolidator is G1-only and drops CPU), so there is no shared predicate.

Out of scope: locality semantics (settled in #11571), the CPU/HostPinned normalizer path (#10368), any FS / OBJ → tier mapping (dynamo deliberately does not consume them — this only flips them from fail-open to fail-closed), and removal/refcount semantics (#11930).

Validation

All commands on a Linux workstation, all green:

cargo test -p dynamo-kv-router --lib
cargo test -p dynamo-kv-router --features standalone-indexer --lib
cargo test -p dynamo-llm --lib kv_router::publisher
cargo test -p dynamo-llm --lib block_manager::kv_consolidator
cargo test -p kvbm-consolidator
cargo fmt --all -- --check
cargo clippy --no-deps --all-targets -- -D warnings

New/updated tests:

• preprocess_with_reason rejects FS / OBJ / XYZ as unknown_medium (store + remove) while recognized media pass.
• an FS event reusing a salted GPU hash is rejected in preprocess (so it never touches namespace state) and the salted GPU chain keeps inheriting the namespace.
• listener-level GPU → FS → GPU (real ZMQ publisher): the two GPU stores land on consecutive event ids (0, 1), proving the filtered FS store burns no next_event_id (no false engine drop).
• convert_event still drops FS / OBJ / XYZ as a backstop (store + remove, any locality); both consolidator ingresses drop FS.

AI assistance

Implemented with AI assistance (Cursor); design, scoping, and review were human-directed.

Summary by CodeRabbit

• New Features

  • Added support for vLLM disk offloading through the documented storage tiering configuration.
  • Clarified that STORAGE events map to Dynamo’s Disk tier.

• Bug Fixes

  • Unknown or unsupported storage media are now rejected instead of silently defaulting to device storage.
  • Invalid events no longer consume event IDs or alter namespace state.
  • Updated event filtering to consistently accept only supported device-tier events where required.

• Documentation

  • Documented rejection of legacy FS and OBJ event types.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-019fb5cf-e4a0-75f1-aab3-ab78f9b86b21.docs.buildwithfern.com/dynamo/dev

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 40.82% (-1.02%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7e48754 | Docs | Datadog PR Page | Give us feedback!}

[dynamo-ops]

/ok to test d5834a5

[dynamo-ops]

/ok to test 630b42f

[devin-ai-integration]

Devin Review found 2 potential issues.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 314b5353-da34-4fb6-8a63-73412d3756f0

📥 Commits

Reviewing files that changed from the base of the PR and between 8a753c3 and 630b42f.

📒 Files selected for processing (8)

• docs/fern/components/router/router-offloading.md
• lib/kv-router/src/protocols.rs
• lib/kv-router/src/zmq_wire/convert.rs
• lib/kv-router/src/zmq_wire/mod.rs
• lib/kv-router/src/zmq_wire/tests.rs
• lib/kvbm-consolidator/src/ingress/zmq_subscriber.rs
• lib/llm/src/block_manager/kv_consolidator/subscriber.rs
• lib/llm/src/kv_router/publisher/tests.rs

💤 Files with no reviewable changes (1)

• lib/kv-router/src/protocols.rs

---

Walkthrough

Unknown storage media are now rejected instead of defaulting to the device tier. Router preprocessing, conversion, G1 ingress filters, documentation, and tests were updated to recognize STORAGE while dropping legacy or unknown media.

Changes

Storage media filtering

Layer / File(s)	Summary
Tier mapping and preprocessing lib/kv-router/src/protocols.rs, lib/kv-router/src/zmq_wire/*, docs/fern/components/router/router-offloading.md	Storage media are classified explicitly; absent media map to Device, recognized tiers are converted, and unknown media are dropped with an UnknownMedium reason. vLLM documentation now identifies STORAGE as the supported disk event.
Router filtering regression coverage lib/kv-router/src/zmq_wire/tests.rs, lib/llm/src/kv_router/publisher/tests.rs	Tests verify unknown media are filtered without namespace mutation or event-ID consumption, while surrounding GPU events retain their hashes and consecutive IDs.
G1 ingress filtering lib/kvbm-consolidator/src/ingress/zmq_subscriber.rs, lib/llm/src/block_manager/kv_consolidator/subscriber.rs	G1 subscribers now accept only explicit device-tier events and test that unrecognized FS events are not tracked.

Estimated code review effort: 3 (Moderate) | ~25 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description thoroughly explains the motivation, behavior, implementation, scope, and validation, but omits the required Related Issues section and reviewer-start guidance.	Add the required Related Issues section, selecting the appropriate issue-linking option, and identify the files or sections where reviewers should start.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the primary change: rejecting unrecognized KV-event media instead of defaulting open.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[dynamo-ops]

/ok to test 896af5a

[dynamo-ops]

/ok to test cd14d9f

[dynamo-ops]

/ok to test 833fe06

[dynamo-ops]

/ok to test 5f352d7

[jthomson04]

Two source review findings.

[dynamo-ops]

/ok to test 7e48754