feat: integrate cache_salt with KV routing for multi-tenant isolation [DYN-3178]

[jthomson04]

Overview:

Adds cache-salt-aware KV routing and reuse isolation across Dynamo request ingestion, block hashing, router selection, bindings, standalone indexing, vLLM, and TensorRT-LLM KV events.

This work is tracked internally as DYN-3178. It is backward compatible for unsalted requests: when no non-empty salt is supplied, Dynamo retains the existing hash and cache-reuse behavior.

Details:

• Resolves the public cache-salt API with this precedence:
  1. Non-empty x-tenant-id gateway header.
  2. Non-empty nvext.cache_salt (recommended request API).
  3. Non-empty top-level cache_salt compatibility field.
• Treats empty strings as absent, including falling back from an empty NvExt value to a non-empty top-level compatibility value.
• Carries the resolved namespace through the standalone preprocessor, C binding, Rust EPP, Python binding, router request, and cache-overlap paths. The Rust EPP applies the gateway header before body-derived values.
• Mixes the namespace into block hashing with a dedicated seed while preserving both the existing unsalted hash output and the established LoRA hash formula shared with dynamo-kv-hashing.
• Threads cache salt into both legacy and unified vLLM TokensPrompt inputs. Dynamo uses an internal tag while the salt is inside vLLM's untyped extra_keys, then removes it in the Rust event decoder so arbitrary public values, including 64-character hex salts and salts equal to a LoRA name, round-trip without ambiguity.
• Preserves cache salt on TensorRT-LLM stored-block events, rejects events that combine conflicting block salts, and drops only the malformed event rather than terminating the publisher thread.
• Adds pre-merge real-engine isolation E2Es for unified vLLM and unified TensorRT-LLM; both submit identical tokens under two salts and verify the router indexes each namespace independently.
• Makes the standalone indexer contract explicit:
  • /query computes salted hashes from token IDs.
  • /query_by_hash requires hashes that were already computed with the intended salt and returns 400 Bad Request for any non-null cache_salt.
• Avoids logging raw cache namespace values in the C-binding and Go EPP paths.
• Documents the API surface, precedence, frontend NvExt gating, empty-value behavior, backend support matrix, isolation usage, and standalone-indexer behavior. End-to-end backend reuse isolation is currently supported for vLLM and TensorRT-LLM, not SGLang.
• Removes the superseded .github/workflows/pr.yaml change; the operator-gating fix is already on main.

The TensorRT-LLM dependency is satisfied: NVIDIA/TensorRT-LLM#13051 is merged, and current Dynamo main pins TensorRT-LLM 1.3.0rc19.

Validation completed:

• cargo fmt --all -- --check
• cargo test -p dynamo-llm
• cargo test -p dynamo-kv-router (615 tests passed on the review-fix head)
• cargo test -p dynamo-kv-router --features standalone-indexer --test standalone_indexer_http query_
• cargo test -p dynamo-ext-proc (10 tests passed on the review-fix head)
• cargo test -p libdynamo_llm
• make dynamo-lib under deploy/inference-gateway/epp
• make test under deploy/inference-gateway/epp, using the repository-pinned golang:1.26.3 builder image because the host has no Go toolchain
• pytest -q components/src/dynamo/trtllm/tests/test_trtllm_request_utils.py components/src/dynamo/trtllm/tests/test_trtllm_fpm_publisher.py (48 tests passed)
• fern check
• fern docs broken-links
• pre-commit run --files on all review-fix files
• Pytest collection with pre_merge and vllm and gpu_1 selects test_unified_vllm_cache_salt_isolation[tcp].

The focused vLLM unit and real-engine E2E require the vLLM CI image and GPU lane; the local environment does not have vLLM installed.

Where should the reviewer start?

• lib/llm/src/protocols/common/extensions.rs for request input precedence and empty-value handling.
• lib/kv-router/src/protocols.rs and lib/llm/src/kv_router.rs for salted hashing and routing propagation.
• components/src/dynamo/vllm/handlers.py, components/src/dynamo/vllm/llm_engine.py, and lib/kv-router/src/zmq_wire/extra_keys.rs for vLLM request and event round-trip handling.
• components/src/dynamo/vllm/llm_engine.py plus test_unified_vllm_cache_salt_isolation for unified vLLM E2E coverage.
• components/src/dynamo/trtllm/utils/request_utils.py and components/src/dynamo/trtllm/publisher.py for TensorRT-LLM request/event handling and malformed-event resilience.
• lib/kv-router/src/services/indexer/server.rs plus lib/kv-router/tests/standalone_indexer_http.rs for the standalone API behavior.
• lib/bindings/c/src/lib.rs and deploy/inference-gateway/ext-proc/src/epp.rs for the shared helper consumers.

Related Issues

🚫 This PR is NOT linked to an issue:

• Confirmed — no related GitHub issue

Internal tracking: DYN-3178.

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 85.05%
• Overall Coverage: 40.77% (-4.61%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 76ec167 | Docs | Give us feedback!}

[jthomson04]

/ok to test 755794f

[jthomson04]

/ok to test 2be3455

[jthomson04]

A few non-correctness cleanup/plumbing observations (all verified — no correctness bugs surfaced). Two are inline below; the other two are on files this PR doesn't modify, so I couldn't inline-anchor them:

components/src/dynamo/router/__main__.py (get_overlap_scores ~L174, best_worker_id ~L156) — the standalone router daemon's diagnostic endpoints call kv_router.get_overlap_scores(...) / best_worker(...) positionally and don't forward the new cache_namespace kwarg, so overlap-score queries on that path report cross-tenant overlap. The main generate path is unaffected (it forwards routing through to generate_from_request, which is salt-aware). Minor, but worth plumbing for consistency.

lib/kv-hashing/tests/request_hashing.rs::producer_block_hash_matches_router_local_block_hash_lora — this test still asserts the old LoRA seed formula and its comment claims parity with compute_block_hash_for_seq's LoRA path, which this PR changed (xxh3_64(name) → xxh3_64_with_seed(name, LORA_HASH_SEED)). It still passes (kv-hashing's salt.rs governs a separate PLH/dedup hash, not the radix-tree match key, and wasn't touched), so there's no functional bug — but the comment is now misleading and worth a quick update.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-019f3e15-fc34-7560-8eb1-f6846f013c8b.docs.buildwithfern.com/dynamo/dev

[jthomson04]

/ok to test 1731da0

[jthomson04]

/ok to test 3f6acac

[jthomson04]

Ran a multi-agent review over this PR (8 finder angles, each candidate adversarially verified against the branch). The core design looks solid — unsalted hashes are preserved bit-for-bit, precedence is well documented and tested, and the standalone-indexer contract is explicit. 7 findings are attached as inline comments (most severe: the LoRA seed change and the SGLang gap). One more finding lands on a file this PR doesn't touch, so it goes here:

lib/llm/src/kv_router/push_router.rs:205-215 — fallback routing-decision recording omits with_cache_namespace. The fallback builds TokensWithHashes::new(...) and applies .with_lora_name(...) but never .with_cache_namespace(request.routing.cache_namespace), while the parallel fallback in lib/bindings/python/rust/llm/kv.rs:1784-1787 was updated in this PR. The branch appears currently unreachable (records_routing_decisions() implies routing_hashes is Some), but as written it records salted requests under unsalted hashes the moment anything makes it live — a one-line fix to match the Python-binding path.

Also noting a cross-cutting theme rather than filing it per-line: salt-resolution precedence now exists in three drifting copies (Rust extensions.rs, Python trtllm request_utils.py, vLLM _iter_nvext_sources), and the "empty string means absent" rule is re-applied ad hoc at 5+ sites. Two of the inline findings are symptoms of that; normalizing once at the ingestion boundaries would eliminate the class.

For transparency, candidates that were checked and refuted during verification: /query_by_hash rejecting "" (intentional, documented, and tested), retained Ambiguous entries leaking (requires a hash collision to matter, deliberate fail-closed), vLLM missing routing.cache_salt (frontend always mirrors the salt into extra_args.nvext), and per-block seed re-hashing (negligible vs. existing per-block work).

🤖 Generated with Claude Code

[jthomson04]

/ok to test a5dcb50

[PeaBrane]

The direct vLLM/TRT-LLM request and event plumbing looks coherent. I found three remaining cross-component concerns below: the consolidator drops the namespace, the canonical hashing helper derives a different salted key, and namespace propagation adds per-block string allocations. One minor follow-up outside the diff: the standalone router best_worker_id and get_overlap_scores diagnostic/custom-scheduling endpoints still do not accept or forward cache_namespace, so external schedulers using those endpoints cannot request namespace-aware results. Leaving this as a neutral comment rather than an approval or request-changes verdict.

[jthomson04]

/ok to test 40c8b0f

[jthomson04]

/ok to test fa59951

[PeaBrane]

CI note: the vLLM amd64 failure is actionable—test_generate_text_mode_applies_nvext_cache_salt still expects tenant-a, while the implementation now intentionally forwards dynamo-cache-salt:tenant-a to vLLM for unambiguous KV-event decoding. The TRT-LLM shard completed 117 passed / 1 skipped, then its container action failed near the 30-minute boundary during teardown/reporting, which looks like a runner timeout rather than a test failure. backend-status-check is the downstream aggregate failure.

[jthomson04]

/ok to test 893e7d8