feat(vllm): enable deterministic DP and KV routing

[connorcarpenter15]

Overview

Part 3 of a 4-PR stack. This layer enables deterministic vLLM data-parallel rank selection and KV-event routing without extending the vLLM protocol.

Details

• Forward Dynamo's selected data-parallel rank as per-request x-data-parallel-rank gRPC metadata; omit the metadata when no rank is selected so vLLM retains normal load balancing.
• Require vLLM to honor the x-data-parallel-rank gRPC metadata for deterministic DP routing. No protocol capability field is used.
• Support one vLLM frontend hosting the complete DP group starting at rank 0; reject nonzero starting ranks because Control does not report local rank count.
• Require one unique, in-range ZMQ KV-event source per DP rank when KV routing is enabled, and warn before skipping unsupported transports.
• Accept vLLM's valid empty KV-event topic, which means subscribe to all topics.
• Consume the preprocessor's redundant nvext.token_in=true marker while preserving strict rejection of unsupported nvext semantics.
• Preserve cache-salt propagation for consistent Dynamo/vLLM block hashes.
• Keep the vLLM mocker aligned with the production transport by parsing and validating the same request metadata.

Multimodal request conversion remains in the top PR.

Stack

1. #12734 — split gRPC services and Control discovery
2. #12736 — preserve decode handoff on cancellation
3. #12735 — deterministic DP and KV routing
4. #12214 — multimodal sidecar requests

Base: #12736

Validation

• cargo fmt --all -- --check
• cargo test -p dynamo-vllm-sidecar — 18 unit tests and 1 executable integration test passed
• cargo test -p dynamo-vllm-mocker — 12 unit tests and 3 sidecar integration tests passed
• cargo clippy -p dynamo-vllm-sidecar --all-targets -- -D warnings
• cargo clippy -p dynamo-vllm-mocker --all-targets -- -D warnings
• cargo test -p dynamo-runtime utils::pool::tests::test_sync_pool_blocking_acquire -- --exact — passed; verifies the prior CI failure was transient
• Changed-file pre-commit hooks

Where should the reviewer start?

• lib/sidecar/vllm/src/client.rs for request-scoped gRPC metadata.
• lib/sidecar/vllm/src/model.rs for the supported full-group DP registration contract.
• lib/sidecar/vllm/src/engine.rs for rank selection and KV-event source validation.
• lib/mocker/servers/vllm/src/server.rs for mock transport parity.

Related Issues

This PR is NOT linked to an issue:

• Confirmed — no related issue

[devin-ai-integration]

Devin Review found 4 potential issues.

[tanmayv25]

Automated multi-agent code-review pass over this PR's diff (framing + expert lenses, each finding adversarially verified). Findings below are all minor / non-blocking.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[connorcarpenter15]

/ok to test 8a37d26

[karen-sy]

Findings

• P1 lib/sidecar/vllm/src/engine.rs:133,371
  KV routing is enabled unconditionally, but an empty GetKvEventSources response is treated as a protocol error. At the vendored vLLM commit, Control intentionally returns no source when kv_events_config is absent, disabled, or uses a non-ZMQ publisher. That makes an otherwise valid non-KV-event vLLM sidecar fail worker startup. Return an empty source list in the genuinely no-source case so the worker’s existing KV-routing opt-out path applies; retain the strict all-ranks check when sources are advertised. Add a no-KV-events startup test.
  Upstream behavior: [vLLM Control source](https://github.com/vllm-project/vllm/blob/3d1f5cee1552b8208f3009c75f8bc856f27e0eff/rust/src/server/src/grpc/control.rs#L117-L121).

• P2 lib/sidecar/vllm/src/engine.rs:181,333
start() returns the freshly observed DP topology, but kv_event_sources() still validates against the bootstrap-time self.model. ensure_same_identity() does not compare parallelism. If the endpoint lands on a same-model frontend with a different DP size between bootstrap and startup, Dynamo registers the new range but source validation uses the old size. Reject topology changes during startup, or retain the observed topology in the engine.

---

Noting some flags above from AI review, but generally looks good!

[connorcarpenter15]

/ok to test d9d1ae4