feat(sidecar): direct-gRPC dispatch from router to stock engine containers (v2)

[tanmayv25]

Draft / RFC — not intended to merge as one PR. This is the full v2 direct-dispatch stack, opened as a draft so the design and end-to-end validation are reviewable in one place. It will be split into small reviewable PRs (the StreamingDispatch seam already merged as #12447). It is currently ~271 commits behind main and will be rebased before any merge-ready split.

Summary

Lets the Dynamo frontend/router dispatch the final request hop directly over gRPC to stock inference-engine containers (vLLM, SGLang, TensorRT-LLM), removing the per-worker request-plane forwarding hop while preserving router richness (selection, fault detection, migration). The sidecar shrinks to a thin registrar (discovery + health); the engine runs as its stock upstream container instead of a custom Dynamo worker image.

Built on the StreamingDispatch transport seam (#12447): PushRouter's final hop is a trait object, so a per-engine GrpcDispatch swaps only the transport below the seam and inherits report-down / overload / migration by mapping tonic::Status → top-level ErrorType.

v2 architecture

• Permanent core footprint = the seam + TransportType::Grpc only. All engine-specific code lives in lib/sidecar/* composition crates, deliberately out of the ai-dynamo wheel (no tonic/engine deps in the wheel).
• run_direct thin registrar shim (dynamo-direct-register) — standalone on runtime+llm, not an LLMEngine; backend-common is clean of the old --direct path.
• One unified dynamo-direct-frontend registers per-engine providers keyed by the model card's runtime_data["direct_backend"].

Before → after

flowchart LR
    subgraph BEFORE["Today — engine embedded in the Dynamo Python worker"]
        FE1["Frontend / Router"] -->|"request plane (NATS/TCP)"| W1["Dynamo Python worker<br/>engine in-process · custom image"]
    end
    subgraph AFTER["This PR — direct gRPC to the stock container"]
        FE2["dynamo-direct-frontend<br/>Router + StreamingDispatch seam"] -->|"direct gRPC — final hop"| ENG["Stock engine container<br/>vLLM · SGLang · TRT-LLM"]
        REG["run_direct registrar<br/>discovery + health"] -.->|"health probe"| ENG
        REG -.->|"MDC + TransportType::Grpc"| DISC[("discovery")]
        DISC -.->|"discovers Grpc endpoint"| FE2
    end

Loading

Disaggregated (two-phase) direct route

flowchart LR
    FE["Frontend<br/>PrefillRouter"] -->|"1 — prefill hop (direct gRPC)"| P["Prefill engine<br/>component: prefill"]
    P -->|"handoff: bootstrap / kv_transfer_params"| FE
    FE -->|"2 — decode hop (direct gRPC)"| D["Decode engine<br/>component: backend"]
    P -.->|"KV over NIXL (cuda_ipc / RDMA)"| D
    D -->|"stream tokens"| FE

Loading

What's included

• Direct aggregated dispatch for all 3 engines (chat + streaming + fault-detection/migration + health recovery).
• Direct disaggregated dispatch for SGLang + vLLM — two-phase prefill→decode over direct gRPC with engine-to-engine NIXL KV transfer.
• Per-endpoint watcher fan-out so heterogeneous dispatches on one endpoint each receive discovery lifecycle events.
• Typed vLLM kv_transfer_params handoff (removes the protobuf-Struct float-port bug class).

Known gaps (called out, not in this PR)

• KV-aware routing is OFF for stock gRPC — engines don't stream KV events the frontend can subscribe to. This is the blocker for retiring the request-plane path.
• trtllm direct-disagg not implemented — TRT-LLM's gRPC is missing fields Dynamo needs (GetModelInfo returns 0 for max seq len; no disaggregation params in the response path).
• Metrics come from the stock container /metrics (trtllm --grpc defaults metrics off).
• Container / distribution packaging deferred.

Validation

• cargo build green across dynamo-runtime, dynamo-backend-common, dynamo-llm, all lib/sidecar/* crates, and dynamo-direct-frontend.
• Rust tests green — runtime push_router (incl. the fan-out + fake-dispatch lifecycle tests), vLLM sidecar (incl. the typed-handoff round-trip + float-port tests), sglang/trtllm sidecar suites.
• Direct-AGG validated E2E on all 3 engines; direct-DISAGG validated E2E on SGLang + vLLM — real generations with NIXL KV transfer proven (decode ran with 0 prefill batches → KV moved, not recomputed).

Follow-ups

Split into reviewable PRs · re-add KV-aware routing over gRPC · trtllm disagg (pending upstream SMG/proto additions) · container packaging.

[datadog-official]

🔄 Datadog auto-retried 13 jobs - 13 passed on retry

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 48.67% (-4.10%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: db1a68f | Docs | Datadog PR Page | Give us feedback!}