Skip to content

feat(router): direct native-gRPC dispatch beside the request plane - #19

Closed
ishandhanani wants to merge 1 commit into
idhanani/p1-kv-router-embedded-selectionfrom
idhanani/p2-direct-dispatch
Closed

ishandhanani wants to merge 1 commit into
idhanani/p1-kv-router-embedded-selectionfrom
idhanani/p2-direct-dispatch

Conversation

@ishandhanani

Copy link
Copy Markdown
Owner

Phase 2 of the runtime-optional frontend roadmap (contract 3, dispatch). Stacked on #18.

Summary

A second RoutingHost transport: admitted KV-routed requests go straight to a worker's native engine RPC when the worker advertises one, so the sidecar leaves the request path while discovery still resolves membership and endpoints. Workers that do not advertise keep using the request plane, so a fleet can migrate one worker at a time.

  • lib/llm/src/kv_router/direct_dispatch.rs: DirectEngine (an AsyncEngine<SingleIn<PreprocessedRequest>, ManyOut<Annotated<LLMEngineOutput>>>, the exact shape the request-plane client returns), DirectEngineFactory, DirectDispatchRegistry (worker id -> engine) with a feeder on the runtime-config watch: connect on runtime_data["native_grpc_endpoint"], drop on departure or endpoint change, retry failed connects on the next membership change. install_direct_engine_factory is the process-wide seam so lib/llm never depends on an engine protocol crate (the sidecar crates depend on lib/llm via backend-common, so the dependency must point this way).
  • RoutingHost::with_direct_dispatch; dispatch_selection (routing_host/kv.rs) chooses the direct engine when the selected worker has one. Everything around it is untouched: guard.abort() still releases the booking before the error propagates (contract 1), the DynamoError chain migration::is_migratable inspects is preserved, and cancellation reaches the engine through the request context. Documented gap: PushRouter fault detection does not see direct-dispatch failures; liveness comes from discovery.
  • backend-common: direct_engine(Arc<dyn LLMEngine>, mode) exposes the worker-side EngineAdapter, so a frontend gets identical streaming, deferred decode abort (decode must reach the engine so NIXL can release transferred KV), and error semantics as the request-plane path. Re-exports the direct-dispatch types and ModelRuntimeConfig.
  • vLLM sidecar: --advertise-grpc-endpoint / DYN_ADVERTISE_GRPC_ENDPOINT publishes native_grpc_endpoint + native_grpc_mode in the worker's runtime data. VllmSidecarEngine::connect_direct (async discovery + start) and VllmDirectEngineFactory implement the factory over the existing gRPC client, request conversion (convert.rs), and status_to_dynamo error mapping (connect/timeout/cancelled/unavailable -> the same BackendErrors the migration taxonomy already classifies).
  • Assembly: DYN_ROUTER_DIRECT_DISPATCH=vllm in the Python frontend installs the vLLM factory; entrypoint/input/common.rs attaches a registry to every KV routing host when a factory is installed. Bindings gain the dynamo-vllm-sidecar dependency.
  • Docs: sidecar-backends.md (Direct Dispatch section), frontend-configuration.mdx, vLLM sidecar README.

Validation

  • cargo test -p dynamo-llm --lib -- kv_router: 334 passed. New direct_dispatch_serves_registered_workers_and_releases_on_failure: a registered fake engine serves the request and the booking is released on completion; a failing fake releases the booking and surfaces ErrorType::CannotConnect through the error chain; an unregistered worker never touches the fake.
  • cargo test -p dynamo-vllm-mocker -p dynamo-vllm-sidecar: 4 + 23 + 13 passed. New direct_engine_factory_streams_from_the_mock_server: the factory connects to the mock vLLM gRPC server, streams 3 tokens with a Length finish, leaves no active requests, and an unreachable endpoint fails to connect.
  • cargo clippy --all-targets on dynamo-llm, dynamo-backend-common, dynamo-vllm-sidecar, dynamo-vllm-mocker: clean. cargo check of the Python bindings: clean. cargo fmt and docs_lint.py clean.
  • Not yet run: an end-to-end sidecar-backed deployment with DYN_ROUTER_DIRECT_DISPATCH=vllm against a real vLLM gRPC server and the worker-shutdown migration tests (needs a GPU worker); these are the Phase 2 exit gates.

Phase 2 of the runtime-optional frontend roadmap: a second RoutingHost
transport that sends admitted KV-routed requests straight to a worker's native
engine RPC, leaving the sidecar off the request path while discovery still
resolves membership and endpoints.

- lib/llm kv_router/direct_dispatch.rs: DirectEngine (an AsyncEngine over the
  same PreprocessedRequest / Annotated<LLMEngineOutput> shapes the request
  plane uses), DirectEngineFactory, and a worker-keyed DirectDispatchRegistry
  whose feeder follows the runtime-config watch: workers advertising
  runtime_data[native_grpc_endpoint] get an engine, workers that leave or
  change endpoint are dropped, failed connects retry on the next membership
  change. A process-wide factory install point lets the assembling binary
  supply the protocol implementation without lib/llm depending on it.
- RoutingHost::with_direct_dispatch; dispatch_selection picks the direct
  engine when the selected worker has one, else the request plane. Booking
  release on failure, the DynamoError chain the migration layer keys on, and
  cancellation via the request context are unchanged; request-plane fault
  detection does not observe direct failures (documented).
- backend-common: direct_engine() exposes the worker-side EngineAdapter so a
  frontend gets identical streaming, deferred decode abort, and error
  semantics; re-exports the direct-dispatch types and ModelRuntimeConfig.
- vLLM sidecar: --advertise-grpc-endpoint publishes native_grpc_endpoint and
  native_grpc_mode in the worker's runtime data; VllmSidecarEngine::
  connect_direct and VllmDirectEngineFactory implement the factory over the
  existing gRPC client, request conversion, and gRPC-status error mapping.
- Assembly: DYN_ROUTER_DIRECT_DISPATCH=vllm in the Python frontend installs
  the vLLM factory; every KV routing host then attaches a registry.

Tests: RoutingHost serves a registered fake direct engine and, on a direct
failure, releases the booking and surfaces a migratable CannotConnect error;
workers without an engine fall back to the request plane. The vLLM factory
streams tokens from the mock vLLM gRPC server and reports an unreachable
endpoint as a connect failure.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Owner Author

Closed while Stack 1 (#45 to #53) and Stack 2 (#40) go upstream; the branch stays for later stacks (direct dispatch, catalog leases, runtime features). See #33.

This branch had an error being deployed

1 failed deployment
external_collaborator — f48efbc9 Deployed Sep 3, 2026 by ishandhanani via ok-to-test #39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat router

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant