Repository navigation
feat(router): direct native-gRPC dispatch beside the request plane - #19
Closed
ishandhanani wants to merge 1 commit into
Closed
ishandhanani wants to merge 1 commit into
ishandhanani wants to merge 1 commit into
Conversation
ishandhanani
had a problem deploying
to
external_collaborator
September 3, 2026 08:33 — with
GitHub Actions
Failure
ishandhanani
force-pushed
the
idhanani/p2-direct-dispatch
branch
from
September 3, 2026 08:33
54aed7c to
a3a255a
Compare
ishandhanani
had a problem deploying
to
external_collaborator
September 3, 2026 08:33 — with
GitHub Actions
Failure
ishandhanani
force-pushed
the
idhanani/p2-direct-dispatch
branch
from
September 3, 2026 08:37
a3a255a to
b1e4e61
Compare
ishandhanani
had a problem deploying
to
external_collaborator
September 3, 2026 08:37 — with
GitHub Actions
Failure
Phase 2 of the runtime-optional frontend roadmap: a second RoutingHost transport that sends admitted KV-routed requests straight to a worker's native engine RPC, leaving the sidecar off the request path while discovery still resolves membership and endpoints. - lib/llm kv_router/direct_dispatch.rs: DirectEngine (an AsyncEngine over the same PreprocessedRequest / Annotated<LLMEngineOutput> shapes the request plane uses), DirectEngineFactory, and a worker-keyed DirectDispatchRegistry whose feeder follows the runtime-config watch: workers advertising runtime_data[native_grpc_endpoint] get an engine, workers that leave or change endpoint are dropped, failed connects retry on the next membership change. A process-wide factory install point lets the assembling binary supply the protocol implementation without lib/llm depending on it. - RoutingHost::with_direct_dispatch; dispatch_selection picks the direct engine when the selected worker has one, else the request plane. Booking release on failure, the DynamoError chain the migration layer keys on, and cancellation via the request context are unchanged; request-plane fault detection does not observe direct failures (documented). - backend-common: direct_engine() exposes the worker-side EngineAdapter so a frontend gets identical streaming, deferred decode abort, and error semantics; re-exports the direct-dispatch types and ModelRuntimeConfig. - vLLM sidecar: --advertise-grpc-endpoint publishes native_grpc_endpoint and native_grpc_mode in the worker's runtime data; VllmSidecarEngine:: connect_direct and VllmDirectEngineFactory implement the factory over the existing gRPC client, request conversion, and gRPC-status error mapping. - Assembly: DYN_ROUTER_DIRECT_DISPATCH=vllm in the Python frontend installs the vLLM factory; every KV routing host then attaches a registry. Tests: RoutingHost serves a registered fake direct engine and, on a direct failure, releases the booking and surfaces a migratable CannotConnect error; workers without an engine fall back to the request plane. The vLLM factory streams tokens from the mock vLLM gRPC server and reports an unreachable endpoint as a connect failure. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
force-pushed
the
idhanani/p2-direct-dispatch
branch
from
September 3, 2026 09:05
b1e4e61 to
f48efbc
Compare
ishandhanani
had a problem deploying
to
external_collaborator
September 3, 2026 09:05 — with
GitHub Actions
Failure
This was referenced Sep 3, 2026
Owner
Author
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of the runtime-optional frontend roadmap (contract 3, dispatch). Stacked on #18.
Summary
A second
RoutingHosttransport: admitted KV-routed requests go straight to a worker's native engine RPC when the worker advertises one, so the sidecar leaves the request path while discovery still resolves membership and endpoints. Workers that do not advertise keep using the request plane, so a fleet can migrate one worker at a time.lib/llm/src/kv_router/direct_dispatch.rs:DirectEngine(anAsyncEngine<SingleIn<PreprocessedRequest>, ManyOut<Annotated<LLMEngineOutput>>>, the exact shape the request-plane client returns),DirectEngineFactory,DirectDispatchRegistry(worker id -> engine) with a feeder on the runtime-config watch: connect onruntime_data["native_grpc_endpoint"], drop on departure or endpoint change, retry failed connects on the next membership change.install_direct_engine_factoryis the process-wide seam solib/llmnever depends on an engine protocol crate (the sidecar crates depend onlib/llmvia backend-common, so the dependency must point this way).RoutingHost::with_direct_dispatch;dispatch_selection(routing_host/kv.rs) chooses the direct engine when the selected worker has one. Everything around it is untouched:guard.abort()still releases the booking before the error propagates (contract 1), theDynamoErrorchainmigration::is_migratableinspects is preserved, and cancellation reaches the engine through the request context. Documented gap:PushRouterfault detection does not see direct-dispatch failures; liveness comes from discovery.direct_engine(Arc<dyn LLMEngine>, mode)exposes the worker-sideEngineAdapter, so a frontend gets identical streaming, deferred decode abort (decode must reach the engine so NIXL can release transferred KV), and error semantics as the request-plane path. Re-exports the direct-dispatch types andModelRuntimeConfig.--advertise-grpc-endpoint/DYN_ADVERTISE_GRPC_ENDPOINTpublishesnative_grpc_endpoint+native_grpc_modein the worker's runtime data.VllmSidecarEngine::connect_direct(async discovery + start) andVllmDirectEngineFactoryimplement the factory over the existing gRPC client, request conversion (convert.rs), andstatus_to_dynamoerror mapping (connect/timeout/cancelled/unavailable -> the sameBackendErrors the migration taxonomy already classifies).DYN_ROUTER_DIRECT_DISPATCH=vllmin the Python frontend installs the vLLM factory;entrypoint/input/common.rsattaches a registry to every KV routing host when a factory is installed. Bindings gain thedynamo-vllm-sidecardependency.sidecar-backends.md(Direct Dispatch section),frontend-configuration.mdx, vLLM sidecar README.Validation
cargo test -p dynamo-llm --lib -- kv_router: 334 passed. Newdirect_dispatch_serves_registered_workers_and_releases_on_failure: a registered fake engine serves the request and the booking is released on completion; a failing fake releases the booking and surfacesErrorType::CannotConnectthrough the error chain; an unregistered worker never touches the fake.cargo test -p dynamo-vllm-mocker -p dynamo-vllm-sidecar: 4 + 23 + 13 passed. Newdirect_engine_factory_streams_from_the_mock_server: the factory connects to the mock vLLM gRPC server, streams 3 tokens with aLengthfinish, leaves no active requests, and an unreachable endpoint fails to connect.cargo clippy --all-targetsondynamo-llm,dynamo-backend-common,dynamo-vllm-sidecar,dynamo-vllm-mocker: clean.cargo checkof the Python bindings: clean.cargo fmtanddocs_lint.pyclean.DYN_ROUTER_DIRECT_DISPATCH=vllmagainst a real vLLM gRPC server and the worker-shutdown migration tests (needs a GPU worker); these are the Phase 2 exit gates.