Repository navigation
feat(frontend): runtime-free assembly over a static worker file - #25
Closed
ishandhanani wants to merge 1 commit into
Closed
ishandhanani wants to merge 1 commit into
ishandhanani wants to merge 1 commit into
Conversation
Add `StaticDirectEngine` (`lib/llm/src/kv_router/static_direct.rs`): a frontend engine that serves one model with no `DistributedRuntime`. Worker membership comes from a workers file (the selection service's `--workers-file` shape), KV-aware selection runs on an embedded `SelectionService` that subscribes to the workers' ZMQ KV events itself (load-based when no worker publishes events), and admitted requests go to each worker's direct engine endpoint through the installed `DirectEngineFactory`. Bookings are freed when the response stream ends. `entrypoint::input::http::run_static` serves an in-process engine over HTTP without a runtime; the shared builder now takes `Option<&DistributedRuntime>` so the runtime-backed path is unchanged. Python: `make_static_engine` and `run_static_input` bindings, `dynamo.frontend --static-workers-file` (`DYN_FRONTEND_STATIC_WORKERS_FILE`), which skips runtime construction entirely. Requires `--model-path` (the model card is read locally) and `DYN_ROUTER_DIRECT_DISPATCH`. `WorkerRequest` derives `Clone`. Test: the mock vLLM gRPC server is named in a workers file, the assembly books it, streams three tokens, and frees the booking, with no runtime in the process. Docs cover the flag and the layout. Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
ishandhanani
had a problem deploying
to
external_collaborator
September 3, 2026 10:30 — with
GitHub Actions
Failure
Owner
Author
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #24. Roadmap Phase 4: frontend assembly without
DistributedRuntime, model card independent of discovery, static-file feeder.Summary
StaticDirectEngine(lib/llm/src/kv_router/static_direct.rs): serves one model with noDistributedRuntime. Membership from a workers file (the selection service's--workers-fileJSON shape;endpoint= the worker's direct gRPC endpoint), KV-aware selection on an embeddedSelectionServicethat subscribes to the workers' ZMQ KV events itself (load-based routing when no worker publishes events), dispatch through the installedDirectEngineFactory. Bookings are freed when the response stream ends;prefill_completeis signalled on the first data chunk.entrypoint::input::http::run_static: HTTP service without a runtime. The builder assembly is shared (http_service_builder_for(.., Option<&DistributedRuntime>, ..)), so the runtime-backed path is unchanged.make_static_engine(args, workers_file),run_static_input(engine, extensions),cancel_static_frontend();dynamo.frontend --static-workers-file(DYN_FRONTEND_STATIC_WORKERS_FILE) takes the runtime-free path and never constructsDistributedRuntime. Requires--model-path(a local model directory; the card is read locally) andDYN_ROUTER_DIRECT_DISPATCH=vllm.WorkerRequestnow derivesClone. Docs:frontend-configuration.mdxparam field;sidecar-backends.md"Running Without A Distributed Runtime".Validation
cargo test -p dynamo-vllm-mocker static_direct: the mock vLLM gRPC server is named in a workers file (plus one worker without an endpoint), the assembly registers both, connects one, streams 3 tokens, and the booking is freed (free_reservation->NotFound) after the stream ends. No runtime in the process.dynamo-vllm-mocker-serverinstances (ports 50061/50062) +python -m dynamo.frontend --static-workers-file ... --model-path <local Qwen3-0.6B> --model-name Qwen/Qwen3-0.6B --router-mode kvwithDYN_ROUTER_DIRECT_DISPATCH=vllm: model served in 4 s, 4/4 chat completions OK (8 tokens each), log markerstatic direct frontend assembled without a distributed runtime workers=2 reachable=2, clean exit on SIGINT.cargo check --workspace --all-targets,cargo clippy --workspace --all-targets, bindingscargo check: clean.docs_lint.py: 0 errors.Limitations: membership is fixed for the process lifetime; aggregated only (disaggregated, encoders, and worker churn use the runtime-backed frontend). Dependency on
dynamo-runtimeremains at compile time (pipeline types); removing it is Phase 5.