Skip to content

feat(frontend): runtime-free assembly over a static worker file - #25

Closed
ishandhanani wants to merge 1 commit into
idhanani/p3-decode-firstfrom
idhanani/p4-static-frontend
Closed

ishandhanani wants to merge 1 commit into
idhanani/p3-decode-firstfrom
idhanani/p4-static-frontend

Conversation

@ishandhanani

Copy link
Copy Markdown
Owner

Stacked on #24. Roadmap Phase 4: frontend assembly without DistributedRuntime, model card independent of discovery, static-file feeder.

Summary

  • StaticDirectEngine (lib/llm/src/kv_router/static_direct.rs): serves one model with no DistributedRuntime. Membership from a workers file (the selection service's --workers-file JSON shape; endpoint = the worker's direct gRPC endpoint), KV-aware selection on an embedded SelectionService that subscribes to the workers' ZMQ KV events itself (load-based routing when no worker publishes events), dispatch through the installed DirectEngineFactory. Bookings are freed when the response stream ends; prefill_complete is signalled on the first data chunk.
  • entrypoint::input::http::run_static: HTTP service without a runtime. The builder assembly is shared (http_service_builder_for(.., Option<&DistributedRuntime>, ..)), so the runtime-backed path is unchanged.
  • Python: make_static_engine(args, workers_file), run_static_input(engine, extensions), cancel_static_frontend(); dynamo.frontend --static-workers-file (DYN_FRONTEND_STATIC_WORKERS_FILE) takes the runtime-free path and never constructs DistributedRuntime. Requires --model-path (a local model directory; the card is read locally) and DYN_ROUTER_DIRECT_DISPATCH=vllm.
  • WorkerRequest now derives Clone. Docs: frontend-configuration.mdx param field; sidecar-backends.md "Running Without A Distributed Runtime".

Validation

  • cargo test -p dynamo-vllm-mocker static_direct: the mock vLLM gRPC server is named in a workers file (plus one worker without an endpoint), the assembly registers both, connects one, streams 3 tokens, and the booking is freed (free_reservation -> NotFound) after the stream ends. No runtime in the process.
  • Live smoke with no etcd/NATS process running: two dynamo-vllm-mocker-server instances (ports 50061/50062) + python -m dynamo.frontend --static-workers-file ... --model-path <local Qwen3-0.6B> --model-name Qwen/Qwen3-0.6B --router-mode kv with DYN_ROUTER_DIRECT_DISPATCH=vllm: model served in 4 s, 4/4 chat completions OK (8 tokens each), log marker static direct frontend assembled without a distributed runtime workers=2 reachable=2, clean exit on SIGINT.
  • cargo check --workspace --all-targets, cargo clippy --workspace --all-targets, bindings cargo check: clean. docs_lint.py: 0 errors.

Limitations: membership is fixed for the process lifetime; aggregated only (disaggregated, encoders, and worker churn use the runtime-backed frontend). Dependency on dynamo-runtime remains at compile time (pipeline types); removing it is Phase 5.

Add `StaticDirectEngine` (`lib/llm/src/kv_router/static_direct.rs`): a
frontend engine that serves one model with no `DistributedRuntime`. Worker
membership comes from a workers file (the selection service's
`--workers-file` shape), KV-aware selection runs on an embedded
`SelectionService` that subscribes to the workers' ZMQ KV events itself
(load-based when no worker publishes events), and admitted requests go to
each worker's direct engine endpoint through the installed
`DirectEngineFactory`. Bookings are freed when the response stream ends.

`entrypoint::input::http::run_static` serves an in-process engine over HTTP
without a runtime; the shared builder now takes `Option<&DistributedRuntime>`
so the runtime-backed path is unchanged. Python: `make_static_engine` and
`run_static_input` bindings, `dynamo.frontend --static-workers-file`
(`DYN_FRONTEND_STATIC_WORKERS_FILE`), which skips runtime construction
entirely. Requires `--model-path` (the model card is read locally) and
`DYN_ROUTER_DIRECT_DISPATCH`.

`WorkerRequest` derives `Clone`. Test: the mock vLLM gRPC server is named in
a workers file, the assembly books it, streams three tokens, and frees the
booking, with no runtime in the process. Docs cover the flag and the layout.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani

Copy link
Copy Markdown
Owner Author

Closed while Stack 1 (#45 to #53) and Stack 2 (#40) go upstream; the branch stays for later stacks (direct dispatch, catalog leases, runtime features). See #33.

This branch had an error being deployed

1 failed deployment
external_collaborator — 0a7443ed Deployed Sep 3, 2026 by ishandhanani via ok-to-test #46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat frontend router

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant