feat: Support static workers, run without etcd. #2281
Conversation
grahamking
requested review from
a team,
GuanLuo,
PeaBrane,
alec-flowers,
biswapanda,
ishandhanani,
jthomson04,
kkranen,
nnshah1,
paulhendricks,
piotrm-nvidia,
ptarasiewiczNV,
rmccorm4,
ryanolson,
tanmayv25,
tedzhouhk and
tmonty12
as code owners
WalkthroughThis update introduces a "static mode" for Dynamo's distributed runtime, allowing the frontend and worker nodes to operate without etcd-based auto-discovery. New CLI flags and validation logic support specifying static endpoints, model names, and paths. Core engine, runtime, and entrypoint logic are extended to recognize and properly handle static configurations. Documentation and Python/Rust examples are updated to demonstrate and explain static mode usage. Changes
Sequence Diagram(s)Static Mode Engine Initialization (Frontend to Worker)sequenceDiagram
participant User
participant Frontend (Static Mode)
participant DistributedRuntime
participant Worker (Static)
User->>Frontend (Static Mode): Launch with --static-endpoint, --model-name, --model-path
Frontend (Static Mode)->>DistributedRuntime: Initialize (static config)
Frontend (Static Mode)->>Worker (Static): Route request to static endpoint
Worker (Static)->>Frontend (Static Mode): Process/generate response
Frontend (Static Mode)->>User: Return response
EngineConfig Selection (Rust)sequenceDiagram
participant CLI
participant Flags Parser
participant Runtime
participant Engine Selector
CLI->>Flags Parser: Parse CLI arguments (auto/static)
Flags Parser->>Runtime: Initialize (static_worker flag)
Runtime->>Engine Selector: Request EngineConfig
Engine Selector->>Runtime: Return StaticRemote/Auto/StaticFull/StaticCore config
Estimated code review effort🎯 4 (Complex) | ⏱️ ~35 minutes Poem
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (4)
docs/guides/dynamo_run.md (1)
150-153: Add language specification to fenced code blockThe static analysis tool correctly identified a missing language specification for the fenced code block.
-``` +```bash Node 1: dynamo-run in=http out=dyn://dynamo.backend.generate --model-name Qwen3-0.6B-Q8_0.gguf --model-path ~/llms/Qwen3-0.6B Node 2: dynamo-run in=dyn://dynamo.backend.generate out=llamacpp ~/llms/Qwen3-0.6B-Q8_0.gguf --static-worker --context-length 4096 -``` +```lib/llm/src/engines.rs (1)
440-524: Consider refactoring to reduce code duplication.Both
build_chat_completionsandbuild_completionsshare nearly identical pipeline construction logic. While the current implementation is correct, consider extracting the common pipeline building logic into a generic helper function to improve maintainability.The functions could share a common pipeline builder that accepts the frontend as a parameter, since that's the only component that differs between them. Would you like me to propose a refactored implementation that reduces this duplication?
lib/bindings/python/examples/hello_world/server_sglang_static.py (2)
29-36: Add type hints to Config class fields.Consider adding type hints to improve code readability and enable better IDE support.
class Config: """Command line parameters or defaults""" - namespace: str - component: str - endpoint: str - model: str + namespace: str + component: str + endpoint: str + model: str + + def __init__(self): + self.namespace: str = "" + self.component: str = "" + self.endpoint: str = "" + self.model: str = ""
139-139: Add newline at end of file.Add a newline character at the end of the file to follow Unix conventions.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (15)
components/frontend/src/dynamo/frontend/main.py(4 hunks)docs/guides/dynamo_run.md(4 hunks)launch/dynamo-run/src/flags.rs(4 hunks)launch/dynamo-run/src/lib.rs(5 hunks)launch/dynamo-run/src/main.rs(2 hunks)launch/dynamo-run/src/opt.rs(3 hunks)lib/bindings/python/examples/hello_world/server_sglang_static.py(1 hunks)lib/bindings/python/examples/hello_world/server_static.py(0 hunks)lib/bindings/python/rust/llm/entrypoint.rs(3 hunks)lib/llm/src/discovery/watcher.rs(2 hunks)lib/llm/src/engines.rs(2 hunks)lib/llm/src/entrypoint.rs(2 hunks)lib/llm/src/entrypoint/input/common.rs(5 hunks)lib/llm/src/entrypoint/input/endpoint.rs(2 hunks)lib/llm/src/entrypoint/input/http.rs(3 hunks)
💤 Files with no reviewable changes (1)
- lib/bindings/python/examples/hello_world/server_static.py
🧰 Additional context used
🧠 Learnings (10)
📓 Common learnings
Learnt from: nnshah1
PR: ai-dynamo/dynamo#1444
File: tests/fault_tolerance/utils/metrics.py:30-32
Timestamp: 2025-07-01T13:55:03.940Z
Learning: The `@dynamo_worker()` decorator in the dynamo codebase returns a wrapper that automatically injects the `runtime` parameter before calling the wrapped function. This means callers only need to provide the non-runtime parameters, while the decorator handles injecting the runtime argument automatically. For example, a function with signature `async def get_metrics(runtime, log_dir)` decorated with `@dynamo_worker()` can be called as `get_metrics(log_dir)` because the decorator wrapper injects the runtime parameter.
Learnt from: GuanLuo
PR: ai-dynamo/dynamo#1371
File: examples/llm/benchmarks/vllm_multinode_setup.sh:18-25
Timestamp: 2025-06-05T01:46:15.509Z
Learning: In multi-node setups with head/worker architecture, the head node typically doesn't need environment variables pointing to its own services (like NATS_SERVER, ETCD_ENDPOINTS) because local processes can access them via localhost. Only worker nodes need these environment variables to connect to the head node's external IP address.
Learnt from: fsaady
PR: ai-dynamo/dynamo#1730
File: examples/sglang/slurm_jobs/scripts/worker_setup.py:230-244
Timestamp: 2025-07-03T10:14:30.570Z
Learning: In examples/sglang/slurm_jobs/scripts/worker_setup.py, background processes (like nats-server, etcd) are intentionally left running even if later processes fail. This design choice allows users to manually connect to nodes and debug issues without having to restart the entire SLURM job from scratch, providing operational flexibility for troubleshooting in cluster environments.
📚 Learning: the asyncenginecontextprovider trait in lib/runtime/src/engine.rs was intentionally changed from `se...
Learnt from: ryanolson
PR: ai-dynamo/dynamo#1919
File: lib/runtime/src/engine.rs:168-168
Timestamp: 2025-07-14T21:25:56.930Z
Learning: The AsyncEngineContextProvider trait in lib/runtime/src/engine.rs was intentionally changed from `Send + Sync + Debug` to `Send + Debug` because the Sync bound was overly constraining. The trait should only require Send + Debug as designed.
Applied to files:
lib/bindings/python/rust/llm/entrypoint.rslib/llm/src/entrypoint/input/http.rslib/llm/src/entrypoint/input/endpoint.rslib/llm/src/entrypoint.rslib/llm/src/entrypoint/input/common.rslaunch/dynamo-run/src/lib.rslib/llm/src/engines.rs
📚 Learning: the `@dynamo_worker()` decorator in the dynamo codebase returns a wrapper that automatically injects...
Learnt from: nnshah1
PR: ai-dynamo/dynamo#1444
File: tests/fault_tolerance/utils/metrics.py:30-32
Timestamp: 2025-07-01T13:55:03.940Z
Learning: The `@dynamo_worker()` decorator in the dynamo codebase returns a wrapper that automatically injects the `runtime` parameter before calling the wrapped function. This means callers only need to provide the non-runtime parameters, while the decorator handles injecting the runtime argument automatically. For example, a function with signature `async def get_metrics(runtime, log_dir)` decorated with `@dynamo_worker()` can be called as `get_metrics(log_dir)` because the decorator wrapper injects the runtime parameter.
Applied to files:
docs/guides/dynamo_run.mdlaunch/dynamo-run/src/lib.rs
📚 Learning: in examples/sglang/slurm_jobs/scripts/worker_setup.py, background processes (like nats-server, etcd)...
Learnt from: fsaady
PR: ai-dynamo/dynamo#1730
File: examples/sglang/slurm_jobs/scripts/worker_setup.py:230-244
Timestamp: 2025-07-03T10:14:30.570Z
Learning: In examples/sglang/slurm_jobs/scripts/worker_setup.py, background processes (like nats-server, etcd) are intentionally left running even if later processes fail. This design choice allows users to manually connect to nodes and debug issues without having to restart the entire SLURM job from scratch, providing operational flexibility for troubleshooting in cluster environments.
Applied to files:
docs/guides/dynamo_run.md
📚 Learning: in multi-node setups with head/worker architecture, the head node typically doesn't need environment...
Learnt from: GuanLuo
PR: ai-dynamo/dynamo#1371
File: examples/llm/benchmarks/vllm_multinode_setup.sh:18-25
Timestamp: 2025-06-05T01:46:15.509Z
Learning: In multi-node setups with head/worker architecture, the head node typically doesn't need environment variables pointing to its own services (like NATS_SERVER, ETCD_ENDPOINTS) because local processes can access them via localhost. Only worker nodes need these environment variables to connect to the head node's external IP address.
Applied to files:
docs/guides/dynamo_run.md
📚 Learning: in lib/llm/src/kv_router/scoring.rs, the user prefers to keep the panic behavior when calculating lo...
Learnt from: PeaBrane
PR: ai-dynamo/dynamo#1285
File: lib/llm/src/kv_router/scoring.rs:58-63
Timestamp: 2025-05-30T06:38:09.630Z
Learning: In lib/llm/src/kv_router/scoring.rs, the user prefers to keep the panic behavior when calculating load_avg and variance with empty endpoints rather than adding guards for division by zero. They want the code to fail fast on this error condition.
Applied to files:
lib/llm/src/entrypoint/input/endpoint.rs
📚 Learning: in lib/runtime/src/component/client.rs, the current mutex usage in get_or_create_dynamic_instance_so...
Learnt from: grahamking
PR: ai-dynamo/dynamo#1962
File: lib/runtime/src/component/client.rs:270-273
Timestamp: 2025-07-16T12:41:12.543Z
Learning: In lib/runtime/src/component/client.rs, the current mutex usage in get_or_create_dynamic_instance_source is temporary while evaluating whether the mutex can be dropped entirely. The code currently has a race condition between try_lock and lock().await, but this is acknowledged as an interim state during the performance optimization process.
Applied to files:
lib/llm/src/entrypoint/input/common.rslaunch/dynamo-run/src/lib.rs
📚 Learning: in lib/llm/src/kv_router/scoring.rs, peabrane prefers panic-based early failure over result-based er...
Learnt from: PeaBrane
PR: ai-dynamo/dynamo#1392
File: lib/llm/src/kv_router/scoring.rs:35-46
Timestamp: 2025-06-05T01:02:15.318Z
Learning: In lib/llm/src/kv_router/scoring.rs, PeaBrane prefers panic-based early failure over Result-based error handling for the worker_id() method to catch invalid data early during development.
Applied to files:
launch/dynamo-run/src/flags.rs
📚 Learning: in lib/llm/src/kv_router/publisher.rs, the functions `create_stored_blocks` and `create_stored_block...
Learnt from: alec-flowers
PR: ai-dynamo/dynamo#1181
File: lib/llm/src/kv_router/publisher.rs:379-425
Timestamp: 2025-05-29T00:02:35.018Z
Learning: In lib/llm/src/kv_router/publisher.rs, the functions `create_stored_blocks` and `create_stored_block_from_parts` are correctly implemented and not problematic duplications of existing functionality elsewhere in the codebase.
Applied to files:
lib/llm/src/engines.rslib/llm/src/discovery/watcher.rs
📚 Learning: the codebase uses async-nats version 0.40, not the older nats crate. error handling should use async...
Learnt from: kthui
PR: ai-dynamo/dynamo#1424
File: lib/runtime/src/pipeline/network/egress/push_router.rs:204-209
Timestamp: 2025-06-13T22:07:24.843Z
Learning: The codebase uses async-nats version 0.40, not the older nats crate. Error handling should use async_nats::error::Error variants, not nats::Error variants.
Applied to files:
lib/llm/src/engines.rslib/llm/src/discovery/watcher.rs
🪛 markdownlint-cli2 (0.17.2)
docs/guides/dynamo_run.md
150-150: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)
- GitHub Check: Build and Test - vllm
- GitHub Check: pre-merge-rust (lib/runtime/examples)
- GitHub Check: pre-merge-rust (.)
- GitHub Check: pre-merge-rust (lib/bindings/python)
🔇 Additional comments (16)
lib/bindings/python/rust/llm/entrypoint.rs (1)
26-26: LGTM: Static engine type additionThe new
Static = 4variant properly extends theEngineTypeenum to support static workers.lib/llm/src/entrypoint/input/endpoint.rs (2)
45-62: LGTM: Conditional model attachment for static configurationsThe conditional model attachment logic correctly prevents static workers from registering with etcd when
is_staticis true. This aligns with the PR objective of supporting workers without etcd dependency.
91-93: LGTM: Appropriate error handling for StaticRemoteThe panic with a clear error message correctly prevents misuse of
StaticRemoteconfigurations in endpoint contexts, as these are intended only for frontend nodes.docs/guides/dynamo_run.md (2)
96-96: LGTM: Consistent documentation updates for output parameter changesThe replacement of
out=dynwithout=autothroughout the documentation correctly reflects the CLI changes and maintains consistency.Also applies to: 144-144, 208-208, 583-583
146-156: LGTM: Clear documentation for static worker functionalityThe new section provides excellent documentation for the static worker feature, including:
- Clear examples for both frontend and worker nodes
- Explanation of the trade-offs (no etcd dependency vs manual configuration)
- Required parameters for static mode
This will help users understand when and how to use static workers.
lib/llm/src/entrypoint.rs (3)
39-40: LGTM: StaticRemote variant additionThe new
StaticRemotevariant properly represents remote networked engines known at startup, complementing the existingDynamicvariant for etcd-discovered engines. The documentation comment clearly explains the distinction.
46-46: LGTM: is_static field additionThe addition of the
is_staticboolean field toStaticFullandStaticCorevariants enables proper differentiation between static and dynamic configurations within these engine types.Also applies to: 53-53
62-62: LGTM: local_model method updateThe
local_modelmethod correctly handles the newStaticRemotevariant, maintaining consistency with the existing pattern matching.launch/dynamo-run/src/main.rs (2)
27-27: LGTM: Updated usage string for new output optionsThe usage string correctly reflects the new
autoand static endpoint (dyn://<path>) output options, as well as the new--static-workerflag. This provides clear guidance to users on the available options.
137-137: LGTM: Updated dynamic output detectionThe logic correctly identifies both
Output::AutoandOutput::Static(_)as dynamic outputs since both utilize the distributed runtime, distinguishing them from local engine outputs.lib/llm/src/entrypoint/input/http.rs (1)
52-89: LGTM! Clean implementation of static remote engine support.The static remote configuration properly initializes the distributed runtime, retrieves the endpoint client, and builds the necessary engines. The conditional KV chooser creation based on router mode is handled correctly.
launch/dynamo-run/src/opt.rs (1)
15-22: Clear documentation and naming improvements.The rename from
DynamictoAutobetter describes the auto-discovery behavior, and the newStaticvariant is well-documented with its requirements clearly stated.lib/llm/src/discovery/watcher.rs (1)
204-233: Excellent refactoring to centralize engine construction.The extraction of pipeline construction logic to the
enginesmodule helper functions reduces code duplication and improves maintainability. The KV chooser logic is correctly preserved and shared between both engines.launch/dynamo-run/src/lib.rs (1)
102-146: LGTM! Consistent implementation of static engine support.The changes properly handle the new
Output::AutoandOutput::Staticvariants, and consistently set theis_staticfield across all engine configurations based on thestatic_workerflag.lib/llm/src/entrypoint/input/common.rs (1)
94-140: LGTM! Well-structured implementation of StaticRemote engine configuration.The implementation properly:
- Initializes distributed runtime with static configuration
- Handles KV routing mode with appropriate chooser creation
- Uses the centralized engine building helper
- Provides clear logging for debugging
components/frontend/src/dynamo/frontend/main.py (1)
39-67: LGTM! Robust validation for static mode arguments.The validation functions properly ensure:
- Static endpoints follow the required
namespace.component.endpointformat- Model names are non-empty strings
- Model paths point to existing directories
Example: ``` python -m dynamo.frontend \ --model-name Qwen3-0.6B-Q8_0.gguf --model-path ~/llms/Qwen3-0.6B \ --static-endpoint dynamo.backend.generate ``` The worker is any engine built with `@dynamo_worker(static=True)` and listening on endpoint matching `--static-endpoint`. See `lib/bindings/python/examples/hello_world/server_sglang_static.py`. Without etcd we cannot auto-discover information about the model, so we have to pass `--model-name` and `--model-path`. For development this is also supported in `dynamo-run`: - Node 1: `dynamo-run in=http out=dyn://dynamo.backend.generate --model-name Qwen3-0.6B-Q8_0.gguf --model-path ~/llms/Qwen3-0.6B` - Node 2: `dynamo-run in=dyn://dynamo.backend.generate out=llamacpp ~/llms/Qwen3-0.6B-Q8_0.gguf --static-worker --context-length 4096`
grahamking
force-pushed
the
gk-remote-static
branch
from
820e2a1 to
3136db4
Compare
|
Lgtm - it's already merged |
Example:
The worker is any engine wrapped with
@dynamo_worker(static=True)and listening on endpoint matching--static-endpoint. Seelib/bindings/python/examples/hello_world/server_sglang_static.py.Without etcd we cannot auto-discover information about the model, so we have to pass
--model-nameand--model-path.For development this is also supported in
dynamo-run:dynamo-run in=http out=dyn://dynamo.backend.generate --model-name Qwen3-0.6B-Q8_0.gguf --model-path ~/llms/Qwen3-0.6Bdynamo-run in=dyn://dynamo.backend.generate out=llamacpp ~/llms/Qwen3-0.6B-Q8_0.gguf --static-worker --context-length 4096Summary by CodeRabbit
New Features
Documentation
Refactor
Bug Fixes
Chores