-
Notifications
You must be signed in to change notification settings - Fork 1.6k
feat(sglang): define sidecar executable contract #11797
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| # SGLang sidecar executable | ||
|
|
||
| `dynamo-sglang-sidecar` connects Dynamo's unified worker lifecycle to an | ||
| out-of-process SGLang engine through SGLang's native gRPC service. | ||
|
|
||
| Build and run it directly from the Dynamo workspace: | ||
|
|
||
| ```bash | ||
| cargo build --release -p dynamo-sglang-sidecar | ||
| ./target/release/dynamo-sglang-sidecar \ | ||
| --sglang-endpoint http://127.0.0.1:30001 | ||
| ``` | ||
|
|
||
| Distribution and container packaging for the standalone executable are | ||
| intentionally deferred to a follow-up change. | ||
|
|
||
| ## SGLang-managed contract | ||
|
|
||
| An SGLang launcher can supervise the executable directly after its native gRPC | ||
| listener is ready: | ||
|
|
||
| ```bash | ||
| python3 -m sglang.launch_server \ | ||
| <args> \ | ||
| --grpc-port 30001 \ | ||
| --sidecar-executable dynamo-sglang-sidecar | ||
| ``` | ||
|
|
||
| The corresponding SGLang implementation should: | ||
|
|
||
| 1. Resolve the executable without invoking a shell. | ||
| 2. Spawn `dynamo-sglang-sidecar --sglang-endpoint <loopback-url>` only after | ||
| native gRPC is listening. Preserve the parent environment so Dynamo's | ||
| namespace, discovery, and observability settings reach the worker. | ||
| 3. Keep the executable as the directly supervised child. A spawn wrapper may | ||
| install SGLang's parent-death handling and then call `os.execvp`, which | ||
| preserves the child PID across the exec. | ||
| 4. Treat an unexpected or non-zero child exit as fatal to the SGLang server. | ||
| 5. On shutdown, send `SIGTERM`, wait for Dynamo's graceful lifecycle, and then | ||
| kill the remaining process tree if the deadline expires. | ||
|
|
||
| The supervisor timeout must be configurable. Its default must exceed Dynamo's | ||
| combined release-mode shutdown budget: the 5-second | ||
| `DYN_GRACEFUL_SHUTDOWN_GRACE_PERIOD_SECS` default plus the 30-second | ||
| `DYN_WORKER_GRACEFUL_SHUTDOWN_TIMEOUT` default. A 40-second default leaves a | ||
| small supervision margin. Operators that increase either Dynamo value must | ||
| increase the SGLang timeout as well. | ||
|
|
||
| SGLang's gRPC discovery response supplies the model identity and aggregated, | ||
| prefill, or decode role, so the managed executable needs only the endpoint | ||
| argument. Prefill deployments may additionally set | ||
| `SGLANG_DISAGGREGATION_BOOTSTRAP_HOST` when the discovered address is not | ||
| routable from decode workers. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| // SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| use std::process::Command; | ||
|
|
||
| #[test] | ||
| fn executable_exposes_sglang_managed_contract() { | ||
| let output = Command::new(env!("CARGO_BIN_EXE_dynamo-sglang-sidecar")) | ||
| .arg("--help") | ||
| .output() | ||
| .expect("run dynamo-sglang-sidecar --help"); | ||
|
|
||
| assert!( | ||
| output.status.success(), | ||
| "--help failed: {}", | ||
| String::from_utf8_lossy(&output.stderr) | ||
| ); | ||
| let stdout = String::from_utf8(output.stdout).expect("help output is UTF-8"); | ||
| assert!(stdout.contains("--sglang-endpoint")); | ||
| assert!(stdout.contains("SGLANG_GRPC_ENDPOINT")); | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.