Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/filters.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,7 @@ sglang:
- 'examples/backends/sglang/**'
- 'components/src/dynamo/sglang/**'
- 'components/src/dynamo/sglang_sidecar/**'
- 'lib/sglang-sidecar/**'
- 'container/templates/sglang_*'
- 'container/deps/sglang/**'
- '!**/*.md'
Expand Down
2 changes: 1 addition & 1 deletion lib/sglang-sidecar/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ authors.workspace = true
license.workspace = true
homepage.workspace = true
repository.workspace = true
description = "Dynamo SGLang sidecar backend — drives an out-of-process SGLang engine over its native gRPC service through the backend-common LLMEngine trait."
description = "Dynamo SGLang sidecar for an out-of-process engine exposed through its native gRPC service."

[package.metadata.cargo-machete]
# Referenced by the protobuf client generated into OUT_DIR at build time.
Expand Down
53 changes: 53 additions & 0 deletions lib/sglang-sidecar/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# SGLang sidecar executable

`dynamo-sglang-sidecar` connects Dynamo's unified worker lifecycle to an
out-of-process SGLang engine through SGLang's native gRPC service.

Build and run it directly from the Dynamo workspace:

```bash
cargo build --release -p dynamo-sglang-sidecar
./target/release/dynamo-sglang-sidecar \
--sglang-endpoint http://127.0.0.1:30001
```

Distribution and container packaging for the standalone executable are
intentionally deferred to a follow-up change.

## SGLang-managed contract

An SGLang launcher can supervise the executable directly after its native gRPC
listener is ready:

```bash
python3 -m sglang.launch_server \
<args> \
--grpc-port 30001 \
--sidecar-executable dynamo-sglang-sidecar
```

The corresponding SGLang implementation should:

1. Resolve the executable without invoking a shell.
2. Spawn `dynamo-sglang-sidecar --sglang-endpoint <loopback-url>` only after
native gRPC is listening. Preserve the parent environment so Dynamo's
namespace, discovery, and observability settings reach the worker.
3. Keep the executable as the directly supervised child. A spawn wrapper may
install SGLang's parent-death handling and then call `os.execvp`, which
preserves the child PID across the exec.
4. Treat an unexpected or non-zero child exit as fatal to the SGLang server.
5. On shutdown, send `SIGTERM`, wait for Dynamo's graceful lifecycle, and then
kill the remaining process tree if the deadline expires.

The supervisor timeout must be configurable. Its default must exceed Dynamo's
combined release-mode shutdown budget: the 5-second
`DYN_GRACEFUL_SHUTDOWN_GRACE_PERIOD_SECS` default plus the 30-second
`DYN_WORKER_GRACEFUL_SHUTDOWN_TIMEOUT` default. A 40-second default leaves a
small supervision margin. Operators that increase either Dynamo value must
increase the SGLang timeout as well.

SGLang's gRPC discovery response supplies the model identity and aggregated,
prefill, or decode role, so the managed executable needs only the endpoint
argument. Prefill deployments may additionally set
`SGLANG_DISAGGREGATION_BOOTSTRAP_HOST` when the discovered address is not
routable from decode workers.
18 changes: 15 additions & 3 deletions lib/sglang-sidecar/src/args.rs
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

//! Command-line arguments and transport configuration for the SGLang gRPC sidecar.
//! Command-line arguments and transport configuration for the SGLang sidecar.

use std::path::PathBuf;
use std::time::Duration;
Expand All @@ -10,7 +10,7 @@ use std::time::Duration;
#[derive(clap::Parser, Debug, Clone)]
#[command(
name = "dynamo-sglang-sidecar",
about = "Dynamo SGLang sidecar — drives an out-of-process SGLang native gRPC server."
about = "Dynamo sidecar for an out-of-process SGLang native gRPC server."
)]
pub struct Args {
/// `host:port` (or URL) of SGLang's native `sglang.runtime.v1` service.
Expand Down Expand Up @@ -120,7 +120,19 @@ pub fn normalize_endpoint(raw: &str) -> Result<String, String> {

#[cfg(test)]
mod tests {
use super::normalize_endpoint;
use super::{Args, normalize_endpoint};
use clap::Parser;

#[test]
fn parses_sglang_managed_executable_args() {
let args = Args::try_parse_from([
"dynamo-sglang-sidecar",
"--sglang-endpoint",
"http://127.0.0.1:30001",
])
.unwrap();
assert_eq!(args.sglang_endpoint, "http://127.0.0.1:30001");
}

#[test]
fn normalizes_bare_and_grpc_endpoints() {
Expand Down
4 changes: 4 additions & 0 deletions lib/sglang-sidecar/src/engine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@ impl SglangSidecarEngine {
}
.map_err(|err| client::invalid_arg(err.to_string()))?;

Self::from_parsed_args(args)
}

pub fn from_parsed_args(args: Args) -> Result<(Self, WorkerConfig), DynamoError> {
let endpoint = normalize_endpoint(&args.sglang_endpoint).map_err(client::invalid_arg)?;
let transport = args.transport();
let discovery = bootstrap_discover(&endpoint, &transport)?;
Expand Down
13 changes: 9 additions & 4 deletions lib/sglang-sidecar/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,18 @@

//! Entry point for the `dynamo-sglang-sidecar` binary.
//!
//! Mirrors the mocker backend: bootstrap-discover the engine in `from_args`
//! (building the [`WorkerConfig`](dynamo_backend_common::WorkerConfig) `run`
//! needs synchronously), then hand the engine to the shared runtime harness.
//! Mirrors the mocker backend: parse the process CLI with clap, bootstrap-discover
//! the engine in `from_parsed_args` (building the
//! [`WorkerConfig`](dynamo_backend_common::WorkerConfig) `run` needs
//! synchronously), then hand the engine to the shared runtime harness.
Comment thread
connorcarpenter15 marked this conversation as resolved.

use std::sync::Arc;

use clap::Parser;
use dynamo_sglang_sidecar::args::Args;

fn main() -> anyhow::Result<()> {
let (engine, config) = dynamo_sglang_sidecar::SglangSidecarEngine::from_args(None)?;
let args = Args::parse();
let (engine, config) = dynamo_sglang_sidecar::SglangSidecarEngine::from_parsed_args(args)?;
dynamo_backend_common::run(Arc::new(engine), config)
}
21 changes: 21 additions & 0 deletions lib/sglang-sidecar/tests/executable.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

use std::process::Command;

#[test]
fn executable_exposes_sglang_managed_contract() {
let output = Command::new(env!("CARGO_BIN_EXE_dynamo-sglang-sidecar"))
.arg("--help")
.output()
.expect("run dynamo-sglang-sidecar --help");

assert!(
output.status.success(),
"--help failed: {}",
String::from_utf8_lossy(&output.stderr)
);
let stdout = String::from_utf8(output.stdout).expect("help output is UTF-8");
assert!(stdout.contains("--sglang-endpoint"));
assert!(stdout.contains("SGLANG_GRPC_ENDPOINT"));
}
Loading