Skip to content

feat(sglang): add native gRPC sidecar with disaggregation - #3

Closed
connorcarpenter15 wants to merge 8 commits into
mainfrom
feat/sglang-sidecar-grpc
Closed

feat(sglang): add native gRPC sidecar with disaggregation#3
connorcarpenter15 wants to merge 8 commits into
mainfrom
feat/sglang-sidecar-grpc

Conversation

@connorcarpenter15

@connorcarpenter15 connorcarpenter15 commented Jul 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Overview:

Add a standalone Dynamo SGLang sidecar that communicates with SGLang through its native gRPC server. The sidecar supports both aggregated serving and prefill/decode disaggregation while keeping the SGLang engine in a separate process.

This is currently based on the gRPC/disaggregation handling in sgl-project/sglang#30440.

Details:

  • Add the dynamo-sglang-sidecar Rust binary and library.
  • Fetch the native gRPC contract from a pinned SGLang commit, verify its SHA-256, and generate the client during the build without vendoring the proto.
  • Map Dynamo requests to SGLang generation parameters, including guided decoding, logprobs, LoRA, and parser metadata.
  • Support prefill and decode worker modes and the bootstrap metadata required for KV transfer.
  • Preserve bootstrap room precision across the sidecar boundary.
  • Add aggregated and disaggregated launch scripts.
  • Register the new crate in the Dynamo workspace.

Where should the reviewer start?

  • lib/sglang-sidecar/src/engine.rs for request handling, worker registration, and disaggregated execution.
  • lib/sglang-sidecar/src/client.rs for the native SGLang gRPC client.
  • examples/backends/sglang/launch/sidecar_disagg.sh for the prefill/decode topology.

Validation:

  • Built the SGLang native gRPC extension from PR #30440 at cf2aab82a3c5342f6bc89b7bae47a078ace1149c.
  • Built and tested the pre-rebase Dynamo head e2311713388f5708405766ddea9bfecdd8c694b3.
  • Rebased the original seven-commit patch series without conflicts onto current main.
  • Verified both the default upstream download and local SGLang checkout override produce the pinned proto digest.
  • Ran cargo check -p dynamo-sglang-sidecar successfully through the local checkout source path.
  • Current head: ddb22bf002a8def8e1ebcaa0b429aaac5607663f.
  • Ran an end-to-end disaggregated smoke test on two NVIDIA H200 GPUs with Qwen/Qwen3-0.6B.
  • Verified both prefill and decode sidecars started, NIXL initialized with the UCX backend, and a routed OpenAI chat request returned HTTP 200 with a non-empty completion.
  • SLURM job 3023079 completed with exit code 0:0.

Related Issues

🚫 This PR is NOT linked to an issue:

  • Confirmed — no related issue

github-advanced-security[bot]
github-advanced-security Bot found potential problems Jul 8, 2026
View reviewed changes
Comment thread components/src/dynamo/common/backend/worker.py Fixed
Comment thread components/src/dynamo/common/backend/worker.py Fixed
Comment thread components/src/dynamo/common/metadata_upload.py Fixed
Comment thread .agents/skills/dynamo-frontend-benchmark/scripts/analyze_folded.py Fixed
Comment thread .agents/skills/dynamo-frontend-benchmark/scripts/extract_throughput.py Fixed
Comment thread components/src/dynamo/common/backend/engine.py Fixed
Comment thread components/src/dynamo/common/backend/engine.py Fixed
Comment thread components/src/dynamo/sglang/request_handlers/handler_base.py Fixed
Comment thread components/src/dynamo/planner/plugins/proto/v1/plugin_pb2_grpc.py Fixed
Comment thread components/src/dynamo/planner/plugins/proto/v1/plugin_pb2.py Fixed
Connor Carpenter added 7 commits July 8, 2026 09:57
Connor Carpenter
feat(sglang): OpenEngine sidecar worker (dynamo-sglang-sidecar)
8047c64 
Add a sidecar integration for SGLang in Dynamo: a native SGLang engine
(`sglang.launch_server --openengine-port`) runs the model, and a separate
Dynamo worker drives it over the vendor-neutral OpenEngine v1 gRPC
contract — the same process-separation shape as the vLLM sidecar, over
OpenEngine.

New crate `lib/sglang-sidecar` (`[[bin]] dynamo-sglang-sidecar`),
a `dynamo_backend_common::LLMEngine` impl with no `sglang` dependency.
It is endpoint-only configured (`--openengine-endpoint`); model, role,
parallelism, KV block config, and KV event sources are all discovered
from the engine over OpenEngine RPCs. Disaggregation reuses the shared
`disagg` helper, relaying SGLang's bootstrap triple through the typed
`KvSessionRef.attributes_struct`.

backend-common re-exports `MultimodalData`/`MultimodalDataMap`/
`RoutingHints` for the sidecar; the crate is registered in the workspace.
Aggregated launch script under examples/backends/sglang/launch.

The in-process backend (`dynamo.sglang`) is unchanged; the sidecar lives
alongside it.

Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Connor Carpenter
feat(sglang): map guided decoding, logprobs, LoRA, parser metadata
12524a2 
Wire the sidecar's OpenEngine client to the contract additions (vendored
proto is additive within openengine.v1):

- build_generate_request fills GenerateRequest.guided (from
  sampling_options.guided_decoding), lora_name (routing hint), and the
  logprobs controls (output_options.logprobs / prompt_logprobs).
- The generate stream maps TokenOutput.logprobs / top_logprobs back into
  LLMEngineOutput.log_probs / top_logprobs.
- from_args copies the engine-advertised ModelInfo.reasoning_parser /
  tool_call_parser into WorkerConfig so registration carries them to the
  frontend (tool-call / reasoning parsing for sidecar-served models).
- backend-common re-exports TopLogprob for the mapping.

Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Connor Carpenter
fix(sglang-sidecar): preserve bootstrap room precision
88169e8 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Connor Carpenter
refactor(sidecar): remove deprecated OpenEngine fields
4ef3b8a 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Connor Carpenter
chore(openengine): sync LoRA lifecycle protocol
592ae59 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Connor Carpenter
feat(sglang): use native gRPC sidecar protocol
b4bd45b 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Connor Carpenter
feat(sglang): enable native gRPC sidecar disaggregation
59e7272 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@connorcarpenter15
connorcarpenter15 force-pushed the feat/sglang-sidecar-grpc branch from e231171 to 59e7272 Compare July 8, 2026 16:57
Connor Carpenter
build(sglang): fetch grpc proto from upstream
ddb22bf 
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@connorcarpenter15

connorcarpenter15 commented Jul 8, 2026

Copy link
Copy Markdown
Owner Author

Superseded by ai-dynamo#11408.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

actions backend::sglang backend::trtllm backend::vllm container deployment::k8s documentation Improvements or additions to documentation feat frontend multimodal planner router xpu

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@connorcarpenter15 @github-advanced-security