Skip to content

feat: add native gRPC sidecar module launcher - #31076

Merged
ishandhanani merged 7 commits into
mainfrom
codex/enable-dynamo-official
Jul 22, 2026
Merged

ishandhanani merged 7 commits into
mainfrom
codex/enable-dynamo-official

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Adds an optional --sidecar <python.module> to launch a local companion process after SGLang's native gRPC listener starts. This removes the Dynamo-specific server contract while preserving SGLang-owned child supervision and lets provider packages remain optional.

How This Was Implemented

  • Replaced the Dynamo-specific flag and launcher with a module contract: main(argv) receives --sglang-endpoint <local-endpoint>.
  • Kept process spawn, parent-death handling, crash watchdog, bounded stop, and forced tree cleanup in SGLang.
  • Starts the sidecar only after native gRPC binds; a sidecar stop failure is logged without skipping gRPC shutdown.
  • Validates a non-empty selector and the existing native-gRPC-only launch constraints.
Walkthrough

Mental model

The selected module is a locally managed process, not an SGLang plugin or registry entry. SGLang supplies only its loopback native-gRPC endpoint; the provider owns its arguments and runtime behavior.

flowchart LR
  A["launch_server --sidecar package.module"] --> B["Native gRPC listener binds"]
  B --> C["SGLang Sidecar supervisor"]
  C --> D["Spawned child imports package.module"]
  D --> E["module.main(--sglang-endpoint local endpoint)"]
  E --> F["Provider connects to native gRPC"]
  C --> G["Watchdog / parent-death handling / bounded stop"]
Loading

Lifecycle and failure behavior

The module is imported only in the spawned child, preserving optional provider dependencies and avoiding parent-process import side effects. An import or main(argv) contract failure terminates the required child and is handled by the existing watchdog; on normal shutdown SGLang stops the child before shutting down native gRPC.

Boundaries

This is intentionally a same-process-namespace sidecar interface: wildcard SGLang binds are converted to loopback for the child. It does not add provider aliases, discovery, provider-specific arguments, or a registry.

Validation

  • .venv/bin/python -m pytest -q test/registered/unit/server_args/test_server_args.py::TestGrpcServerArgs — 16 passed.
  • uvx pre-commit run --files python/sglang/srt/entrypoints/sidecar.py python/sglang/srt/entrypoints/http_server.py python/sglang/srt/server_args.py test/registered/unit/server_args/test_server_args.py — passed.
  • Live Qwen3-0.6B launch with --sidecar dynamo.sglang_sidecar.main: native gRPC discovery succeeded, Dynamo established eight gRPC connections, and served sglang-sidecar-e2e.backend.generate.

CI States

Latest PR Test (Base): 🚫 Run #29924027291
Latest PR Test (Extra): ❌ Run #29924026692

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@ishandhanani
ishandhanani marked this pull request as ready for review July 13, 2026 22:46
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@ishandhanani
ishandhanani force-pushed the codex/enable-dynamo-official branch from 6eb853b to 197b70a Compare July 13, 2026 22:52
@ishandhanani
ishandhanani marked this pull request as draft July 13, 2026 22:52
@ishandhanani
ishandhanani marked this pull request as ready for review July 14, 2026 06:54
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@ishandhanani

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci

Comment thread python/sglang/srt/entrypoints/sidecar.py Outdated
Comment thread python/sglang/srt/entrypoints/sidecar.py Outdated
Comment thread test/registered/unit/server_args/test_server_args.py Outdated

@ishandhanani ishandhanani left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@connorcarpenter15 PTAL and review this PR

Comment thread python/sglang/srt/entrypoints/sidecar.py Outdated
Comment thread python/sglang/srt/server_args.py Outdated
Comment thread test/registered/unit/server_args/test_server_args.py
Comment thread test/registered/unit/server_args/test_server_args.py
ishandhanani and others added 6 commits July 22, 2026 10:17
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
@ishandhanani
ishandhanani force-pushed the codex/enable-dynamo-official branch from 4277853 to 9ac19ba Compare July 22, 2026 10:18
@ishandhanani

Copy link
Copy Markdown
Collaborator Author

Relevant CI has passed. Getting this in as it does not affect any GPU pieces at all

@ishandhanani
ishandhanani merged commit 21065bc into main Jul 22, 2026
82 of 104 checks passed
@ishandhanani
ishandhanani deleted the codex/enable-dynamo-official branch July 22, 2026 13:39
Qiaolin-Yu pushed a commit that referenced this pull request Jul 22, 2026
…32074)

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Connor Carpenter <connorc@nvidia.com>
Zhylkaaa pushed a commit to Zhylkaaa/sglang that referenced this pull request Jul 29, 2026
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Connor Carpenter <connorc@nvidia.com>
jinzhenfan pushed a commit to jinzhenfan/sglang that referenced this pull request Jul 29, 2026
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Connor Carpenter <connorc@nvidia.com>
Chronostasys pushed a commit to MindLab-Research/sglang that referenced this pull request Aug 24, 2026
…ect#31076) (sgl-project#32074)

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Connor Carpenter <connorc@nvidia.com>
OrangeRedeng pushed a commit to OrangeRedeng/sglang that referenced this pull request Aug 24, 2026
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Connor Carpenter <connorc@nvidia.com>
jakki-amd pushed a commit to jakki-amd/sglang that referenced this pull request Sep 9, 2026
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Connor Carpenter <connorc@nvidia.com>
Atituiset pushed a commit to Atituiset/sglang that referenced this pull request Sep 10, 2026
Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
Signed-off-by: Connor Carpenter <connorc@nvidia.com>
Co-authored-by: Connor Carpenter <connorc@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants