feat: add SGLang recipe for DeepSeek-V4 - #8704
Merged
Merged
Conversation
Adds Dockerfile and DynamoGraphDeployment YAML for serving DeepSeek-V4-Flash via SGLang on Blackwell GPUs with Dynamo frontend.
KrishnanPrash
commented
Apr 24, 2026
Adds DynamoGraphDeployment YAML for serving DeepSeek-V4-Pro (1.6T) via SGLang on B200x8 (TP8) with Dynamo frontend. Reuses the same container image as the Flash recipe.
KrishnanPrash
changed the title
KrishnanPrash
marked this pull request as ready for review
April 24, 2026 23:53
Contributor
WalkthroughIntroduces infrastructure configuration for deploying DeepSeek-V4 models via SGLang in Dynamo environments. Adds a Dockerfile for building Dynamo runtime with V4 support and parser validation, and two Kubernetes DynamoGraphDeployment manifests for orchestrating V4-Flash and V4-Pro instances with configured tensor parallelism, speculative decoding, and worker GPU constraints. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
♻️ Duplicate comments (1)
recipes/deepseek-v4-flash/sglang/Dockerfile.dsv4-sglang (1)
30-36:⚠️ Potential issue | 🟠 MajorPin the Dynamo source to immutable refs instead of branch + cherry-pick.
This build depends on mutable refs, so it is not reproducible and can fail later if either ref moves or is deleted. Please switch to a stable release branch/commit that already includes both changes (or pin exact SHAs in args).
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@recipes/deepseek-v4-flash/sglang/Dockerfile.dsv4-sglang` around lines 30 - 36, The Dockerfile currently clones the mutable branch "codex/deepseek-v4-parsers", fetches a branch "ishan/investigate-tmrw" and cherry-picks commit 44fb33a5ad53caead03780b5a2a4593b5388c88f which makes the build non-reproducible; instead pin the source to immutable refs by replacing that sequence with a single clone/checkout of an exact commit or tag that already contains both changes (or introduce build ARGs for exact SHAs and checkout those SHAs), e.g., stop using the branch name in git clone and use the specific commit SHA for the repository checkout (reference: git clone -b codex/deepseek-v4-parsers, git fetch origin ishan/investigate-tmrw, git cherry-pick 44fb33a5ad53cae).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@recipes/deepseek-v4-flash/sglang/Dockerfile.dsv4-sglang`:
- Around line 51-90: The final image runs as root because the Dockerfile never
sets a non-root USER; update the final stage (after WORKDIR /workspace and
before ENTRYPOINT/CMD) to create or use a non-root user (e.g., add a user/group,
set ownership of runtime directories like /workspace and any copied artifacts)
and then add a USER <user> instruction so the container runs unprivileged;
ensure files needed at runtime are chown'ed to that user and that any
environment or entrypoint logic (ENTRYPOINT [] / CMD ["bash"]) works for the
non-root account.
- Line 58: The ENV line sets PATH to a file path (/usr/local/bin/etcd) instead
of a directory which breaks command lookup; update the ENV PATH assignment (the
ENV PATH=... line in the Dockerfile.dsv4-sglang) to reference the directory
/usr/local/bin so PATH entries are directories (e.g., use /usr/local/bin:${PATH}
instead of /usr/local/bin/etcd:${PATH}).
In `@recipes/deepseek-v4-flash/sglang/sglang-dgd.yaml`:
- Around line 1-12: Add the repository-standard SPDX header to the top of this
YAML manifest so the copyright-checkers pass CI: insert the required
SPDX/license comment line(s) (e.g., the SPDX-License-Identifier and copyright
owner line used across the repo) at the very beginning of sglang-dgd.yaml before
any other content, preserving existing comments and formatting.
- Around line 46-48: The decode worker requests 4 GPUs (resources.limits.gpu:
"4") but CUDA_VISIBLE_DEVICES exposes 8 device IDs, causing wrong device mapping
for TP4 tensor parallelism; update the decode worker's environment variable
CUDA_VISIBLE_DEVICES so it lists exactly 4 device IDs (matching
resources.limits.gpu) or generate the list dynamically from that value to ensure
TP4 tensor-parallel ranks map to the correct GPUs.
In `@recipes/deepseek-v4-pro/sglang/sglang-dgd.yaml`:
- Around line 1-3: This manifest is missing the repository-standard SPDX
copyright header; add the required SPDX header text (the same header used across
other recipe manifests) at the very top of sglang-dgd.yaml so the
copyright-checkers CI passes — insert the exact repo-standard SPDX header string
as the first lines of the file before any YAML content.
---
Duplicate comments:
In `@recipes/deepseek-v4-flash/sglang/Dockerfile.dsv4-sglang`:
- Around line 30-36: The Dockerfile currently clones the mutable branch
"codex/deepseek-v4-parsers", fetches a branch "ishan/investigate-tmrw" and
cherry-picks commit 44fb33a5ad53caead03780b5a2a4593b5388c88f which makes the
build non-reproducible; instead pin the source to immutable refs by replacing
that sequence with a single clone/checkout of an exact commit or tag that
already contains both changes (or introduce build ARGs for exact SHAs and
checkout those SHAs), e.g., stop using the branch name in git clone and use the
specific commit SHA for the repository checkout (reference: git clone -b
codex/deepseek-v4-parsers, git fetch origin ishan/investigate-tmrw, git
cherry-pick 44fb33a5ad53cae).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: caa967ed-6484-419d-b095-e15cce6f1e26
📒 Files selected for processing (3)
recipes/deepseek-v4-flash/sglang/Dockerfile.dsv4-sglangrecipes/deepseek-v4-flash/sglang/sglang-dgd.yamlrecipes/deepseek-v4-pro/sglang/sglang-dgd.yaml
Comment on lines
+51
to
+90
| FROM ${DSV4_BASE_IMAGE} | ||
|
|
||
| ENV DEBIAN_FRONTEND=noninteractive | ||
|
|
||
| # Infra from dynamo sglang-runtime (etcd, nats, UCX, NIXL) | ||
| COPY --from=dynamo_src /usr/bin/nats-server /usr/bin/nats-server | ||
| COPY --from=dynamo_src /usr/local/bin/etcd /usr/local/bin/etcd | ||
| ENV PATH=/usr/local/bin/etcd:${PATH} | ||
|
|
||
| # UCX libs | ||
| COPY --from=dynamo_src /usr/lib/x86_64-linux-gnu/ucx /usr/lib/x86_64-linux-gnu/ucx | ||
| COPY --from=dynamo_src /usr/lib/x86_64-linux-gnu/libuc*.so* /usr/lib/x86_64-linux-gnu/ | ||
|
|
||
| # NIXL + deps (pip packages with native libs) | ||
| COPY --from=dynamo_src /usr/local/lib/python3.12/dist-packages/nixl* /usr/local/lib/python3.12/dist-packages/ | ||
| COPY --from=dynamo_src /usr/local/lib/python3.12/dist-packages/nixl_cu12* /usr/local/lib/python3.12/dist-packages/ | ||
|
|
||
| # Dynamo wheels with V4 parsers (overrides the base dynamo from donor) | ||
| COPY --from=wheel_builder /opt/dynamo/wheelhouse/ai_dynamo_runtime*.whl /opt/dynamo/wheelhouse/ | ||
| COPY --from=wheel_builder /opt/dynamo/wheelhouse/ai_dynamo-*.whl /opt/dynamo/wheelhouse/ | ||
|
|
||
| RUN pip install --no-cache-dir --force-reinstall --no-deps \ | ||
| /opt/dynamo/wheelhouse/ai_dynamo_runtime*.whl \ | ||
| /opt/dynamo/wheelhouse/ai_dynamo-*.whl && \ | ||
| python3 -c "from dynamo._core import get_tool_parser_names; assert 'deepseek_v4' in get_tool_parser_names(), 'V4 parser missing!'; print('V4 parser verified')" | ||
|
|
||
| # Dynamo Python components from V4 branch | ||
| COPY --from=wheel_builder /workspace/components/src/dynamo /workspace/components/src/dynamo | ||
|
|
||
| # Fix: sglang repo dir at /workspace/sglang shadows the Python package | ||
| ENV PYTHONPATH=/workspace/sglang/python:/workspace/components/src:${PYTHONPATH} | ||
|
|
||
| # DeepGEMM JIT env vars | ||
| ENV SGLANG_JIT_DEEPGEMM_PRECOMPILE=0 \ | ||
| SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 | ||
|
|
||
| WORKDIR /workspace | ||
|
|
||
| ENTRYPOINT [] | ||
| CMD ["bash"] |
Contributor
There was a problem hiding this comment.
Run the final image as a non-root user (DS-0002).
The final stage does not set USER, so the container runs as root. This is a security hardening gap and matches the Trivy misconfig finding.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@recipes/deepseek-v4-flash/sglang/Dockerfile.dsv4-sglang` around lines 51 -
90, The final image runs as root because the Dockerfile never sets a non-root
USER; update the final stage (after WORKDIR /workspace and before
ENTRYPOINT/CMD) to create or use a non-root user (e.g., add a user/group, set
ownership of runtime directories like /workspace and any copied artifacts) and
then add a USER <user> instruction so the container runs unprivileged; ensure
files needed at runtime are chown'ed to that user and that any environment or
entrypoint logic (ENTRYPOINT [] / CMD ["bash"]) works for the non-root account.
| # Infra from dynamo sglang-runtime (etcd, nats, UCX, NIXL) | ||
| COPY --from=dynamo_src /usr/bin/nats-server /usr/bin/nats-server | ||
| COPY --from=dynamo_src /usr/local/bin/etcd /usr/local/bin/etcd | ||
| ENV PATH=/usr/local/bin/etcd:${PATH} |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Locate and examine the Dockerfile
find . -path "*/recipes/deepseek-v4-flash/sglang/Dockerfile.dsv4-sglang" -type fRepository: ai-dynamo/dynamo
Length of output: 118
🏁 Script executed:
# Read the Dockerfile with line numbers to see the context
cat -n ./recipes/deepseek-v4-flash/sglang/Dockerfile.dsv4-sglangRepository: ai-dynamo/dynamo
Length of output: 4494
Fix PATH: it currently points to the etcd binary file, not a directory.
PATH entries must be directories. Using /usr/local/bin/etcd breaks command resolution; use /usr/local/bin instead.
Suggested patch
-ENV PATH=/usr/local/bin/etcd:${PATH}
+ENV PATH=/usr/local/bin:${PATH}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| ENV PATH=/usr/local/bin/etcd:${PATH} | |
| ENV PATH=/usr/local/bin:${PATH} |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@recipes/deepseek-v4-flash/sglang/Dockerfile.dsv4-sglang` at line 58, The ENV
line sets PATH to a file path (/usr/local/bin/etcd) instead of a directory which
breaks command lookup; update the ENV PATH assignment (the ENV PATH=... line in
the Dockerfile.dsv4-sglang) to reference the directory /usr/local/bin so PATH
entries are directories (e.g., use /usr/local/bin:${PATH} instead of
/usr/local/bin/etcd:${PATH}).
Comment on lines
+1
to
+12
| # DeepSeek-V4-Flash SGLang DynamoGraphDeployment | ||
| # Aggregated mode (no P/D disagg), B200x4 TP4, MXFP4 MoE, EAGLE MTP 3/4 | ||
| # | ||
| # Deploy: | ||
| # kubectl apply -f sglang-dgd.yaml -n <namespace> | ||
| # | ||
| # Test: | ||
| # kubectl port-forward -n <namespace> svc/sglang-dsv4-flash-frontend 8000:8000 | ||
| # curl http://localhost:8000/v1/models | ||
| # curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \ | ||
| # -d '{"model":"deepseek-ai/DeepSeek-V4-Flash","messages":[{"role":"user","content":"Hello"}]}' | ||
|
|
Contributor
There was a problem hiding this comment.
Add the required SPDX header; CI is currently failing on this file.
copyright-checkers reports a missing/invalid header, so this manifest will not pass pipeline checks until the repo-standard SPDX header is added.
🧰 Tools
🪛 GitHub Actions: Copyright Checks
[error] 1-1: Invalid/Missing Header: copyright-checkers failed to find a required SPDX header in this file.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@recipes/deepseek-v4-flash/sglang/sglang-dgd.yaml` around lines 1 - 12, Add
the repository-standard SPDX header to the top of this YAML manifest so the
copyright-checkers pass CI: insert the required SPDX/license comment line(s)
(e.g., the SPDX-License-Identifier and copyright owner line used across the
repo) at the very beginning of sglang-dgd.yaml before any other content,
preserving existing comments and formatting.
Comment on lines
+46
to
+48
| resources: | ||
| limits: | ||
| gpu: "4" |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python - <<'PY'
import re
from pathlib import Path
p = Path("recipes/deepseek-v4-flash/sglang/sglang-dgd.yaml")
s = p.read_text()
gpu = int(re.search(r'gpu:\s*"(\d+)"', s).group(1))
cvd = re.search(r'name:\s*CUDA_VISIBLE_DEVICES\s*\n\s*value:\s*"([^"]+)"', s).group(1)
count = len([x for x in cvd.split(",") if x.strip()])
print(f"gpu_limit={gpu}, cuda_visible_devices_count={count}, value={cvd}")
if gpu != count:
raise SystemExit("Mismatch detected")
PYRepository: ai-dynamo/dynamo
Length of output: 161
🏁 Script executed:
cat -n recipes/deepseek-v4-flash/sglang/sglang-dgd.yamlRepository: ai-dynamo/dynamo
Length of output: 4384
🏁 Script executed:
grep -n "CUDA_VISIBLE_DEVICES" recipes/deepseek-v4-flash/sglang/sglang-dgd.yamlRepository: ai-dynamo/dynamo
Length of output: 105
🏁 Script executed:
grep -E "(DIS-|DYN-|OPS-|DEP-|[A-Z]+-[0-9]{4})" recipes/deepseek-v4-flash/sglang/sglang-dgd.yamlRepository: ai-dynamo/dynamo
Length of output: 42
Align CUDA_VISIBLE_DEVICES with requested GPU count for TP4 tensor parallelism.
The decode worker requests 4 GPUs but exposes 8 device IDs in the CUDA_VISIBLE_DEVICES environment variable. For TP4, this causes incorrect GPU device mapping and can lead to startup or runtime failures.
Suggested patch
- name: CUDA_VISIBLE_DEVICES
- value: "0,1,2,3,4,5,6,7"
+ value: "0,1,2,3"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@recipes/deepseek-v4-flash/sglang/sglang-dgd.yaml` around lines 46 - 48, The
decode worker requests 4 GPUs (resources.limits.gpu: "4") but
CUDA_VISIBLE_DEVICES exposes 8 device IDs, causing wrong device mapping for TP4
tensor parallelism; update the decode worker's environment variable
CUDA_VISIBLE_DEVICES so it lists exactly 4 device IDs (matching
resources.limits.gpu) or generate the list dynamically from that value to ensure
TP4 tensor-parallel ranks map to the correct GPUs.
Comment on lines
+1
to
+3
| # DeepSeek-V4-Pro SGLang DynamoGraphDeployment | ||
| # Aggregated mode, B200x8 TP8, MXFP4 MoE, EAGLE MTP 3/4 | ||
|
|
Contributor
There was a problem hiding this comment.
Add the required SPDX header; CI is currently failing on this file.
copyright-checkers reports a missing/invalid header, so this manifest will keep failing pipeline checks until the repo-standard SPDX header is added at the top.
🧰 Tools
🪛 GitHub Actions: Copyright Checks
[error] 1-1: Invalid/Missing Header: copyright-checkers failed to find a required SPDX header in this file.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@recipes/deepseek-v4-pro/sglang/sglang-dgd.yaml` around lines 1 - 3, This
manifest is missing the repository-standard SPDX copyright header; add the
required SPDX header text (the same header used across other recipe manifests)
at the very top of sglang-dgd.yaml so the copyright-checkers CI passes — insert
the exact repo-standard SPDX header string as the first lines of the file before
any YAML content.
The release/deepseekv4 branch includes V4 parsers (PR #8665) plus proper runtime kwarg compat fixes (filter_supported_async_generate_kwargs, _callable_accepts_kwarg). No manual cherry-pick needed.
1 task
nv-tusharma
added a commit
that referenced
this pull request
Apr 25, 2026
VincyZhang
pushed a commit
to VincyZhang/dynamo
that referenced
this pull request
Apr 27, 2026
Signed-off-by: VincyZhang <wenxin.zhang@intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a SGLang serving recipe for DeepSeek-V4-Flash on Blackwell (B200) GPUs with Dynamo frontend.
Files
recipes/deepseek-v4-flash/sglang/Dockerfile.dsv4-sglang— Multi-stage Dockerfile that builds dynamo wheels from the V4 parsers branch and layers them onto thelmsysorg/sglang:deepseek-v4-blackwellbase imagerecipes/deepseek-v4-flash/sglang/sglang-dgd.yaml— DynamoGraphDeployment manifest (frontend + decode worker)SGLang config
--dyn-tool-call-parser deepseek_v4 --dyn-reasoning-parser deepseek_v4Dependencies
codex/deepseek-v4-parsers(PR chore(frontend): Add DeepSeek V4 parser support + Test Cases #8665) for V4 formatter + parsers44fb33afromishan/investigate-tmrw(PR [INVESTIGATE] sglang + DSv4: kwarg workaround breaks live — needs diagnosis #8671) for sglang kwarg compatSummary by CodeRabbit