Skip to content

support qwen 3.8 #34585

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Qiaolin-Yu wants to merge 2 commits into from
Closed

support qwen 3.8 #34585

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d0551b6
introduce qwen 3.8
Qiaolin-Yu Aug 12, 2026
ff6b4b5
Pass uniform_q_len to the FlashInfer prefill plan (#34670)
b8zhong Aug 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .dockerignore
View file
Open in desktop

This file was deleted.

53 changes: 53 additions & 0 deletions .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Build-context allowlist.
#
# This file used to be a symlink to .gitignore. That let the entire repository
# into the build context, and `COPY . /sgl-workspace/sglang` then shipped it all
# inside a distributable image: the test suite, internal benchmarks, the docs
# site, CI workflows, the model gateway, and the agent skill directories under
# .claude -- including python/sglang/multimodal_gen/.claude, which sits inside
# the runtime tree and so survives any top-level filtering.
#
# None of that has a runtime role, and all of it is proprietary: anyone able to
# pull the image could read it. The context is therefore an explicit allowlist of
# what the image actually needs, rather than a denylist inherited from git.
#
# .gitignore and .dockerignore answer different questions -- "what should not be
# committed" versus "what should not leave the building" -- and sharing one file
# between them is what let this happen.

# Exclude everything, then re-admit only what is needed.
*

# The runtime package: what `pip install -e python` installs.
!python/
# The Dockerfiles run cuda_pins.sh and apply_deepep_v2_patch.sh out of the copied
# tree, not only from /opt/qwen38, so the build needs this directory.
!docker/
# Required at BUILD time, not run time: python/setup.py points setuptools-rust at
# ../rust (lines 3 and 178), so `pip install -e python` fails without it --
# "no cargo workspace at rust/Cargo.toml". Excluding it would force
# SGLANG_BUILD_RUST_EXTS=none, i.e. an image with no Rust extensions at all,
# which is a capability change disguised as a size reduction. The Dockerfile
# still deletes rust/*/target afterwards, so no build output ships.
!rust/
# Both are tiny. Dropping them on a guess that nothing reads them would risk
# breaking gRPC codegen or a vendored build for no meaningful reduction.
!proto/
!3rdparty/
# Apache-2.0 requires the licence to accompany distributions of a derived work,
# and this image is one. It is not runtime code, but it does have to ship.
!LICENSE

# Re-exclude what must not travel even inside the allowlisted trees. These come
# after the allowlist entries on purpose: the last matching pattern wins.
**/.claude/

# Build artifacts. Previously covered by .gitignore through the symlink; restated
# so a `docker build` from a dirty checkout cannot smuggle them in either.
**/__pycache__/
**/*.py[cod]
**/*.so
**/build/
**/dist/
**/*.egg-info/
.git/
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
default_stages: [pre-commit, pre-push, manual]
exclude: ^(python/sglang/multimodal_gen/csrc|python/sglang/kernels/ops/diffusion/render|python/sglang/kernels/ops/attention/flash_attn/cute)
exclude: ^(python/sglang/multimodal_gen/csrc|python/sglang/kernels/ops/diffusion/render|python/sglang/kernels/ops/attention/flash_attn/cute|python/sglang/srt/layers/flashinfer_fallback/comm/(mnnvl_cutedsl/|mnnvl_cutedsl_ar\.py))

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
Expand Down
120 changes: 120 additions & 0 deletions docker/qwen38/apply_deepep_v2_patch.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
#!/bin/bash
# Qwen38 DeepEP v2 patch + build -- replaces the base image's stock DeepEP.
#
# The base image ships DeepEP at the legacy pin, which does not serve GB300 on
# CUDA 13. This installs the v2 tree instead:
# - v2 moved to a CMake/JIT layout, so the cross-node timeout knobs now live
# in csrc/kernels/legacy/compiled.cuh behind LEGACY_-prefixed macros
# - the wheel links the nvidia-nvshmem/nccl wheels rather than a system NCCL
# - CUDA 13 relocated the cccl headers; the legacy setuptools path needs an
# extra include dir, the CMake path resolves them itself
#
# Both timeout bumps are asserted after the fact because `sed -i` exits 0 on no
# match: shipping the stock 100s CPU timeout is invisible at build time and
# surfaces only as GB300 multi-node init aborts in production.
#
# Env knobs (all optional):
# DEEPEP_V2_COMMIT pinned commit to build
# TORCH_CUDA_ARCH_LIST arches to compile cubins for
# MAX_JOBS nvcc build parallelism
# DEEPEP_DIR where the source tree lands in the image
set -euo pipefail

: "${DEEPEP_V2_COMMIT:=01dc3aaac82068020353dce2c302e38153c0bfaa}"
: "${TORCH_CUDA_ARCH_LIST:=9.0;10.0;10.3}"
: "${MAX_JOBS:=8}"
: "${DEEPEP_DIR:=/sgl-workspace/DeepEP}"
: "${CUDA_HOME:=/usr/local/cuda}"

BUILD_DIR=/build/DeepEP
TIMEOUT_HEADER=csrc/kernels/legacy/compiled.cuh

rm -rf "${BUILD_DIR}"
git clone https://github.com/deepseek-ai/DeepEP.git "${BUILD_DIR}"
cd "${BUILD_DIR}"
git checkout "${DEEPEP_V2_COMMIT}"

# --- Cross-node timeout headroom ---
sed -i \
's/#define LEGACY_NUM_CPU_TIMEOUT_SECS 100/#define LEGACY_NUM_CPU_TIMEOUT_SECS 1000/' \
"${TIMEOUT_HEADER}"
sed -i \
's/#define LEGACY_NUM_TIMEOUT_CYCLES 200000000000ull/#define LEGACY_NUM_TIMEOUT_CYCLES 2000000000000ull/' \
"${TIMEOUT_HEADER}"

if ! grep -q '#define LEGACY_NUM_CPU_TIMEOUT_SECS 1000' "${TIMEOUT_HEADER}"; then
echo "ERROR: CPU timeout bump did not apply; the macro moved or was reformatted" >&2
exit 1
fi
if ! grep -q '#define LEGACY_NUM_TIMEOUT_CYCLES 2000000000000ull' "${TIMEOUT_HEADER}"; then
echo "ERROR: cycle timeout bump did not apply; the macro moved or was reformatted" >&2
exit 1
fi

# --- CUDA 13 cccl include dir (legacy setuptools path only) ---
# A missing anchor is the expected outcome on a CMake tree, so report which
# path was taken rather than letting sed no-op silently.
if grep -q "^ include_dirs = \['csrc/'\]" setup.py; then
sed -i \
"/^ include_dirs = \['csrc\/'\]/a\\ include_dirs.append('${CUDA_HOME}/include/cccl')" \
setup.py
echo "Applied the CUDA 13 cccl include fix to setup.py"
else
echo "setup.py has no legacy include_dirs anchor; relying on the CMake build to resolve cccl"
fi

# --- Build and swap in ---
TORCH_CUDA_ARCH_LIST="${TORCH_CUDA_ARCH_LIST}" MAX_JOBS="${MAX_JOBS}" \
python3 setup.py bdist_wheel -d /wheels

# Remove whatever currently owns the deep_ep module, by asking the metadata
# rather than by name. The base image's provider has changed name: v0.5.17
# shipped a distribution literally called `deep_ep`, while the dev images ship
# `sgl-deep-ep`, which installs the same deep_ep/ path (plus its own
# deep_ep_cpp*.so). `pip uninstall deep_ep` exits 0 with only a warning when no
# such distribution exists, so hardcoding either name leaves the other in place
# and two distributions then contend for deep_ep/ -- whichever wrote the files
# last wins, which is not a decision this build should leave to chance.
#
# packages_distributions() reads metadata and does not import deep_ep, so the
# packaged provider's driver check cannot fire here.
#
# Run from / rather than the build tree: `setup.py bdist_wheel` above leaves a
# deep_ep.egg-info in ${BUILD_DIR}, and `python3 -` puts the cwd first on
# sys.path, so importlib.metadata would report that build artifact as an
# installed distribution named deep_ep on top of the real one.
mapfile -t DEEP_EP_DISTS < <(cd / && python3 - <<'PY'
from importlib.metadata import packages_distributions

for dist in sorted(set(packages_distributions().get("deep_ep", ()))):
print(dist)
PY
)
if [ "${#DEEP_EP_DISTS[@]}" -gt 0 ]; then
echo "Removing existing deep_ep provider(s): ${DEEP_EP_DISTS[*]}"
python3 -m pip uninstall -y "${DEEP_EP_DISTS[@]}"
else
echo "No installed distribution provides deep_ep; nothing to remove"
fi

python3 -m pip install /wheels/*.whl

# Exactly one provider must remain. Two would mean the removal above missed a
# name and the source build is now sharing deep_ep/ with a packaged copy.
# From / for the same reason as above -- the egg-info left in the build tree is a
# build artifact, not an installed distribution, and counting it here reported
# ['deep_ep', 'deep_ep'] and failed a build that was in fact correct.
( cd / && python3 - <<'PY'
from importlib.metadata import packages_distributions

providers = sorted(set(packages_distributions().get("deep_ep", ())))
if len(providers) != 1:
raise SystemExit(
f"expected exactly one distribution to provide deep_ep, found {providers}"
)
print(f"deep_ep is provided by {providers[0]} alone")
PY
)

rm -rf "${DEEPEP_DIR}" /wheels "${BUILD_DIR}/build" "${BUILD_DIR}/dist" /root/.cache/pip
mv "${BUILD_DIR}" "${DEEPEP_DIR}"
Loading
Loading