Skip to content

Update glm-5 b200 sglang image to nightly-dev-cu13-20260523-c112f762 #1567

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hshrivastava-droid merged 13 commits into from
Jun 12, 2026
Merged

Update glm-5 b200 sglang image to nightly-dev-cu13-20260523-c112f762 #1567

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
d06b958
Update glm-5 b200 sglang container image to nightly-dev-cu13-20260523…
Ankur-singh May 26, 2026
12cae11
remove cutlass gemm backend
Ankur-singh May 28, 2026
d2263d5
Merge branch 'main' into glm-update
Ankur-singh May 28, 2026
e80ab6e
glm5-b200-sglang: bump SGLang image to nightly-dev-cu13-20260605-7dc7…
Ankur-singh Jun 5, 2026
c23bea7
Merge remote-tracking branch 'origin/main' into glm-update
Ankur-singh Jun 5, 2026
bf52ead
remove transformers version ping
xinli-sw Jun 8, 2026
1469214
Merge branch 'main' into glm-update
xinli-sw Jun 8, 2026
bd62328
remove transformers version ping and remove deepgemm war
xinli-sw Jun 8, 2026
b14879a
glm5-fp4-b200-sglang-mtp: drop --mem-fraction-static to 0.8 for 8k1k …
Ankur-singh Jun 8, 2026
49cfc1f
Merge remote-tracking branch 'origin/main' into glm-update
Ankur-singh Jun 8, 2026
5b06baf
Lower mem-fraction-static at high concurrency for glm5 fp4 b200 mtp
hshrivastava-droid Jun 10, 2026
07220de
Merge branch 'main' into glm-update
hshrivastava-droid Jun 12, 2026
7f01c81
Update SGLang image and memory fraction settings
hshrivastava-droid Jun 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions .github/configs/nvidia-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2205,7 +2205,7 @@ qwen3.5-fp4-b200-sglang-mtp:
- { tp: 2, ep: 1, conc-start: 4, conc-end: 64, spec-decoding: mtp }

glm5-fp8-b200-sglang:
image: lmsysorg/sglang:v0.5.12-cu130
image: lmsysorg/sglang:nightly-dev-cu13-20260605-7dc73766
model: zai-org/GLM-5-FP8
model-prefix: glm5
runner: b200
Expand All @@ -2224,7 +2224,7 @@ glm5-fp8-b200-sglang:
- { tp: 8, ep: 1, conc-start: 4, conc-end: 256 }

glm5-fp8-b200-sglang-mtp:
image: lmsysorg/sglang:v0.5.12-cu130
image: lmsysorg/sglang:nightly-dev-cu13-20260605-7dc73766
model: zai-org/GLM-5-FP8
model-prefix: glm5
runner: b200
Expand Down Expand Up @@ -2304,7 +2304,7 @@ glm5-fp8-b300-sglang-mtp:
- { tp: 8, ep: 1, conc-start: 4, conc-end: 256, spec-decoding: mtp }

glm5-fp4-b200-sglang:
image: lmsysorg/sglang:v0.5.12-cu130
image: lmsysorg/sglang:nightly-dev-cu13-20260605-7dc73766
model: nvidia/GLM-5-NVFP4
model-prefix: glm5
runner: b200
Expand All @@ -2325,7 +2325,7 @@ glm5-fp4-b200-sglang:
- { tp: 4, ep: 1, conc-start: 4, conc-end: 256 }

glm5-fp4-b200-sglang-mtp:
image: lmsysorg/sglang:v0.5.12-cu130
image: lmsysorg/sglang:nightly-dev-cu13-20260605-7dc73766
model: nvidia/GLM-5-NVFP4
model-prefix: glm5
runner: b200
Expand Down
11 changes: 8 additions & 3 deletions benchmarks/single_node/fixed_seq_len/glm5_fp4_b200_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,16 +19,21 @@ nvidia-smi

if [[ "$MODEL" != /* ]]; then hf download "$MODEL"; fi

pip install --no-deps "transformers==5.2.0" "huggingface-hub==1.4.1"

export SGL_ENABLE_JIT_DEEPGEMM=1
export SGLANG_ENABLE_JIT_DEEPGEMM=1
export SGLANG_ENABLE_SPEC_V2=1

SERVER_LOG=/workspace/server.log


echo "CONC: $CONC, ISL: $ISL, OSL: $OSL"

MEM_FRACTION_STATIC=0.85
if [[ "$CONC" -gt 128 ]]; then
MEM_FRACTION_STATIC=0.8
fi

@cursor cursor Bot Jun 10, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mem fraction ignores ISL OSL

Medium Severity

glm5_fp4_b200_mtp.sh lowers --mem-fraction-static to 0.8 whenever CONC is above 128, but the perf changelog states that override applies only for ISL=8192, OSL=1024, and CONC>128. High-concurrency 1k/1k points still get 0.8 instead of the documented default 0.85.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 5b06baf. Configure here.

echo "MEM_FRACTION_STATIC: $MEM_FRACTION_STATIC"

EVAL_CONTEXT_ARGS=""
if [ "${EVAL_ONLY}" = "true" ]; then
setup_eval_context
Expand All @@ -49,7 +54,7 @@ PYTHONNOUSERSITE=1 python3 -m sglang.launch_server --model-path=$MODEL --host=0.
--nsa-decode-backend trtllm --nsa-prefill-backend trtllm \
--moe-runner-backend flashinfer_trtllm \
--cuda-graph-max-bs $CONC --max-running-requests $CONC \
--mem-fraction-static 0.85 \
--mem-fraction-static $MEM_FRACTION_STATIC \
--chunked-prefill-size 32768 --max-prefill-tokens 32768 \
--enable-flashinfer-allreduce-fusion --disable-radix-cache \
--stream-interval 30 \
Expand Down
3 changes: 1 addition & 2 deletions benchmarks/single_node/fixed_seq_len/glm5_fp4_b300_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,8 @@ fi
nvidia-smi


pip install --no-deps "transformers==5.2.0" "huggingface-hub==1.4.1"

export SGL_ENABLE_JIT_DEEPGEMM=1
export SGLANG_ENABLE_JIT_DEEPGEMM=1
export SGLANG_ENABLE_SPEC_V2=1

SERVER_LOG=/workspace/server.log
Expand Down
5 changes: 1 addition & 4 deletions benchmarks/single_node/fixed_seq_len/glm5_fp8_b200.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,7 @@ nvidia-smi

if [[ "$MODEL" != /* ]]; then hf download "$MODEL"; fi

pip install --no-deps "transformers==5.2.0" "huggingface-hub==1.4.1"

export SGL_ENABLE_JIT_DEEPGEMM=1
export SGLANG_ENABLE_JIT_DEEPGEMM=1

SERVER_LOG=/workspace/server.log

Expand All @@ -44,7 +42,6 @@ PYTHONNOUSERSITE=1 python3 -m sglang.launch_server --model-path=$MODEL --host=0.
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--kv-cache-dtype fp8_e4m3 --quantization fp8 \
--fp8-gemm-backend cutlass \

@cursor cursor Bot Jun 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

B200 DeepGemm workaround removed

Medium Severity

This change drops --fp8-gemm-backend cutlass from the GLM-5 FP8 B200 launch scripts while SGL_ENABLE_JIT_DEEPGEMM stays 1. The repo previously added that flag to avoid DeepGemm CUDA graph capture failures on B200; without an equivalent mitigation or a confirmed fix in the new nightly image, the server may fail during startup or warmup.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit c23bea7. Configure here.

--attention-backend nsa \
--nsa-decode-backend trtllm --nsa-prefill-backend trtllm \
--moe-runner-backend flashinfer_trtllm \
Expand Down
5 changes: 1 addition & 4 deletions benchmarks/single_node/fixed_seq_len/glm5_fp8_b200_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,7 @@ nvidia-smi

if [[ "$MODEL" != /* ]]; then hf download "$MODEL"; fi

pip install --no-deps "transformers==5.2.0" "huggingface-hub==1.4.1"

export SGL_ENABLE_JIT_DEEPGEMM=1
export SGLANG_ENABLE_JIT_DEEPGEMM=1
Comment thread
functionstackx marked this conversation as resolved.
export SGLANG_ENABLE_SPEC_V2=1

SERVER_LOG=/workspace/server.log
Expand All @@ -45,7 +43,6 @@ PYTHONNOUSERSITE=1 python3 -m sglang.launch_server --model-path=$MODEL --host=0.
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--kv-cache-dtype fp8_e4m3 --quantization fp8 \
--fp8-gemm-backend cutlass \
--attention-backend nsa \
--nsa-decode-backend trtllm --nsa-prefill-backend trtllm \
--moe-runner-backend flashinfer_trtllm \
Expand Down
9 changes: 1 addition & 8 deletions benchmarks/single_node/fixed_seq_len/glm5_fp8_b300.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,7 @@ fi

nvidia-smi


pip install --no-deps "transformers==5.2.0" "huggingface-hub==1.4.1"

# Workaround for sgl-project/sglang#25551: v0.5.12 DeepGemm TMA-descriptor
# regression on B300 (sm_120) crashes CUDA graph capture with
# CUDA_ERROR_ILLEGAL_ADDRESS. Disabling JIT DeepGemm bypasses the affected
# kernel path. Restore to =1 once the upstream regression is fixed.
export SGL_ENABLE_JIT_DEEPGEMM=0
export SGLANG_ENABLE_JIT_DEEPGEMM=1

@cursor cursor Bot Jun 8, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

B300 script drops DeepGemm workaround

High Severity

The B300 FP8 launch scripts now set SGLANG_ENABLE_JIT_DEEPGEMM=1 and drop the prior SGL_ENABLE_JIT_DEEPGEMM=0 guard for sglang#25551, while glm5-fp8-b300-sglang and MTP configs in nvidia-master.yaml still use lmsysorg/sglang:v0.5.12-cu130. That pairing can bring back CUDA graph capture failures on B300 benchmark runs.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit bd62328. Configure here.


SERVER_LOG=/workspace/server.log

Expand Down
7 changes: 1 addition & 6 deletions benchmarks/single_node/fixed_seq_len/glm5_fp8_b300_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,8 @@ fi
nvidia-smi


pip install --no-deps "transformers==5.2.0" "huggingface-hub==1.4.1"

# Workaround for sgl-project/sglang#25551: v0.5.12 DeepGemm TMA-descriptor
# regression on B300 (sm_120) crashes CUDA graph capture with
# CUDA_ERROR_ILLEGAL_ADDRESS. Disabling JIT DeepGemm bypasses the affected
# kernel path. Restore to =1 once the upstream regression is fixed.
export SGL_ENABLE_JIT_DEEPGEMM=0
export SGLANG_ENABLE_JIT_DEEPGEMM=1
export SGLANG_ENABLE_SPEC_V2=1

SERVER_LOG=/workspace/server.log
Expand Down
10 changes: 10 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3609,3 +3609,13 @@
- "8 topologies sweeping low-latency (1p1d-tp8-tp8) through max throughput (6p1d-dep8-dep12)."
- "Updated 1p1d and 2p1d configs to match https://github.com/shyeh25/srt-slurm/commit/ede724d7cc9a780be5b84659f599733bf9fd0097"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1675

- config-keys:
- glm5-fp4-b200-sglang
- glm5-fp4-b200-sglang-mtp
- glm5-fp8-b200-sglang
- glm5-fp8-b200-sglang-mtp
description:
- "Update SGLang image from v0.5.12-cu130 to nightly-dev-cu13-20260605-7dc73766"
- "glm5-fp4-b200-sglang-mtp: override --mem-fraction-static from 0.85 to 0.8 when ISL=8192, OSL=1024, and CONC>128 (default 0.85 elsewhere)"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1567
Loading