Skip to content

[Handoff to @Oseltamivir Claude /loop] [Klaud Cold] Test sgl-deep-gemm==0.0.1 pin for sgl#25551 (glm5-fp8-b300 DeepGemm regression) #1512

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Oseltamivir merged 2 commits into from
May 20, 2026
+29 −14
Merged

[Handoff to @Oseltamivir Claude /loop] [Klaud Cold] Test sgl-deep-gemm==0.0.1 pin for sgl#25551 (glm5-fp8-b300 DeepGemm regression) #1512

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b3dc78f
test(glm5-fp8-b300): pin sgl-deep-gemm==0.0.1 inside v0.5.12 container
functionstackx May 18, 2026
f24746e
fix(glm5_fp8_b300): add --break-system-packages to pip install for sg…
functionstackx May 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions benchmarks/single_node/glm5_fp8_b300.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,17 @@ nvidia-smi

if [[ "$MODEL" != /* ]]; then hf download "$MODEL"; fi

pip install --no-deps "transformers==5.2.0" "huggingface-hub==1.4.1"

# Workaround for sgl-project/sglang#25551: v0.5.12 DeepGemm TMA-descriptor
# regression on B300 (sm_120) crashes CUDA graph capture with
# CUDA_ERROR_ILLEGAL_ADDRESS. Disabling JIT DeepGemm bypasses the affected
# kernel path. Restore to =1 once the upstream regression is fixed.
export SGL_ENABLE_JIT_DEEPGEMM=0
pip install --break-system-packages --no-deps "transformers==5.2.0" "huggingface-hub==1.4.1"

# Testing @trevor-m's suggestion in sgl-project/sglang#25551 (comment 4481466979):
# downgrade sgl-deep-gemm 0.1.0 → 0.0.1 inside the v0.5.12 container to check
# whether the deepgemm version jump is what causes the B300 TMA-descriptor
# CUDA_ERROR_ILLEGAL_ADDRESS regression. Re-enabling JIT DeepGemm so the
# downgraded version actually runs.
# --break-system-packages required: the container's Python is PEP-668 externally-managed,
# so the previous attempt silently failed and left the bundled 0.1.0 in place.
pip install --break-system-packages --no-deps "sgl-deep-gemm==0.0.1"
export SGL_ENABLE_JIT_DEEPGEMM=1

SERVER_LOG=/workspace/server.log
PORT=${PORT:-8888}
Expand Down
18 changes: 11 additions & 7 deletions benchmarks/single_node/glm5_fp8_b300_mtp.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,17 @@ nvidia-smi

if [[ "$MODEL" != /* ]]; then hf download "$MODEL"; fi

pip install --no-deps "transformers==5.2.0" "huggingface-hub==1.4.1"

# Workaround for sgl-project/sglang#25551: v0.5.12 DeepGemm TMA-descriptor
# regression on B300 (sm_120) crashes CUDA graph capture with
# CUDA_ERROR_ILLEGAL_ADDRESS. Disabling JIT DeepGemm bypasses the affected
# kernel path. Restore to =1 once the upstream regression is fixed.
export SGL_ENABLE_JIT_DEEPGEMM=0
pip install --break-system-packages --no-deps "transformers==5.2.0" "huggingface-hub==1.4.1"

# Testing @trevor-m's suggestion in sgl-project/sglang#25551 (comment 4481466979):
# downgrade sgl-deep-gemm 0.1.0 → 0.0.1 inside the v0.5.12 container to check
# whether the deepgemm version jump is what causes the B300 TMA-descriptor
# CUDA_ERROR_ILLEGAL_ADDRESS regression. Re-enabling JIT DeepGemm so the
# downgraded version actually runs.
# --break-system-packages required: the container's Python is PEP-668 externally-managed,
# so the previous attempt silently failed and left the bundled 0.1.0 in place.
pip install --break-system-packages --no-deps "sgl-deep-gemm==0.0.1"
export SGL_ENABLE_JIT_DEEPGEMM=1
export SGLANG_ENABLE_SPEC_V2=1

SERVER_LOG=/workspace/server.log
Expand Down
7 changes: 7 additions & 0 deletions perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3043,3 +3043,10 @@
description:
- "Update SGLang image from v0.5.11-cu130 (5d old) to v0.5.12-cu130"
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1475

- config-keys:
- glm5-fp8-b300-sglang
- glm5-fp8-b300-sglang-mtp
description:
- "Test @trevor-m's suggestion in sgl-project/sglang#25551: pin sgl-deep-gemm==0.0.1 inside v0.5.12 container to isolate whether the deep-gemm 0.0.1→0.1.0 upgrade is the source of the B300 CUDA_ERROR_ILLEGAL_ADDRESS regression."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1512

@claude claude Bot May 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new changelog entry's pr-link is the literal placeholder PR_PLACEHOLDER instead of 1512 (perf-changelog.yaml:2843). If merged as-is the link 404s, and utils/merge_with_reuse.sh will refuse the entry because it recognizes only XXX (or a fully-formed /pull/<pr> URL) as the placeholder. Replace with https://github.com/SemiAnalysisAI/InferenceX/pull/1512 (or the canonical XXX).

Extended reasoning...

What's wrong

perf-changelog.yaml line 2843 was added by this PR as:

  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PR_PLACEHOLDER

Every other entry in the file uses a real numeric ID (e.g. /pull/1422, /pull/1490, /pull/1491). The token PR_PLACEHOLDER is not a string this repo's tooling knows how to substitute — searching the codebase, PR_PLACEHOLDER appears in exactly one place (this newly-added line) and no automation rewrites it.

Why existing tooling doesn't save it

The merge helper at utils/merge_with_reuse.sh:136 is the only mechanism that rewrites pr-link lines, and it recognizes only the canonical XXX token (or a link that already ends with /pull/{pr}):

link = str(entry.get("pr-link") or "")
if "XXX" not in link and not link.endswith(f"/pull/{pr}"):
    continue

/pull/PR_PLACEHOLDER contains neither XXX nor /pull/1512, so this entry is skipped. If this is the only PR contribution in the file, the script exits at line 153–157 with No PR contributions found in perf-changelog.yaml.

Impact

Two failure modes, depending on how the PR is merged:

  1. Plain git merge — the broken /pull/PR_PLACEHOLDER URL lands on main. Anyone clicking through from the changelog hits a 404, and downstream consumers of perf-changelog.yaml get a permanently dangling reference.
  2. utils/merge_with_reuse.sh — the merge tool refuses the entry and aborts, blocking the merge entirely.

Step-by-step proof

  1. Read perf-changelog.yaml line 2843 → pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PR_PLACEHOLDER (confirmed verbatim).
  2. git show 2bc7402 -- perf-changelog.yaml shows the line was added literally as PR_PLACEHOLDER. (The PR-diff display in some review surfaces renders 1512, but the committed blob has the unsubstituted token.)
  3. Grep for PR_PLACEHOLDER across the repo → exactly one match, this added line. No script substitutes it.
  4. Trace the merge path through utils/merge_with_reuse.sh: link = "https://github.com/.../pull/PR_PLACEHOLDER"; "XXX" not in link is True and link.endswith("/pull/1512") is False, so continue skips the entry; contribs ends up empty; line 153 if not contribs: sys.exit(...) aborts.

Fix

Either:

-  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/PR_PLACEHOLDER
+  pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/1512

…or use the conventional XXX placeholder that merge_with_reuse.sh knows how to rewrite. Note: the PR description says "do not merge as-is," but the PR is labeled full-sweep-enabled and can still be merged; the changelog link needs to be correct regardless of when this PR converts into a real fix.

Loading