[AMD][MI35X] Add qwen3.8-flash-next-fp8-mi355x-sglang-agentic-mtp on v0.5.19-rocm700-mi35x-20260909

[ChangLiu0709]

Summary

• Add day-zero Qwen3.8-Flash-Next FP8 agentic benchmark on MI355X with NEXTN MTP speculative decoding
• Image: lmsysorg/sglang-rocm:v0.5.19-rocm700-mi35x-20260909 (includes SGLang PR #37500, merged 2026-09-08, full Qwen3.8-Flash-Next model support)
• Model: Qwen/Qwen3.8-Flash-Next-FP8 (176B total, 6B active, hybrid GDN+QSA)
• TP=4, EP=1, NEXTN MTP (3 steps, eagle-topk 1, 4 draft tokens)

Changes

File	Change
configs/amd-master.yaml	New config block with v0.5.19-rocm700-mi35x-20260909 image, TP=4
benchmarks/single_node/agentic/qwen3.8next_fp8_mi355x_sglang_mtp.sh	Benchmark script; no runtime patches needed
perf-changelog.yaml	Changelog entry documenting the new config

Notes

• Decode CUDA graphs disabled (--cuda-graph-backend-decode disabled) due to a tilelang MFMA warp partition bug (N must be divisible by 16, but got 8) in the v0.5.19 image. Will re-enable once upstream fix lands.
• Previous runtime patches (backport of SGLang PR #36497) for FP8 PLE weight_scale are no longer needed — the new image includes PR #37500 which supersedes them.

Verification

Check	Status
FP8 model loads without weight_scale assertion	✅
Server starts with --cuda-graph-backend-decode disabled	✅
NEXTN MTP speculative decoding configured	✅

Test Plan

Run on MI355X node:

MODEL=Qwen/Qwen3.8-Flash-Next-FP8 TP=4 CONC=4 EP_SIZE=1 \
KV_OFFLOADING=none TOTAL_CPU_DRAM_GB=512 RESULT_DIR=/tmp/qwen38-test \
DURATION=60 PORT=30010 \
bash benchmarks/single_node/agentic/qwen3.8next_fp8_mi355x_sglang_mtp.sh

[github-actions]

Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase As a PR reviewer and CODEOWNER, I have reviewed this and have.

For PR verification, add the full-sweep-fail-fast label (strongly recommended) to this PR — the benchmark sweep only runs on labeled PRs. Use full-sweep-enabled only if you need matrix jobs to keep running past a failure.

PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs

---

感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 As a PR reviewer and CODEOWNER, I have reviewed this and have。

如需进行 PR 验证，请为此 PR 添加 full-sweep-fail-fast 标签（强烈推荐）— 基准测试 sweep 仅在带有标签的 PR 上运行。仅当需要矩阵任务在失败后继续运行时才使用 full-sweep-enabled。

PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 4dc6c45. Configure here.}

[cursor]

Wrong model-prefix breaks script lookup

High Severity

model-prefix is qwen3.8-flash-next, but the recipe is named qwen3.8next_fp8_mi355x_sglang_mtp.sh. The MI355X launcher resolves the script from ${EXP_NAME%%_*}, which comes from model-prefix, so CI will look for qwen3.8-flash-next_fp8_mi355x_sglang_mtp.sh and fail to start. Sibling Qwen3.8-Flash-Next configs and MODELS.md use qwen3.8next.

Additional Locations (1)

• benchmarks/single_node/agentic/qwen3.8next_fp8_mi355x_sglang_mtp.sh#L1

 

^{Reviewed by Cursor Bugbot for commit 4dc6c45. Configure here.}

[claude]

Findings marked 🟡 are optional suggestions and need no follow-up push.

[claude]

🔴 This script rewrites the pinned image's installed sglang source (5 python3 heredocs patching qwen4_exp.py) before launch, which docs/PR_REVIEW_CHECKLIST.md explicitly bars except with a filled-out docs/waiver/<PR_NUMBER>.md; no docs/waiver/2924.md is included in this PR (only pre-existing docs/waiver/2508.md exists). Merging as-is means the pinned lmsysorg/sglang-rocm:qwen38flashnext image never actually runs as shipped, silently defeating the checklist's engine-integrity guarantee for anyone reviewing/trusting this recipe. Fix: add docs/waiver/2924.md stating what is patched, why the unmodified image can't run this benchmark, the upstream PR link, and a removal plan, or drop the inline patches and wait for a fixed image.

Extended reasoning...

docs/PR_REVIEW_CHECKLIST.md line 30 (checked against docs/PR_REVIEW_CHECKLIST.md in this repo) requires: 'this PR does not patch the inference engine or serving stack ... The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md'. This PR's script applies 5 python3-heredoc patches plus a sed patch to /workspace/sglang-qwen-next/.../qwen4_exp.py and /sgl-workspace/sglang/.../multi_tokenizer_mixin.py before sglang.launch_server runs, i.e. exactly the patched-inference-engine pattern the checklist calls out by name. The diff includes no docs/waiver/2924.md (verified via ls docs/waiver, which only contains 2508.md unrelated to this PR/model). The PR's own changelog text acknowledges these are stopgap patches rather than pointing to a waiver, so the required sign-off artifact is missing and the resulting benchmark numbers are produced by a modified engine, not the pinned image as shipped.

Verification: normal (conflicts with stated purpose: the patching is the PR's mechanism, but the checklist requires it be gated on a waiver, which fixing does not undo). The script rewrites the pinned image's installed engine sources before launch — five python3 heredocs (lines 137-256) that read/replace/write ${_SGLANG_ROOT}/srt/models/qwen4_exp.py (default… | normal. The script rewrites installed SGLang…

[claude]

🟡 (optional) Patch 4's "already applied" pre-check uses grep -q 'ngram_embedding.reduce(embeddings)\n.*\* ...weight_scale' with a literal \n inside a single-line BRE pattern; grep matches per-line and never sees an embedded newline, so this grep always fails to match and the else "Already applied" branch (line 233) is permanently dead code. Fix: detect already-applied state with a pattern that fits on one line (e.g. grep for the inserted embeddings = embeddings * self.ple_embedding.ngram_embedding.weight_scale line alone), or drop the outer grep gate and rely solely on the inner Python old not in content check.

Extended reasoning...

Every invocation of the script takes the if branch and re-runs the Patch 4 python3 heredoc, even on a container where the patch was already applied by a prior run (this benchmark's own comment says the container/source persists across the 9-point conc-list sweep, and patches are meant to be idempotent no-ops on re-run). The python3 heredoc's own old not in content guard happens to prevent a second textual replacement, so no double-patch occurs, but the operator-facing log always prints "PATCH 4: Adding weight_scale multiply..." followed by "Target pattern not found, skipping" instead of the intended "Already applied" — misleading anyone debugging a later failed run (e.g. a real image update that legitimately makes the patch a no-op looks identical to a normal re-run).

Verification: nit. Line 214: if ! grep -q 'ngram_embedding.reduce(embeddings)\n.*\* self.ple_embedding.ngram_embedding.weight_scale' "$_QWEN4_EXP" embeds a literal \n in a single-line grep pattern. grep matches per line (newlines stripped), and the two anchor substrings live on separate lines in the file, so this grep can never match. ! grep is therefore always true, the if branch (line 215)…

[functionstackx]

Is there no mxfp4 AMD wuant yet?

[ChangLiu0709]

Is there no mxfp4 AMD wuant yet?

That will be released soon.