Skip to content

[AMD][MI35X] Qwen3.8-Flash-Next FP4 SGLang agentic MTP: 20261031 image, monokernel at c1-c8 / [AMD][MI35X] Qwen3.8-Flash-Next：20261031 镜像，c1-c8 启用 monokernel #4542

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chunfangamd merged 27 commits into from
Nov 3, 2026
+44 −14
Merged

[AMD][MI35X] Qwen3.8-Flash-Next FP4 SGLang agentic MTP: 20261031 image, monokernel at c1-c8 / [AMD][MI35X] Qwen3.8-Flash-Next：20261031 镜像，c1-c8 启用 monokernel #4542

Show file tree
Hide file tree
Changes from 20 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
00b99af
Qwen3.8-Flash-Next MI355X AgentX baseline: merged upstream PRs and th…
ChangLiu0709 Oct 21, 2026
2e018fe
Qwen3.8-Flash-Next MI355X AgentX baseline: mem-fraction-static 0.9 at…
ChangLiu0709 Oct 21, 2026
33c460f
Qwen3.8-Flash-Next MI355X AgentX baseline round 02: hc_mix, GEMM tabl…
ChangLiu0709 Oct 21, 2026
400295d
Qwen3.8-Flash-Next MI355X baseline round 02: aiter #5992, #6670 and #…
ChangLiu0709 Oct 26, 2026
a1dbc93
Qwen3.8-Flash-Next MI355X baseline round 02: pr-link #4542
ChangLiu0709 Oct 26, 2026
6b3ddb6
Qwen3.8-Flash-Next MI355X baseline round 02: lmsysorg/sglang-rocm:v0.…
ChangLiu0709 Oct 26, 2026
69234e2
qwen3.8next: import CapabilityRequirement in monokernel patch for SGL…
ChangLiu0709 Oct 26, 2026
ecb3b0f
qwen3.8next: regenerate the SGLang patches, move the monokernel to sg…
ChangLiu0709 Oct 29, 2026
093c13b
qwen3.8next: move to the 20261030 image, patch only merged PRs
ChangLiu0709 Oct 30, 2026
c4d912a
Merge remote-tracking branch 'origin/main' into chang/qwen3.8-flash-f…
ChangLiu0709 Oct 30, 2026
66c33fd
qwen3.8next: recipe comments match the patch set
ChangLiu0709 Oct 30, 2026
a5c969f
qwen3.8next: drop the AITER patches, run the image's AITER as shipped
ChangLiu0709 Oct 30, 2026
f9a8f9e
qwen3.8next: rename the setup script to qwen3.8next-sgl-amd-mono-29.sh
ChangLiu0709 Oct 30, 2026
964932c
qwen3.8next: run c8, c12 and c16 on the default path, drop the monoke…
ChangLiu0709 Oct 30, 2026
83e5d30
Revert "qwen3.8next: run c8, c12 and c16 on the default path, drop th…
ChangLiu0709 Oct 30, 2026
dc0ba10
qwen3.8next: patch in sglang#47485, enable the AITER lm_head GEMM
ChangLiu0709 Oct 30, 2026
83a477f
qwen3.8next: credit the round 02 co-authors
ChangLiu0709 Oct 30, 2026
297196b
qwen3.8next: drop the sglang#47485 lm_head patch from the round 02 ba…
ChangLiu0709 Oct 31, 2026
7ba9d07
qwen3.8next: disable integer all-reduce on the round 02 baseline
ChangLiu0709 Oct 31, 2026
bb8bd50
qwen3.8next: move round 02 onto the 20261031 image and drop the setup…
ChangLiu0709 Oct 31, 2026
90fc8ff
Merge origin/main into the round 02 baseline
ChangLiu0709 Oct 31, 2026
bdf93a7
Merge branch 'main' into chang/qwen3.8-flash-fp4-mi355x-sglang-agenti…
ChangLiu0709 Oct 31, 2026
54b9aad
Merge origin/main into the round 02 baseline
ChangLiu0709 Nov 2, 2026
006cf66
Merge origin/main into the round 02 baseline
chunfangamd Nov 2, 2026
e7a7306
fix(qwen3.8next): evaluate the MI355X monokernel points and serve thi…
chunfangamd Nov 2, 2026
64527ed
Merge branch 'main' into chang/qwen3.8-flash-fp4-mi355x-sglang-agenti…
ChangLiu0709 Nov 2, 2026
ed02f66
Merge branch 'main' into chang/qwen3.8-flash-fp4-mi355x-sglang-agenti…
chunfangamd Nov 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 26 additions & 12 deletions ...e/benchmarks/single_node/srt-slurm-recipes/qwen3.8next/sglang/mi355x-fp4-mtp/agentic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,16 +67,21 @@ base:
AIPERF_REQUIRED_SERVER_METRIC_PREFIX: 'sglang:'
AIPERF_APPLY_CHAT_TEMPLATE: 'true'

# One variant per point. Every point keeps ReplaySSM, the PLE table offload and the
# default path, admits 2x CONC requests and captures decode graphs up to that
# admission, so bursts past CONC stay in a graph; mem-fraction-static is used as
# given instead of the aiter backend's 0.85 scaling.
# c1, c2, c4 and c8 run the FlyDSL monokernel (--enable-amd-monokernel) for MTP
# target verify and draft. Its verify writes the per-draft intermediate SSM
# states, so ReplaySSM is off, and its prelude reads the PLE table on the GPU,
# so the table is not offloaded. Admission equals the concurrency. c12 and c16
# keep ReplaySSM, the PLE offload and 2x CONC admission. Every point uses
# mem-fraction-static as given instead of the aiter backend's 0.85 scaling.
override_c1:
roles:
agg:
args:
max-running-requests: 2
cuda-graph-max-bs-decode: 2
enable-linear-replayssm-spec: null
ple-offload-embedding: null
enable-amd-monokernel: true
Comment on lines +81 to +83

@claude claude Bot Oct 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 After merge, no default sweep runs an accuracy eval on the new monokernel used for c1-c8. Only c16, on the default path, gets GSM8K. Eval groups are keyed on engine env, not CLI args. The c1-c8 overrides switch kernels only through args (enable-amd-monokernel, nulled ReplaySSM/PLE offload), and all six points share one env. Fix: give the monokernel points (c1-c8) their own engine env, using the image's env switch if it has one, so they get their own GSM8K job, or require an all-evals run before /use. Same pattern at 4 sites (agentic.yaml:82, :98, :114, :130).

Why this was flagged

mark_eval_entries (infx/matrix/generate.py:802-815) groups single-node agentic rows by model, runner, framework, precision, spec-decoding, dp-attn, image and _engine_env(entry). It picks only the highest-conc row in each group for GSM8K. engine_environment (infx/matrix/fingerprint.py:81-102) serializes only the recipe environment and each role's env. EVALS.md:21-26 says a point that switches to a mono or megakernel through an env var gets its own eval, but CLI args do not split groups. override_c1/c2/c4/c8 differ from c12/c16 only in args (lines 80-84, 96-100, 112-116, 128-132). All six points have the same role env, base env plus SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION='1'. So with the default label, the single group runs GSM8K only at c16, which uses ReplaySSM and the default verify path. On the base branch, all six points ran that same path, so the c16 eval covered every point. After this change, the FlyDSL verify/draft kernel at c1-c8 has no accuracy check. The PR description's own validation cites only "the c16 eval" plus throughput runs.

Verification: engine_environment in infx/matrix/fingerprint.py:81-102 serializes only recipe environment and each role's env. In agentic.yaml, c1/c2/c4/c8 (lines 76-138) switch to the monokernel only through args. So they form one group, and only c16 gets GSM8K. On the base commit, c1-c8 ran the same kernel path as c16, so the c16 eval covered them.

max-running-requests: 1
cuda-graph-max-bs-decode: 1
env:
SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION: '1'
benchmark:
Expand All @@ -88,8 +93,11 @@ override_c2:
roles:
agg:
args:
max-running-requests: 4
cuda-graph-max-bs-decode: 4
enable-linear-replayssm-spec: null
ple-offload-embedding: null
enable-amd-monokernel: true
max-running-requests: 2
cuda-graph-max-bs-decode: 2
env:
SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION: '1'
benchmark:
Expand All @@ -101,8 +109,11 @@ override_c4:
roles:
agg:
args:
max-running-requests: 8
cuda-graph-max-bs-decode: 8
enable-linear-replayssm-spec: null
ple-offload-embedding: null
enable-amd-monokernel: true
max-running-requests: 4
cuda-graph-max-bs-decode: 4
env:
SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION: '1'
benchmark:
Expand All @@ -114,8 +125,11 @@ override_c8:
roles:
agg:
args:
max-running-requests: 16
cuda-graph-max-bs-decode: 16
enable-linear-replayssm-spec: null
ple-offload-embedding: null
enable-amd-monokernel: true
max-running-requests: 8
cuda-graph-max-bs-decode: 8
env:
SGLANG_AITER_HONOR_EXPLICIT_MEM_FRACTION: '1'
benchmark:
Expand Down
2 changes: 1 addition & 1 deletion inferencex-e2e/configs/amd-master.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -489,7 +489,7 @@ qwen3.5-fp4-mi355x-sglang-agentic-mtp:
# FP8 PLE, AgentX on MI355X via SGLang with native NEXTN MTP (3 steps,
# eagle-topk 1, 4 draft tokens). TP1: the ~116 GiB checkpoint fits on one MI355X.
qwen3.8next-fp4-mi355x-sglang-agentic-mtp:
image: lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261030@sha256:1ff42c238adfca9a55e8d3d3dc859a4e4e88b23e7f9340e86d17ff042b933bda
image: lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261031@sha256:3867dc07ba2530f914e1f210ed829f7a572a75286c8c362dbc22cbecf9b5a7e6
model: amd/Qwen3.8-Flash-Next-Quark-MXFP4-AttnFP8-PLEFP8
model-prefix: qwen3.8next
runner: cluster:mi355x-amds
Expand Down
9 changes: 9 additions & 0 deletions inferencex-e2e/perf-changelog.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11724,3 +11724,12 @@
- "Extend the MI355X Qwen3.8-Flash-Next SGLang AgentX sweep to concurrency 1/2/4/8/12/16 on the existing pinned upstream image and default serving path."
- "Add c1/c2/c4 recipe variants with max-running-requests and cuda-graph-max-bs-decode set to 2/4/8, preserving the existing 2x concurrency sizing and all other serving settings."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4680

- config-keys:
- qwen3.8next-fp4-mi355x-sglang-agentic-mtp
scenario-type:
- agentic-coding
description:
- "Move the MI355X Qwen3.8-Flash-Next SGLang AgentX recipe to lmsysorg/sglang-rocm:v0.5.23-rocm10-mi35x-20261031@sha256:3867dc07ba2530f914e1f210ed829f7a572a75286c8c362dbc22cbecf9b5a7e6. This nightly's SGLang pins sgl-amd-mono at b10f3b71 (sglang #47483, the merge of sgl-amd-mono #29), so the checkout setup script is removed and no patch is applied."
- "c1, c2, c4 and c8 enable the FlyDSL monokernel (--enable-amd-monokernel) for MTP target verify and draft. The kernel writes the per-draft intermediate SSM states and reads the PLE table on the GPU, so those points drop ReplaySSM and PLE offload and admit CONC requests (max-running-requests = cuda-graph-max-bs-decode = 1, 2, 4 and 8). c12 and c16 stay on the round 03 default path: ReplaySSM, PLE offload, and 2x CONC admission. Integer all-reduce stays off (ROCM_QUICK_REDUCE_QUANTIZATION=NONE)."
pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/4542
Loading