[yifehuan]

This PR adds whole-layer mono decode for Qwen3.8-Flash-Next (Quark MXFP4) on MI355X, TP1, MTP3. It now follows the layout of the other mono models: everything that is Qwen3.8's lives in atom/models/qwen38/mono/, as DeepSeek-V4.1's lives in atom/models/deepseek_v41/mono/. The model is the single-file qwen4_exp.py, so the package is named after the model, as glm52 is beside deepseek_v2.py.

Layout.

atom/models/qwen38/mono/
  config.py        the deployment contract (config_refusal)
  dispatch.py      install_mono_decode: wraps the target and the drafter after load
  runner.py        the target's 48 layers on their programs (TargetRunner)
  draft_runner.py  the drafter's layer on its program (DraftRunner)
  draft_model.py   MonoDraftModel: the loaded Qwen4ExpMTP, wrapped (as V4.1 wraps its DSpark model)
  draft_fallback.py, draft_experts.py, heads.py   the fallback draft and the certificate argmax heads
  step_meta.py     served steps, metadata args, pool checks (CPU-testable)
  ple.py           the PLE n-gram table in page-locked host memory (until #2771 merges)
  weights.py, sources.py
  kernels/         the layer programs: blocks/ (stages), device/ (gfx950 primitives), weights/ (layouts),
                   program / layer / emit / kernel, recipe + frozen schedules, and the Triton step prep and heads

Outside the package the PR changes: one line in models/registry.py (the installer); the PLE table's construction in model_ops/qwen4_exp/ple_layer.py asks ple.make_host_embedding first (17 lines, until #2771); resident_grid_refusals in atom/mono/runtime/deployment.py takes the KV connectors a model admits (empty by default, so Kimi-K3 and the DSA indexer are unchanged); EagleProposer gets one optional draft-model hook (below); the recipe, the ATOM_MONO_ENABLE row and a CLAUDE.md pointer. There are no lint exemptions: the kernels are black- and ruff-clean, and pyproject.toml is untouched.

What runs on mono. On a target MTP3 verify step with 1 to 4 requests (4 rows each), every decoder layer runs as one resident FlyDSL program: 36 GDN and 12 QSA layers; the PLE layer runs ATOM's PLE prelude first. The drafter's layer runs a BF16 program: the first draft step attends all 4 verify rows and runs the MLP on each request's last row; later steps run one row. The draft ids come from a Group64 W4 screen with a global error certificate, refined on the original BF16 rows: the BF16 head's argmax for 1, 2 and 4 rows, ATOM's argmax head for 3. Steps with more than 4 requests, prefill and mixed steps run ATOM's model; draft steps the program does not serve run the fallback draft (fused QSA attention, ATOM's BF16 MLP).

A layer program is an ordered list of stages placed on the 256 resident CTAs of one launch. Stages hand off through tagged (value, tag) mailbox regions in scratch, so one launch covers the whole layer without grid barriers. Hand-off contracts are checked at build time with atom.mono.plan.check / trace, and the register primitives and MX helpers come from atom/mono/device. Linear task shapes and schedules are frozen per (kind, batch, spec) in kernels/schedules.py; nothing is searched or cached at build time.

The drafter. install_mono_decode wraps the drafter's loaded Qwen4ExpMTP (MonoDraftModel), the way V4.1 wraps its DSpark model; spec_decode/drafter.py is untouched. EagleProposer gets one optional protocol: a draft model that sets draft_returns_head_input returns (state, head input) from forward_with_head (with selected_rows on step 0 of a decode, so only each request's last row runs the MLP) and maps the head input to ids with compute_draft_ids_from_head, in a declared draft pass and eagerly alike. Other draft models take the existing path. Qwen3.8 is an MRoPE model, so on main its draft steps 1+ have no declared pass (that is #2663) and run eagerly; the program is one launch a layer, so they cost the same as in a captured graph (below).

Deployment contract. ATOM_MONO_ENABLE (default on) is the only switch, and it only turns mono off. The GDN programs run whichever MTP verify ATOM picks: ReplaySSM, its default under MTP (one checkpoint a request plus (k, u, g) records), or the classic verify (ATOM_ENABLE_REPLAYSSM=0). They read and write ATOM's checkpoint, records and pools in place, so a request moves between mono and Native steps. config_refusal checks the deployment before any state is written (at the PLE table's construction and at install); any other deployment runs Native for the whole model, and the log names the reason (Qwen mono off: <reason>).

Item	Served configuration
Model	Full 48-layer Qwen3.8-Flash-Next Quark MXFP4 checkpoint
GPU / parallelism	MI355X (gfx950) with all 256 CUs (no compute partition), TP1, no EP / DP / PP / CP / TBO, not a plugin, no KV connector but LMCache CPU offload (lmcache_offload)
Speculative decode	MTP with 3 draft tokens; either GDN verify (ReplaySSM by default, up to 64 records)
Caches	--kv_cache_dtype fp8 --index-cache-dtype bf16 --block-size 64, FP32 GDN state, BF16 draft caches

All 16 program shapes and the fallback draft's one-row experts are built at startup. A deployment the contract admits is served whole, so a shape that does not build stops the server; ATOM_MONO_ENABLE=0 then runs it Native. --model may be a local directory or a hub id. The recipe is recipes/Qwen3.8-Flash-Next-Mono-Program.md.

Related open PRs.

• feat(spec): capture the Qwen3.8 MTP draft steps in CUDA graphs (MRoPE) #2663 (MRoPE draft-step graphs): not needed. With it, the drafter's steps 1+ run in its graphs through the same hook; without it they run eagerly at the same speed (measured below). The two PRs merge in either order.
• perf(qwen4_exp): keep the PLE n-gram table in page-locked host memory, read over UVA #2771 (the PLE n-gram table in page-locked host memory, for every deployment): mono needs the same for itself, because with its expert copy and the table in HBM the KV budget (11.5 GB) cannot hold the per-request state of --max-num-seqs 160 (19.7 GB) and the server does not start. Until perf(qwen4_exp): keep the PLE n-gram table in page-locked host memory, read over UVA #2771 merges, ple.py places the table for the deployments mono serves; it goes when this branch rebases onto perf(qwen4_exp): keep the PLE n-gram table in page-locked host memory, read over UVA #2771.

Memory. Mono keeps its own A8W4 copy of the 48 layers' routed experts next to ATOM's A4W4 tensors (the Native path still uses them), so its weights take about 81 GB more than Native's. At --gpu-memory-utilization 0.95, --max-num-seqs 160, ReplaySSM, one MI355X:

Server	KV blocks (64 tokens)	Cacheable tokens
Native, PLE table in HBM (main)	64,590	4.1M
Native, PLE table in host memory (#2771)	148,794	9.5M
Mono, PLE table in host memory (this branch)	76,952	4.9M

The decode measurements below are not limited by the pool; many long concurrent sessions are. Reading ATOM's expert layout directly would remove the expert copy, which is a follow-up.

Validation (049, one MI355X, full 48-layer checkpoint, TP1, MTP3, Zhang Ling's A_lo server). The first table is from this head; the checks after it were measured on main 825b63b before the rebase onto 63e64d8, whose commits share no code with this PR (the engram host-memory change, #2898, leaves the functions ple.py copies through unchanged).

• Native against mono by concurrency, on this head (24b2d68), same card, Native (author default: ReplaySSM, the PLE table in HBM) -> mono -> Native; 4,000 output tokens, acceptance length forced to 2.32, 3 seeds; request TPOT in ms. GSM8K: full 1319 questions with real MTP, mono, at the same client concurrency (author-default Native at c2: 1287 / 1319).

  Load (input)	Native TPOT (before / after)	Mono TPOT	Speedup (before / after)	GSM8K, mono
  C1 (64K)	3.908 / 3.886	1.877	2.08x / 2.07x	1290 / 1319 (97.8%), c1
  C2 (16K)	3.924 / 3.934	2.422	1.62x / 1.62x	1293 / 1319 (98.0%), c2
  C2 (64K)	4.017 / 3.977	2.487	1.62x / 1.60x
  C4 (16K)	4.251 / 4.234	3.674	1.16x / 1.15x	1283 / 1319 (97.3%), c4
  C8 (16K)	5.148 / 5.125	5.146	1.00x / 1.00x	1289 / 1319 (97.7%), c8

  Native drifts under 1% between its two arms. Mono serves verify steps of 1 to 4 requests, so at C8 the target runs Native and the draft runs the fallback, at Native's speed. At C4 the step with all four requests decoding takes 8.54 ms against Native's 9.85 ms (so mixed prefill steps are not what limits it): the B4 verify's 16 rows move many more expert weights than C2's 8 (5.62 ms against 9.10 ms).

• Kernels: the 16 classic programs (target delta / qsa S4, draft mtp S4 / S1, batches 1 to 4) and the 4 ReplaySSM delta programs, built from this branch and from the previous head in one session: 20 / 20 identical normalized ISA, resources and stage lists. (The sparse-attention programs build into one of two ISA variants run to run on either source: one subtract on the scalar or the vector unit.)

• Fixed load: prompts of 64K (C1) or 16K / 64K tokens (C2), 4,000 output tokens, acceptance length forced to 2.32, 3 seeds per arm, request TPOT. The restructure, with the previous head before and after it (seed-paired medians -2.3% / -2.2% at C1, within 0.7% at C2):

  Load	Previous head	Restructured	Previous head (after)
  C1 64K	1.900 ms	1.871 ms	1.895 ms
  C2 16K	2.425 ms	2.435 ms	2.437 ms
  C2 64K	2.477 ms	2.482 ms	2.490 ms

  Then the drafter's steps 1+ in feat(spec): capture the Qwen3.8 MTP draft steps in CUDA graphs (MRoPE) #2663's graphs against this branch (eager, on the program), with feat(spec): capture the Qwen3.8 MTP draft steps in CUDA graphs (MRoPE) #2663 again after it; seed-paired medians within 1%. For comparison, the same steps eager on ATOM's layer instead of the program cost +54% at C1:

  Load	With feat(spec): capture the Qwen3.8 MTP draft steps in CUDA graphs (MRoPE) #2663	This branch	With feat(spec): capture the Qwen3.8 MTP draft steps in CUDA graphs (MRoPE) #2663 (after)	Eager on ATOM's layer
  C1 64K	1.904 ms	1.882 ms	1.893 ms	2.936 ms
  C2 16K	2.437 ms	2.422 ms	2.440 ms	2.988 ms
  C2 64K	2.489 ms	2.468 ms	2.478 ms	3.009 ms

• GSM8K, full 1319 questions, real MTP, default switches (ReplaySSM): 1293 / 1319 (0.980), MTP acceptance rate 0.734 (previous head 1291 / 1319, author-default Native 1287 / 1319). Under real MTP one build's greedy text differs run to run, so builds are compared by score.

• Real-MTP smoke at 1, 2, 3, 4 and 6 requests (the last on the Native target and the fallback draft): all served, no errors, 76,955 KV blocks.

• Unit tests: CI's CPU suite (.github/scripts/run_unit_tests.sh, CPU torch, no aiter / Triton / FlyDSL) passes 11,025; its 5 failures are the same on main in my runner (3 torch-inductor tests without Python headers, 2 that need a .git checkout). 219 GPU tests pass on MI355X. Black and ruff are clean on every changed file with the repository's configuration.

Commits.

1. feat(qwen38): the mono layer programs (kernels/).
2. feat(qwen38): the target's verify steps on mono: deployment contract, runner, step metadata, weights, the PLE host table, installer, tests.
3. feat(qwen38): the MTP drafter on mono, the proposer hook, recipe and docs.
4. fix(qwen38): review: one REPLAY_MAX, token-major GDN conv pools only, the hub-id install test (ISA, GPU tests and a real-MTP smoke re-checked).

🤖 Generated with Claude Code

[github-actions]

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

• ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

• ✅ Run after the PR is approved and Pre Checkin passes
• ✅ Run immediately when an approval review is submitted
• ✅ Can be requested before approval with labels

Label	Tests
ci:full	Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom	Run native ATOM model accuracy tests
ci:vllm	Run ATOM vLLM OOT model accuracy tests
ci:sglang	Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2610 --add-label <label>

[github-actions]

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

ruff

---

⚠️ [ruff] <C408> _{reported by reviewdog 🐶}
Unnecessary dict() call (rewrite as a literal)

ATOM/tools/qwen38_mono_repro/moe_reference.py

Lines 16 to 17 in 0617ed4

	return dict(rel_l2=float(delta.norm() / reference.norm().clamp_min(1e-12)),
	max_abs=float(delta.abs().max()), exact=bool(torch.equal(actual, reference)))

---

⚠️ [ruff] <C408> _{reported by reviewdog 🐶}
Unnecessary dict() call (rewrite as a literal)

ATOM/tools/qwen38_mono_repro/moe_reference.py

Lines 50 to 53 in 0617ed4

	report = dict(rank=path.stem, router=error(field('router_logits', torch.bfloat16), logits),
	routes=ids.tolist(), global_routes=expected_ids.tolist(),
	route_weights=error(route_weights[active], expected_weights[active])
	if bool(active.any()) else None)

---

⚠️ [ruff] <C408> _{reported by reviewdog 🐶}
Unnecessary dict() call (rewrite as a literal)

ATOM/tools/qwen38_mono_repro/moe_reference.py

Line 69 in 0617ed4

details.append(dict(slot=slot, mid=error(middle[slot], expected.bfloat16())))

---

⚠️ [ruff] <C408> _{reported by reviewdog 🐶}
Unnecessary dict() call (rewrite as a literal)

ATOM/tools/qwen38_mono_repro/moe_reference.py

Lines 102 to 103 in 0617ed4

	endpoint = dict(oracle_vs_mono=error(oracle, analyzed[0][2]),
	oracle_vs_native=error(oracle, analyzed[0][3]))

---

⚠️ [ruff] <C408> _{reported by reviewdog 🐶}
Unnecessary dict() call (rewrite as a literal)

ATOM/tools/qwen38_mono_repro/moe_reference.py

Line 104 in 0617ed4

report = dict(endpoint=endpoint, ranks=[row[0] for row in analyzed])

[ZLkanyo009]

@yifehuan Thanks for adding Quark MXFP4 + FP8 KV. We tried 42b73abf on top of our Qwen3.8-Flash-Next stack (amd/Qwen3.8-Flash-Next-Quark-MXFP4, MI355X, TP1, AgentX inferencex-agentx-mvp, 3600 s, seed 42, two rounds on two nodes, averaged). Two findings and the numbers:

1. On current aiter main, Mono declines every build. The startup log has Qwen mono on. Then each of width 4 / 2 / 1 fails with MonoUnsupported: Quark route/quant consumer ABI changed and logs Qwen mono declined the N-row build, so all decode runs natively, with no error surfaced. The cause is that kernels/quark_moe.py::validate_native_config expects the M = 1 / 2 / 4 MoE config to be block_m=32, prequant, FlyDSL stage 1, no output_aux. ROCm/aiter#6708 (merged 10-26) added tuned rows for tokens = 1 / 2 / 4 in aiter/configs/model_configs/qwen3_8_flash_fp4_tuned_fmoe.csv, and those are inline-sort configs. Dropping those three rows (via AITER_CONFIG_FMOE) makes Mono build for widths 1 / 2 / 4. It might be worth having Mono select its MoE config explicitly instead of through the tuned table, or fail loudly when validation rejects it.

2. Mono's admission excludes MTP and online quantization. config_refusal rejects speculative decode, and uses_native_mxfp4_moe requires not online_quant, so our MTP 3 + online PTPC (--online_quant_config) deployment cannot use it. Prefix caching stays on in our runs. A long-prefix hit check and GSM8K were fine with Mono and our state checkpoints: 0.995 on 200 questions at concurrency 4.

	c1	c2	c4
Mono on vs off, same config (MTP off, PTPC off), tput/GPU	+7.2%	+9.1%	+10.7%
same, p90 ITL interactivity	+26%	+25%	+22%
Mono line vs our MTP 3 + PTPC config, tput/GPU	−9.8%	−23.2%	−14.2%
same, p90 ITL interactivity	−26%	−27%	−11%

So Mono is a clear win over the native path at the same settings. On this model, though, turning off MTP and PTPC costs 16–30% throughput, more than Mono recovers. With Mono declining (MTP on), #2610 does not change the native path (c1 / c2 / c4 within ±1.3%).

Minor: with Mono on at c4, one warmup request per round (max_tokens=1) came back with usage but no content (InvalidInferenceResultError in aiperf). The server logged no error, and the same requests were fine with Mono off.

Is MTP (and online-quant) support on the roadmap? On MI355X, Qwen3.8's low-concurrency points depend on MTP.

[yhl-amd]

@valar-cop

[yifehuan]

@ZLkanyo009 an update for your l35 lines on this head (198057d). Both of them now enable Mono by default, as long as your tree carries this PR together with #2636. It needs #2655 too: with FP8 QSA caches, lmcache_offload needs the scale planes registered.

• A_lo (OFFLOAD=0). Run MONO=1 with nothing else. Your script's default MONO=0 still exports ATOM_MONO_ENABLE=0, which is your Native control. ATOM_ENABLE_REPLAYSSM=0 and the QWEN_* exports are no longer needed: Mono runs ATOM's default ReplaySSM verify. Measured on this branch: every shape served, the KV pool at 76,953 blocks, fixed-load TPOT at parity with the earlier classic-verify build, and full GSM8K 1319 at 1291 / 1319 with real MTP.
• A (OFFLOAD=1, lmcache_offload). Mono now runs beside the connector; any other connector is still refused. I checked this on this branch merged with fix(qwen4-exp): register the QSA FP8 scale planes for KV offload #2655 (which carries [Qwen 3.8] LMCache offload for QSA + GDN + PLE state #2636). Settings: your A line with a small GPU pool (UTIL 0.60, MAX_SEQS 16). Load: 32 prompts of 12K tokens, then three Mono decodes while 16 of those prompts reloaded from the CPU tier. That moved 195,584 KV tokens and 16 GDN state images, with no errors or timeouts. Decode event gaps matched Native's (max 89 vs 87 ms). GSM8K 200 with offload scored 0.985.

I have not run your actual integration tree. When this PR goes into it, expect conflicts in three files, because this PR carries the same mechanisms as two of yours:

• atom/spec_decode/draft_graph.py and atom/spec_decode/eagle_proposer.py overlap feat(spec): capture the Qwen3.8 MTP draft steps in CUDA graphs (MRoPE) #2663, the batch-last MRoPE staging.
• atom/model_ops/qwen4_exp/ple_layer.py overlaps perf(qwen4_exp): keep the PLE n-gram table in page-locked host memory, read over UVA #2771, the host PLE table.

I expect your side of those files to work with Mono, since Mono's PLE layer runs ATOM's own PLE first, but I have not tested that combination. Once #2663 and #2771 land, this PR will drop its copies.

If you retest in your tree, the startup log should show these lines:

• Qwen mono: GDN verify ReplaySSM (16 records)
• Qwen mono: 48 layers, ... served [(1, 4, 4096), (2, 4, 4096), (3, 4, 4096), (4, 4, 4096)]
• Qwen mono draft: layer 48 (bf16) hooked, ... served [...], listing 8 shapes

If something refuses, the log shows Qwen mono off: <reason>.

[yifehuan]

@valar-cop could you re-review? Since your review of 2b24902, the head is now 198057d.

• Your six points, one commit each, with replies in each thread:
  • r3 whole_layer lint: be0d1e2
  • r5 one-row experts built at startup: 7d77cf4
  • r2 resident_grid_refusals: 29760bb
  • r1 hub-id checkpoint: 50ca5cf
  • r4 env doc: f6c8ee1
  • your earlier "silent per-width decline" point: f6c8ee1, where a shape that does not build now stops startup.
• r6 (C4 max_tokens=1, first-step stall): re-measured, with results in the runner.py:311 thread. Both behave the same on Native and Mono.
• ReplaySSM (194fd9e, 20ac535): the GDN programs now run ATOM's default ReplaySSM verify, so Mono engages with no engine switch; ATOM_MONO_ENABLE=0 only turns it off. Results: KV pool 32,284 → 76,953 blocks, fixed-load TPOT at parity, full GSM8K 1319 at 1291 / 1319 with real MTP.
• r2 follow-up (198057d): lmcache_offload is allowed beside the resident programs, and every other connector is still refused. The evidence is in the r2 thread.

Pre Checkin is green on 198057d, and the heavy CI is queued. The commits will be squashed at merge.

[valar-cop]

some changes needed..

ci green.

[valar-cop]

ci triage for 24b2d68:

ATOM SGLang Test passed. https://github.com/ROCm/ATOM/actions/runs/41751457080

ATOM vLLM Test passed. https://github.com/ROCm/ATOM/actions/runs/41751457242

ATOM Test passed. https://github.com/ROCm/ATOM/actions/runs/41751457240

[yifehuan]

Restructured and force-pushed (198057d → 28389c0, rebased on main af4d581), following the layout of the other mono models:

• Layout. Everything that is Qwen3.8's is in atom/models/qwen38/mono/, like deepseek_v41/mono/. The package is named after the model, as glm52 is, since qwen4_exp.py is a single file. atom/mono/program/ moved to qwen38/mono/kernels/; what remains in atom/mono is a 7-line change (resident_grid_refusals takes the KV connectors a model admits, empty by default).
• Drafter. It is wrapped at install (MonoDraftModel, as V4.1 wraps its DSpark model), so spec_decode/drafter.py is untouched. EagleProposer keeps one optional hook (draft_returns_head_input / forward_with_head / compute_draft_ids_from_head).
• No lint exemptions. pyproject.toml is untouched; the kernels are black- and ruff-clean. The 20 programs (16 classic, 4 ReplaySSM) build to the same normalized ISA as the previous head.
• feat(spec): capture the Qwen3.8 MTP draft steps in CUDA graphs (MRoPE) #2663 is not needed. Without it the draft steps 1+ run eagerly on the mono program, within 1% of running them in feat(spec): capture the Qwen3.8 MTP draft steps in CUDA graphs (MRoPE) #2663's graphs (C1 1.882 ms against 1.904 / 1.893 ms), so the two PRs merge in either order. @ZLkanyo009
• The PLE host table stays (qwen38/mono/ple.py) until perf(qwen4_exp): keep the PLE n-gram table in page-locked host memory, read over UVA #2771 merges. With the table in HBM, mono's KV budget (11.5 GB) cannot hold the per-request state of --max-num-seqs 160 (19.7 GB), so the server does not start.
• Tests: tests/test_qwen38_mono_{plan,prep,ple}.py.

Same-card fixed load against the previous head: C1 1.900 / 1.871 / 1.895 ms, C2 within 1%. Full GSM8K with real MTP: 1293 / 1319. Details are in the description. Earlier replies in this thread name the old paths (whole_layer/, qwen4_exp_mono/); the fixes they describe are unchanged.

[valar-cop]

some changes needed..

ci Run non-GPU unit tests (Pre Checkin) failed — not sure yet (low): The captured log is 0 bytes, so the failure cannot be classified. The same job failed on main at 2026-11-03T14:17 with successes before it, so main-broken is plausible but unproven; the PR does add CPU tests importing new qwen38 modules, which I checked import cleanly without aiter/flydsl.

next: Re-run the job and attach the log; if it reproduces, check whether the new CPU tests collect (they use importorskip only for flydsl). https://github.com/ROCm/ATOM/actions/runs/41653380070/job/125977139564

ci ATOM Test failed — not sure yet (low): The run is reported as failure but no failed job or log was captured, so nothing can be attributed to this PR.

next: Inspect the run's job list for the failing job and its log. https://github.com/ROCm/ATOM/actions/runs/41653380244

[yifehuan]

Notes on the changes outside atom/models/qwen38/: why each one is there.

[yhl-amd]

@valar-cop can we merge

[valar-cop]

@yhl-amd lgtm. ci green on 24b2d68, and my last three check out in the code: one REPLAY_MAX from geometry.py, the dead transpose branch gone, hub-id install test back. the aiter source pin is gone too, so a plain wheel install works now. c8 p90 is the only thing i don't love, but it's small; merge.