Qwen3.8-27B on AMD MI355X: 2.67x tuned vLLM ROCm recipe

[prashant182]

Adds qwen3.8-27b-mi355x/: the vLLM ROCm configuration we tuned for Qwen3.8-27B (Qwen/Qwen3.8-27B-FP8, FP8 block-scale weights, 48 Gated DeltaNet + 16 full-attention layers, one MTP layer) on one AMD Instinct MI355X, with the harness and every measurement behind it. The run is recorded in OpenRelayInc/OpenRelay#2469.

Results

One MI355X, TP1, amd-mi355-node-146 (ROCm 7.2), same host and harness before and after. Workload: the production 5:1 shape (~2,480 prompt tokens of unique random text, exactly 500 output tokens, thinking off), closed loop. Baseline is what production runs: vllm/vllm-openai-rocm:v0.26.0 with the production flags.

Metric	Baseline	Tuned	Change
Output tokens/min per GPU at 48 concurrent	55.4K	148.2K	2.67x
Same, desynchronized stream starts	54.5K	146.6K	2.69x
Knee (most streams with TTFT p95 <= 2 s)	4	64 (the --max-num-seqs ceiling)	16x
Prefill tok/s, 16 concurrent 8K prompts	7.5K	27.9K	3.7x
Mean TPOT at 16 / 48 concurrent	20.4 / 44.6 ms	10.7 / 18.7 ms	-48% / -58%
GSM8K, first 250, thinking off	97.2%	97.2% (98.0% final validation)	unchanged
Cold start / warm restart	230 / 69 s	316 / 100 s	+86 / +31 s

What the stack is: stock vllm/vllm-openai-rocm:v0.30.0 plus

• 103 tuned aiter a8w8_blockscale_bpreshuffle gfx950 rows for the model's five FP8 linear shapes (2.0x throughput, 3.4x prefill on their own: without them every GEMM falls back to CK's default ABScale instance at ~0.46 PFLOP/s);
• one aiter unified-attention entry, D_GEQ_256.Q_GEQ_256.DT_fp8_fp8 (BLOCK_M 128), restoring the prefill tile for head_dim 256 fp8 (+15% prefill);
• --mamba-ssm-cache-dtype bfloat16 (+5% throughput);
• MTP with 3 draft tokens (+21% at c48, +35% at c16).

The ablation (one change per stage), knee ladders, kernel profiles and the full list of discarded experiments are in results/. bench/README.md has the exact command behind each table.

Review notes

• Read every file once before committing: no credentials (the load generator reads its key only from OR_KEY, and the local runs pass a placeholder), no workstation paths. The host paths in bench/common.sh and the tuner scripts are the measurement host's defaults and are documented as overridable.
• Two stale comments fixed in passing: bench/kernels/tune_gemm.sh named itself tune.sh and said six shapes (there are five), bench/profreq.py named the Gemma model as its default.
• Root README.md gets a row for the new recipe.
• A second commit fills the README's Upstream table with what was found on upstream main: the GEMM rows went to [Config] Add Qwen3.8-27B TP1 a8w8 blockscale B-preshuffle GEMM tunings for gfx950 ROCm/aiter#6033 (re-tuned on aiter main, 119 rows); the attention entry is already on aiter main ([Triton/Gluon] [Config] Enable unified-attention skip-mask for gfx950 hd256 FP8 prefill ROCm/aiter#5598); the vLLM fallback gate no longer exists on vLLM main ([ROCm][Bugfix] Revert #56433 + #51692 to fix accuracy breakdown for DeepSeek-V4 vllm-project/vllm#57132 reverted the preshuffled kernel); the GDN decode fix is on vLLM main as [ROCm][Perf] Enable the AITER GDN decode fast path for flat qkvz layouts vllm-project/vllm#53623.

[graphify-labs]

Graphify reviewed this change.

Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.

---

Graphify review — findings

Adds a Qwen3.8-27B FP8 recipe for one MI355X: stock vLLM v0.30.0 ROCm plus tuned aiter block-scale GEMM rows and a head_dim-256 fp8 attention prefill tile, run with bf16 GDN state and 3-token MTP. Against the production v0.26.0 config it reaches 2.67x output tokens per GPU at 48 concurrent with GSM8K unchanged, at the cost of about 86 s more cold start. patch_gdn_aiter_decode.py stays out of the image because MTP never takes its path, and the bench harness and per-change ablation ship alongside.

Worth a look

• Base image is not pinned by digest — qwen3.8-27b-mi355x/Dockerfile:4 · Escalate · medium
  • agreed by 2 of 2 members but NOT verified (no proof, no reproducing execution) — consensus is not a verdict; needs human review

Review partial — this diff was larger than one review pass covers, so later files were not reviewed; some findings may be missing.

Analysis details — impact, health, verification

Impact & health

Graphify review

Impact — 111 functions depend on the 111 functions this change touches.

Health — grade A; no new coupling hotspots.

Verification — 111 functions in the blast radius were not formally verified this run (proofs are advisory here).

Gate & verification

graphify gate

PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.

Advisory (not blocking):

• verification_scope: 111 function(s) in the blast radius were not formally verified this run

Risky patterns (advisory)

• medium no-stderr-to-stdout in qwen3.8-27b-mi355x/bench/kernels/tune_gemm.sh:10: stderr redirected to stdout: docker rm -f qopt-tune >/dev/null 2>&1 true
• medium no-stderr-to-stdout in qwen3.8-27b-mi355x/bench/kernels/tune_gemm.sh:15: stderr redirected to stdout: -o /tune/qwen38_bpreshuffle_tuned.csv -o2 /tune/qwen38_bpreshuffle_all.csv --preshuffle --libtype all --mp 4 2>&1 tee
• medium no-stderr-to-stdout in qwen3.8-27b-mi355x/bench/kernels/tune_gemm2.sh:10: stderr redirected to stdout: docker rm -f qopt-tune2 >/dev/null 2>&1 true
• medium no-stderr-to-stdout in qwen3.8-27b-mi355x/bench/kernels/tune_gemm2.sh:15: stderr redirected to stdout: -o /tune/qwen38_bpreshuffle_tuned2.csv -o2 /tune/qwen38_bpreshuffle_all2.csv --preshuffle --libtype all --mp 2 2>&1 te
• medium no-stderr-to-stdout in qwen3.8-27b-mi355x/bench/launch.sh:16: stderr redirected to stdout: docker rm -f "$name" >/dev/null 2>&1 true
• medium no-stderr-to-stdout in qwen3.8-27b-mi355x/bench/run.sh:4: stderr redirected to stdout: "$BENCH_DIR/gate.sh" "$1" "$2" > "$WORK/results/measure-$1.out" 2>&1
• medium no-stderr-to-stdout in qwen3.8-27b-mi355x/bench/run.sh:5: stderr redirected to stdout: "$BENCH_DIR/measure.sh" "$1" "$2" ${3:-} >> "$WORK/results/measure-$1.out" 2>&1
• medium no-stderr-to-stdout in qwen3.8-27b-mi355x/bench/waitready.sh:8: stderr redirected to stdout: if ! docker ps --format '{{.Names}}' grep -qx "$name"; then echo "$name exited"; docker logs --tail 40 "$name" 2>&1; e
• medium no-unbounded-fetch in qwen3.8-27b-mi355x/bench/profreq.py:10: unbounded read (no limit/bound): return urllib.request.urlopen(req, timeout=900).read()
• medium no-unbounded-fetch in qwen3.8-27b-mi355x/bench/table.py:21: unbounded read (no limit/bound): g = open(f"{W}/{tag}/gsm8k.txt").read().split("=")[1].split("%")[0].strip() + "%" if os.path.exists(f"{W}/{tag}/gsm8k.tx

…and 2 more.

Docs that may be stale (advisory)

• gemma-4-31b-mi355x/README.md § Results (lines 12-49): references changed symbols rows
• gemma-4-31b-mi355x/README.md § Run (lines 94-133): references changed symbols num
• gemma-4-31b-mi355x/README.md § What each piece does (lines 169-231): references changed symbols num
• gemma-4-31b-mi355x/bench/README.md § Scripts (lines 25-70): references changed symbols num
• gemma-4-31b-mi355x/bench/README.md § Exploration (exploration.md) (lines 126-169): references changed symbols num
• gemma-4-31b-mi355x/results/ablation.md § Ablation: one change per stage, on one GPU (lines 1-29): references changed symbols num
• gemma-4-31b-mi355x/results/exploration.md § Engine configs, correctness gate and prefill probe (lines 13-45): references changed symbols num
• gemma-4-31b-mi355x/results/exploration.md § max-num-seqs 128 (discarded) (lines 100-117): references changed symbols num
• gemma-4-31b-mi355x/results/knee.md § Knee: offered load against p95 time to first token (lines 1-37): references changed symbols num
• gemma-4-31b-mi355x/results/migration-and-rollout.md § Day one: finding a working MI355X config (lines 11-32): references changed symbols num

…and 1 more.