[iraj465]

Overview

Run three tuned large-M MXFP8 block32 GEMM shapes on Gluon kernels on gfx950, and group output tiles for two shapes. GEAK-EVO discovered these implementations for DeepSeek-V4.1-Flash TP2 on MI355X; this port preserves their kernel bodies and launch code with compatibility changes.

This updates the earlier grouped-only proposal. Its source and separate historical +4.73% INT4 serving observation are preserved in this note; that observation is not attributed to this patch.

Claims

• Measured callable speedup: 1.1448×, call-weighted across 21 captured prefill cases / 1,590 calls, median of three ORIGINAL–candidate–ORIGINAL rounds on MI355X. The comparator is the unpatched installed vLLM nightly a2bcb5657.
• Byte-identical output on all 63 qualification draws, with input contents, output dtype, shape and strides preserved. All 14 new and 183 existing items in the selected test file passed.
• Eligibility is restricted to gfx950, BF16 output and 8193 <= M <= 16384. Gluon dispatch additionally requires one of three exact (N, K, cfg) keys, prequantized input and aligned weight scales. Other calls retain the existing path, including quant_in_kernel.
• Current AgentX E2E gain: not measured. This callable result is not a workload roofline or serving-throughput result.

Validation

Completed November 6, 2026, on one MI355X/gfx950 GPU:

Item	Result
Runtime	Image vllm/vllm-openai-rocm@sha256:03ad7d2c72ee2e7ec95e14dfc53d6f91c98f38503f5bb752874900064d4987f2; vLLM 0.33.0rc1.dev806+ga2bcb5657; Torch 2.13.0+rocm10.0.0; HIP 7.15.26333
Triton	Distribution 3.8.0+git669b31ac, verified to own the imported module (__version__ == 3.8.0)
Installed source and test origin	Original blobs checked before patching; exact tested candidate hashes verified; no checkout-package shadowing
Selected test file	197 passed: 14 new + 183 existing, zero failures/errors/skips
Qualification output / inputs	63/63 byte-identical outputs, inputs preserved
Path engagement	Gluon exercised for exactly 9 cases; grouped tiles for 3; unchanged path for 9
Three weighted round ratios	1.1448779177 / 1.1447676676 / 1.1434877993
Median / descriptive range	1.1447676676× / 0.0013901184; range is not a confidence interval

Tests came from source commit a2bcb5657a881a002e79d0adab3c6b5e39436121 and imported the patched installed package. Checkout tests/conftest.py, including its autouse cleanup and dynamo-reset fixtures, was excluded. This is the selected test file, not the full upstream suite. Its existing tests passed, so no separate unpatched-main suite rerun was needed. The benchmark separately measured ORIGINAL and candidate.

Linear family	N × K	M=16384	M=10732	M=9199
query (wq_b), Gluon mixed	16384 × 1280	1.3482	1.3395	1.2896
engram, Gluon output + grouping	12800 × 6144	1.2111	1.2646	1.2587
output (wo_b), Gluon output	5120 × 4096	1.1650	1.1465	1.1274
shared_gate_up, grouped tiles	2304 × 5120	1.0604	1.0444	1.0174
fused_qkv, unchanged	1792 × 5120	1.0012	0.9970	0.9944
shared_down, unchanged	5120 × 1152	1.0216	1.0061	1.0260
index_query, unchanged	4096 × 1280	1.0012	1.0247	0.9929

Each cell is the median of three per-case ratios. Unchanged rows' variation is descriptive, not a calibrated noise floor or evidence of an every-case gain.

Exact benchmark, qualification driver, cases, patch and completed results, with file hashes and reproduction instructions. The executed entry was run_qual.py --layout installed --test-src <a2bcb5657 source> --prepatch-record <fresh record>, after its prepatch check and application of the exact provided patches.

Timing uses eager CUDA events, 10 warmups, the upper median of 50 samples, and a 512 MiB read-evict before each timed sample. Each case is bracketed ORIGINAL–candidate–ORIGINAL in one process. Per round: sum(calls * mean(O_before, O_after)) / sum(calls * candidate). The scope is the 21 tuned cases, BF16 output, contiguous inputs and seeds 0–2; other configurations, strides, quant_in_kernel, other M and MI350X are not benchmarked here.

Ruff 0.14.0 check/format, SPDX and forbidden-import checks passed in port preparation; git diff --check passed on the delivered tree. The existing MI355 DPX Buildkite kernel step selects this test file and covers the changed kernel paths in its source dependencies. Hosted CI has not yet supplied a GPU result for this revision.

Details

• wq_b: two independent 64-row programs replace each logical 128×256 tile. Double-buffered asynchronous activation copies feed ordered scaled-MFMA steps; weights are repacked into 32×32 panels and loaded into registers.
• wo_b / engram: asynchronous copies stage both operands while the prior MFMA runs. Scales are repacked per call; engram retains 4M×8N tile grouping.
• shared_gate_up: 8M×4N tile grouping preserves output ownership and arithmetic.
• The Gluon module loads AMD support explicitly. Missing imports fall back to Triton with a warning; the gfx950 import test fails if the intended Gluon table is unavailable.

The patch adds 523 non-test lines and uses experimental Gluon APIs. wq_b repacks roughly 21 MB of weight data each call, adding a transient buffer and its read/write traffic. No packed copy survives across calls. Serving, natural quality and model-level graph-replay validation remain unmeasured; the draft does not assert that those obligations or human submitter line review are complete.

The earlier EN01 pre-port callable result was 1.1277×, on a different runtime/source. It used the same warmup/sample/evict parameters but fresh-process whole-case O–X–O legs; this qualification uses per-case O–X–O in one process. They are distinct measurements.

Overlap checked: #60169 adds AITER FlyDSL dispatch and may bypass this path when AITER linear is enabled; it does not supply these Gluon kernels. #63821 and #64459 tune other shape entries. #65716 overlaps the architecture flag and may require a rebase. A refreshed open-PR search found no duplicate Gluon implementation for this callable.

Provenance / AI assistance: GEAK-EVO generated candidate c421256c0ceb6afc9d6115789d4e2f08c0a226ecaa7f74ae5addc207774e9e56. Claude prepared the port and qualification; Codex and the third researcher independently reviewed the source and results. The port preserves the generated kernel AST except for two documented dead-code deletions and the compatibility bindings. This result does not establish an EVO-versus-GEAK method advantage.

---

Pull Request Checklist

• I used vLLM's /pr-checklist skill. (Mandatory for agents, optional for humans).

• AI assistance was used during the creation of this PR.

• Design Fit: Minimizes impact on core components, reuses existing functionality, and justifies added complexity.

• Testing and Validation: Validates the change and ensures any added tests are meaningful and reliable, with CI coverage or documented CI resource constraints and validation performed outside CI.

• Code Quality and Style: Keeps code and comments clear and concise, and updates relevant documentation and examples.

• Pull Request Contents: Includes a brief summary and relevant links, supports claims with evidence, explains root causes and implementation trade-offs, and follows the contributing guide.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

[iraj465]

Preserving the previous candidate's evidence before updating this PR with the separately qualified Gluon candidate:

The earlier grouped-tile candidate remains in commit 30eccf8d68efcf57fc573c829ce2fa261df88cff. Its captured-callable results were 1.03817× on the first historical host and 1.02575× in a separate post-hoc diagnostic on a second host. Five of the 21 measured shapes regressed on both hosts.

One historical DeepSeek-V4.1-Flash TP2/EP1 C64 serving pair, 3,600 scored seconds per leg, recorded 172,722.19689 → 180,896.34475 total tokens/s/GPU (+4.73254%). It used the October 27 recipe with QuickReduce INT4 and image vllm/vllm-openai-rocm:nightly-e68efc685ab1e6f1eda004ced7157e3ca411da2f, preceding the NONE recipe in InferenceX #4666.

This was one sequential pair without replication. Official baseline/candidate quality evaluation did not complete; scored-window optimized-path engagement was not established; another server startup overlapped only ORIGINAL's scoring window. Scored successes/errors were 9,190/0 and 9,599/3. The historical integration restricted eligibility to captured eager main-model calls, whereas that original upstream port exposed a broader dispatch domain.

Consequently, +4.73% is a historical observation, not an established causal effect or an accepted improvement over current AgentX. It belongs to the previous candidate and must not be attributed to the new Gluon patch. The new patch has its own completed MI355X callable qualification; current AgentX E2E gain for it is not measured.