perf(rocm): use Kimi-K3 B1 gate projection - #20
JohnQinAMD wants to merge 1 commit into
Conversation
Route only the AMD Kimi-K3 router through a compile-safe AITER custom op. Preserve GateLinear and exact FP32-logit fallback semantics for opt-out, older AITER builds, unsupported shapes, and other platforms. Assisted-by: OpenAI Codex Signed-off-by: Yanyuan Qin <yanyuan.qin@amd.com>
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment Once the PR is approved or has the If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
Summary
Route the AMD Kimi-K3 router projection through the narrow AITER gfx950 kernel
while preserving
GateLinearbehavior for explicit AITER opt-out, older AITERbuilds, unsupported shapes, and every non-AMD model path.
This PR depends on JohnQinAMD/aiter-amd#6.
Design
GateLinearsubclass.implementation.
No generic router tier, NVIDIA model, model-name condition, or routing/sort
policy is changed.
Performance
On MI355X/gfx950, the fixed
1x7168 @ 896x7168.Tprojection improves from13.69 to 4.21 us (3.13x). The complete route boundary improves 1.55x. In the
accepted TP8 B1 stack, this change advances decode from 50.063 to 50.635
tok/s/GPU (+1.1%).
Correctness and validation
GateLinear, fake/custom-op schema,AITER delegation, and explicit opt-out fallback.
logits; AITER perf(rocm): fuse Kimi-K3 MLA output gate #6 carries the fixed-shape numerical tests.
3.10 mypy, SPDX, import guards, configuration checks, and the CUDA API guard.
git diff --checkpasses.accuracy gates; no isolated model-level accuracy delta is attributed to this
dispatch-only PR.
Disclosure
Developed with assistance from OpenAI Codex. The author reviewed the custom-op
contract, fallback semantics, tests, and performance evidence.