perf(moe): add Kimi-K3 B1 prepared A8W4 pipeline - #2
Closed
JohnQinAMD wants to merge 1 commit into
Closed
JohnQinAMD wants to merge 1 commit into
JohnQinAMD wants to merge 1 commit into
Conversation
Add a fail-closed gfx950 batch-one pipeline that prepares stable route metadata and MXFP8 activation once, then consumes it through tuned MXFP4 expert GEMMs and weighted BF16 reduction. The specialization requires the exact native TP8 checkpoint layout and routing semantics. Unsupported configurations retain the existing path. Assisted-by: OpenAI Codex Signed-off-by: Yanyuan Qin <yanyuan.qin@amd.com>
This was referenced Jul 29, 2026
Owner
Author
|
Superseded by DCO-clean, patch-equivalent replacement #22. The replacement adds the required Signed-off-by trailers without force-pushing this reviewed branch; performance, correctness, and accuracy evidence is preserved. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
perf(moe): add Kimi-K3 B1 prepared A8W4 pipeline
Summary
Add a narrow gfx950 Kimi-K3 batch-one MoE pipeline that owns route metadata,
MXFP8 activation quantization, tuned A8W4 expert GEMMs, SiTU, and weighted BF16
reduction behind one typed request.
The fast path is fail-closed. It requires the exact native TP8 checkpoint
layout, validated kernel registry entry, routing semantics, device, dtypes, and
runtime features. Unsupported inputs return to the existing path.
Design
preserving exact incumbent tie behavior.
production variant rather than widening the generic tuning combinations.
native-384 stage-2 atomic weighted reduction.
tensor layout, model configuration, and runtime-feature contracts.
prepared routing exactly once.
Performance
MI355X, one GPU microbenchmark, rotating through all 896 experts:
On the full TP8 Kimi-K3 endpoint (B1, 8,192 input tokens, 1,024 output tokens,
no speculative decoding), two measured runs were 69.7324 and 69.8257 tok/s.
The 69.7791 tok/s mean is 8.56% above the hash-pinned 64.2744 tok/s parent and
reduces mean TPOT by 1.2275 ms.
The final clean signed commit was rebuilt as a separately hash-pinned image
and confirmed at 69.4669 and 69.8105 tok/s (69.6387 mean), within 0.21% of
the accuracy-tested image.
Correctness and tests
git diff --check: passed.response. The hash-pinned parent scored 1,244/1,319 (0.943139), with two
invalid responses; paired outcomes were 32 candidate wins and 20 losses.
Scope and dependencies
This is stacked on
JohnQinAMD/aiter-amd:perf/kimi-k3-preroute-bf16.It intentionally excludes the rejected direct-BF16 stage-1 experiment and does
not change the generic Kimi-K3 tuning CSV.
The matching vLLM integration is a separate PR against
JohnQinAMD/vllm-amd.AI assistance
OpenAI Codex assisted with implementation, profiling analysis, test
construction, and PR preparation. The human submitter reviewed the changed
lines and is responsible for the design, measurements, and submission.