perf(flydsl): Kimi-K3 B1 route-sort + prepared-MoE handoff - #29
Open
JohnQinAMD wants to merge 4 commits into
Open
JohnQinAMD wants to merge 4 commits into
JohnQinAMD wants to merge 4 commits into
Conversation
Add a narrow FlyDSL specialization for the 896-expert, top-16, single-group decode shape. Preserve AITER tie ordering and emit Opus-compatible metadata directly from the routing launch. Assisted-by: Codex Signed-off-by: Yanyuan Qin <yanyuan.qin@amd.com>
Register one validated BF16 x MXFP4 16-row FlyDSL variant and advance it with the producer 32-row sort-bucket stride. Select it only through the existing exact gfx950 Kimi-K3 B1 tuning row. The complete expert boundary improves from 20.70 us to 18.20 us across two independent processes, with 90/90 adversarial correctness cases passing. Signed-off-by: Yanyuan Qin <yanyuan.qin@amd.com>
Select the existing direct atomic weighted-reduction kernel through the exact gfx950 Kimi-K3 B1 tuning row. Larger batch rows and generic dispatch remain unchanged. The complete expert boundary improves from 20.70 us to 18.62 us across two independent processes, with 90/90 adversarial correctness cases passing. Signed-off-by: Yanyuan Qin <yanyuan.qin@amd.com>
Add a typed, fail-closed gfx950 B1 handoff that owns Kimi-K3 routing and feeds its prepared metadata directly into the existing MXFP4 expert stages. Support both production A8W4/interleaved and A16W4/separated layouts while preserving fallback outside the exact shape contract. The live A8W4 route-to-expert boundary improves from 41.83-42.01 us to 25.40-25.41 us (1.647-1.653x), with exact route IDs, 6.66e-8 routing-weight RRMSE, zero output RRMSE, and no duplicate route/sort launch. Assisted-by: Codex Signed-off-by: Yanyuan Qin <yanyuan.qin@amd.com>
🏷️ CI GuideRuns automatically on every PR:
Extended tests (opt-in via labels):
|
| num_valid_ids: fx.Pointer, | ||
| moe_buf: fx.Pointer, | ||
| moe_buf_i32_elements: fx.Int32, | ||
| stream: fx.Stream = fx.Stream(None), |
|
|
||
|
|
||
| def _register_production_variants_stage1( | ||
| kernels: Dict[str, Dict], a_dtype: str, b_dtype: str, out_dtype: str |
|
|
||
|
|
||
| def _register_production_variants_stage1( | ||
| kernels: Dict[str, Dict], a_dtype: str, b_dtype: str, out_dtype: str |
Comment on lines
+4
to
+23
| import importlib | ||
| import math | ||
| from dataclasses import replace | ||
|
|
||
| import aiter | ||
| import pytest | ||
| import torch | ||
| from aiter import dtypes | ||
| from aiter.fused_moe import fused_moe_2stages, moe_sorting | ||
| from aiter.jit.utils.chip_info import get_gfx_runtime | ||
| from aiter.ops.flydsl.kimi_k3_moe_handoff import ( | ||
| KimiK3MXFP4ExpertRequest, | ||
| KimiK3MXFP4W13Layout, | ||
| kimi_k3_mxfp4_expert_handoff, | ||
| kimi_k3_mxfp4_expert_mode, | ||
| supports_kimi_k3_mxfp4_expert_handoff, | ||
| ) | ||
| from aiter.ops.flydsl.moe_common import GateMode | ||
|
|
||
| pytestmark = pytest.mark.skipif( |
There was a problem hiding this comment.
Import block is un-sorted or un-formatted
Suggested change
| import importlib | |
| import math | |
| from dataclasses import replace | |
| import aiter | |
| import pytest | |
| import torch | |
| from aiter import dtypes | |
| from aiter.fused_moe import fused_moe_2stages, moe_sorting | |
| from aiter.jit.utils.chip_info import get_gfx_runtime | |
| from aiter.ops.flydsl.kimi_k3_moe_handoff import ( | |
| KimiK3MXFP4ExpertRequest, | |
| KimiK3MXFP4W13Layout, | |
| kimi_k3_mxfp4_expert_handoff, | |
| kimi_k3_mxfp4_expert_mode, | |
| supports_kimi_k3_mxfp4_expert_handoff, | |
| ) | |
| from aiter.ops.flydsl.moe_common import GateMode | |
| pytestmark = pytest.mark.skipif( | |
| import importlib | |
| import math | |
| from dataclasses import replace | |
| import pytest | |
| import torch | |
| import aiter | |
| from aiter import dtypes | |
| from aiter.fused_moe import fused_moe_2stages, moe_sorting | |
| from aiter.jit.utils.chip_info import get_gfx_runtime | |
| from aiter.ops.flydsl.kimi_k3_moe_handoff import ( | |
| KimiK3MXFP4ExpertRequest, | |
| KimiK3MXFP4W13Layout, | |
| kimi_k3_mxfp4_expert_handoff, | |
| kimi_k3_mxfp4_expert_mode, | |
| supports_kimi_k3_mxfp4_expert_handoff, | |
| ) | |
| from aiter.ops.flydsl.moe_common import GateMode | |
| pytestmark = pytest.mark.skipif( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Re-cut of the Kimi-K3 B1 route/prepared-MoE work onto current
main, DCO-signed.This is the AITER half of the campaign's largest single accepted step (+20.3%
end-to-end, 50.635 → 60.909 tok/s/GPU at TP8/B1/8K-in/1K-out; route boundary 1.62x).
The original PR (#7) carried a nine-commit stack that mixed this
work with the MLA split override, the KDA decode/
f_bfusion, the latent MoE tail andthe B1 gate projection. Those four have since been re-cut and published separately
(ROCm#4405, ROCm#4495, ROCm#4496, and #6), and #7 was closed CONFLICTING —
leaving this part, the actual route/handoff change, with no live PR anywhere.
This branch is only the four commits that were never re-cut:
[Kimi-K3] Fuse gfx950 B1 routing and MoE sort metadatakimi_k3_moe_routeentry pointperf(moe): add Kimi-K3 B1 stage1 direct tileperf(moe): use atomic Kimi-K3 B1 stage2 reductionperf(flydsl): consume Kimi-K3 prepared MoE routingkimi_k3_moe_handoff— accept prepared routing from the caller+1212/-2 across 10 files, mostly new files under
aiter/ops/flydsl/.Test Plan
op_tests/flydsl_tests/suites ship with the change — route-sort, B1 stage1,B1 stage2, and the MoE handoff
8192-in / 1024-out, temperature 0,
--ignore-eos, with a paired GSM8K-100 accuracycheck against the parent build
Test Result
Measured at the time of acceptance, on 8x MI355X (gfx950), against its exact parent:
GSM8K 100q 5-shot temp 0: no accuracy regression against the parent.
These are the original campaign numbers, not a re-run of this branch. The change is
byte-identical to what was measured — the four commits are cherry-picked unmodified,
and the conflict-free rebase onto current
maintouched none of them — but the endpointresult itself has not been reproduced on today's
main.