Skip to content

perf(flydsl): Kimi-K3 B1 route-sort + prepared-MoE handoff - #29

Open
JohnQinAMD wants to merge 4 commits into
mainfrom
perf/kimi-k3-route-prepared-moe-clean
Open

JohnQinAMD wants to merge 4 commits into
mainfrom
perf/kimi-k3-route-prepared-moe-clean

Conversation

@JohnQinAMD

Copy link
Copy Markdown
Owner

Purpose

Re-cut of the Kimi-K3 B1 route/prepared-MoE work onto current main, DCO-signed.

This is the AITER half of the campaign's largest single accepted step (+20.3%
end-to-end, 50.635 → 60.909 tok/s/GPU at TP8/B1/8K-in/1K-out; route boundary 1.62x).
The original PR (#7) carried a nine-commit stack that mixed this
work with the MLA split override, the KDA decode/f_b fusion, the latent MoE tail and
the B1 gate projection. Those four have since been re-cut and published separately
(ROCm#4405, ROCm#4495, ROCm#4496, and #6), and #7 was closed CONFLICTING —
leaving this part, the actual route/handoff change, with no live PR anywhere.

This branch is only the four commits that were never re-cut:

commit change
[Kimi-K3] Fuse gfx950 B1 routing and MoE sort metadata fused B1 route-sort kernel + kimi_k3_moe_route entry point
perf(moe): add Kimi-K3 B1 stage1 direct tile direct-tile stage1 selection + tuned config row
perf(moe): use atomic Kimi-K3 B1 stage2 reduction atomic stage2 reduction + tuned config row
perf(flydsl): consume Kimi-K3 prepared MoE routing kimi_k3_moe_handoff — accept prepared routing from the caller

+1212/-2 across 10 files, mostly new files under aiter/ops/flydsl/.

Test Plan

  • four op_tests/flydsl_tests/ suites ship with the change — route-sort, B1 stage1,
    B1 stage2, and the MoE handoff
  • the campaign gate this was accepted under: endpoint-measured TP8 / batch 1 /
    8192-in / 1024-out, temperature 0, --ignore-eos, with a paired GSM8K-100 accuracy
    check against the parent build

Test Result

Measured at the time of acceptance, on 8x MI355X (gfx950), against its exact parent:

parent with this change
decode 50.635 tok/s/GPU 60.909 tok/s/GPU (+20.3%)
TPOT 19.718 ms 16.418 ms
route boundary 1.62x

GSM8K 100q 5-shot temp 0: no accuracy regression against the parent.

These are the original campaign numbers, not a re-run of this branch. The change is
byte-identical to what was measured — the four commits are cherry-picked unmodified,
and the conflict-free rebase onto current main touched none of them — but the endpoint
result itself has not been reproduced on today's main.

Add a narrow FlyDSL specialization for the 896-expert, top-16, single-group decode shape. Preserve AITER tie ordering and emit Opus-compatible metadata directly from the routing launch.

Assisted-by: Codex
Signed-off-by: Yanyuan Qin <yanyuan.qin@amd.com>
Register one validated BF16 x MXFP4 16-row FlyDSL variant and advance it with the producer 32-row sort-bucket stride. Select it only through the existing exact gfx950 Kimi-K3 B1 tuning row.

The complete expert boundary improves from 20.70 us to 18.20 us across two independent processes, with 90/90 adversarial correctness cases passing.

Signed-off-by: Yanyuan Qin <yanyuan.qin@amd.com>
Select the existing direct atomic weighted-reduction kernel through the exact gfx950 Kimi-K3 B1 tuning row. Larger batch rows and generic dispatch remain unchanged.

The complete expert boundary improves from 20.70 us to 18.62 us across two independent processes, with 90/90 adversarial correctness cases passing.

Signed-off-by: Yanyuan Qin <yanyuan.qin@amd.com>
Add a typed, fail-closed gfx950 B1 handoff that owns Kimi-K3 routing and feeds its prepared metadata directly into the existing MXFP4 expert stages. Support both production A8W4/interleaved and A16W4/separated layouts while preserving fallback outside the exact shape contract.

The live A8W4 route-to-expert boundary improves from 41.83-42.01 us to 25.40-25.41 us (1.647-1.653x), with exact route IDs, 6.66e-8 routing-weight RRMSE, zero output RRMSE, and no duplicate route/sort launch.

Assisted-by: Codex

Signed-off-by: Yanyuan Qin <yanyuan.qin@amd.com>
@github-actions

Copy link
Copy Markdown

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 29 --add-label <label>

num_valid_ids: fx.Pointer,
moe_buf: fx.Pointer,
moe_buf_i32_elements: fx.Int32,
stream: fx.Stream = fx.Stream(None),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [ruff] <B008> reported by reviewdog 🐶
Do not perform function call fx.Stream in argument defaults; instead, perform the call within the function, or read the default from a module-level singleton variable



def _register_production_variants_stage1(
kernels: Dict[str, Dict], a_dtype: str, b_dtype: str, out_dtype: str

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [ruff] <F821> reported by reviewdog 🐶
Undefined name Dict



def _register_production_variants_stage1(
kernels: Dict[str, Dict], a_dtype: str, b_dtype: str, out_dtype: str

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [ruff] <F821> reported by reviewdog 🐶
Undefined name Dict

Comment on lines +4 to +23
import importlib
import math
from dataclasses import replace

import aiter
import pytest
import torch
from aiter import dtypes
from aiter.fused_moe import fused_moe_2stages, moe_sorting
from aiter.jit.utils.chip_info import get_gfx_runtime
from aiter.ops.flydsl.kimi_k3_moe_handoff import (
KimiK3MXFP4ExpertRequest,
KimiK3MXFP4W13Layout,
kimi_k3_mxfp4_expert_handoff,
kimi_k3_mxfp4_expert_mode,
supports_kimi_k3_mxfp4_expert_handoff,
)
from aiter.ops.flydsl.moe_common import GateMode

pytestmark = pytest.mark.skipif(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [ruff] <I001> reported by reviewdog 🐶
Import block is un-sorted or un-formatted

Suggested change
import importlib
import math
from dataclasses import replace
import aiter
import pytest
import torch
from aiter import dtypes
from aiter.fused_moe import fused_moe_2stages, moe_sorting
from aiter.jit.utils.chip_info import get_gfx_runtime
from aiter.ops.flydsl.kimi_k3_moe_handoff import (
KimiK3MXFP4ExpertRequest,
KimiK3MXFP4W13Layout,
kimi_k3_mxfp4_expert_handoff,
kimi_k3_mxfp4_expert_mode,
supports_kimi_k3_mxfp4_expert_handoff,
)
from aiter.ops.flydsl.moe_common import GateMode
pytestmark = pytest.mark.skipif(
import importlib
import math
from dataclasses import replace
import pytest
import torch
import aiter
from aiter import dtypes
from aiter.fused_moe import fused_moe_2stages, moe_sorting
from aiter.jit.utils.chip_info import get_gfx_runtime
from aiter.ops.flydsl.kimi_k3_moe_handoff import (
KimiK3MXFP4ExpertRequest,
KimiK3MXFP4W13Layout,
kimi_k3_mxfp4_expert_handoff,
kimi_k3_mxfp4_expert_mode,
supports_kimi_k3_mxfp4_expert_handoff,
)
from aiter.ops.flydsl.moe_common import GateMode
pytestmark = pytest.mark.skipif(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant