[AMD] Perf Kimi-K3 low concurrency optimizations

[kkHuang-amd]

Summary

• Vendor selected Kimi-K3 gfx950 FlyDSL kernels in SGLang with a fail-closed source selector.
• Add MLA output-gate, KDA group64/B2, 12-head AITER MLA decode-side padding, and optional latent-tail integrations.
• Use one cooperative preactivated MoE producer for exact M2/M4 while larger batches retain the existing fallback.
• Add the Kimi-K3 M16384 GEMM tuning profile.
• Use the AITER revision already pinned by SGLang; no AITER update is required.

Runtime configuration

Base Kimi-K3 AITER configuration:

export SGLANG_USE_AITER=1
export SGLANG_AITER_K3_OPT=1
export AITER_FLYDSL_FORCE=1
export AITER_SITUV2_A8W4=1
export AITER_SITUV2_A4W4=0
export SGLANG_K3_FLYDSL_SOURCE=sglang

Enable the PR features:

# MLA output-gate fusion
export SGLANG_K3_AITER_MLA_GATE=1

# KDA input projection; B2_FUSIONS adds its exact-M2 specialization
export SGLANG_K3_AITER_KDA_GROUP64=1
export SGLANG_K3_AITER_B2_FUSIONS=1

# Unified exact-M2/M4 cooperative MoE preroute
export SGLANG_K3_AITER_MOE_PREROUTE_FP8=1
export SGLANG_K3_PREROUTE_PREACTIVATED_SHARED=1

# M=16384 GEMM tuning rows
export SGLANG_K3_AITER_M16384_PROFILE=1

The latent-tail profile is independent and remains off by default because it reduces token capacity. Enable it only when explicitly evaluating that trade-off:

export SGLANG_K3_AITER_LATENT_TAIL_FP8=1

The 12-head padding path has no feature flag. It is selected automatically for the covered Kimi-K3 shape when using AITER MLA decode, for example:

sglang serve \
  --model-path /path/to/Kimi-K3 \
  --tp-size 8 \
  --attention-backend aiter \
  --decode-attention-backend aiter

All integrations are shape/architecture guarded and fail closed to the existing implementation when the runtime contract is not covered.

Performance

• Unified M2 cooperative preroute: +1.08% throughput, -1.11% TPOT over the prior M2 design.
• Unified M4 cooperative preroute: +3.08% throughput, -3.33% TPOT over the matched M4 baseline.
• M16384 profile: approximately +0.7% throughput at C8/C16/C32 and -2.34% C32 TTFT.

Accuracy

• GSM8K 1319: 0.953
• Invalid responses: 0.001

Test plan

• Selected FlyDSL tests against pinned AITER d9e5ef7: 35 passed
• C2 8192/256 smoke: 8/8 requests succeeded
• C4 8192/256 smoke: 8/8 requests succeeded
• GSM8K 50: 1.000
• GSM8K 1319: 0.953
• Forbidden listed-PR path audit and git diff --check

The current container installs a newer FlyDSL than the old AITER pin expects for full-model MoE startup. The vendored focused kernels were tested directly against the exact pin; full-model smoke used the current compatible AITER checkout.

Made with Cursor

---

CI States

Latest PR Test (Base): ✅ Run #32579252160
Latest PR Test (Extra): ❌ Run #32579251698
Latest PR Test (AMD ROCm 7.2): ❌ Run #32579251895

[kkHuang-amd]

@amd-bot ci-status

[amd-bot]

@kkHuang-amd

CI Status for PR #35287

Merge verdict: ❌ Do not merge yet. The PR's core deliverable — the vendored Kimi-K3 gfx950 (MI355X) FlyDSL kernels — is not verified by PR CI: all 14 numerical/correctness tests are gated skipif get_gfx_runtime() != "gfx950" and the jit-kernel-unit stage ran only on mi300/gfx942, so every kernel-correctness assertion SKIPPED. Separately, PR CI is incomplete: PR Test (Base) fast-failed on an unrelated base-a-test-cpu failure, cascading skips across base-a/b/c GPU stages and all NVIDIA jit-kernel jobs; 2 AMD shards are still running. None of the executed failures are attributable to this PR's changed code.

Caution

This PR's gfx950 kernel code is NOT exercised by any PR-CI test. On the only AMD jit-kernel-unit job (mi300/gfx942, job) only the 21 structural/fail-closed guard tests ran; the 14 gfx950 numerical tests SKIPPED (test_group64_projection_matches_dequantized_reference_on_gfx950, test_kimi_k3_preroute_fp8_matches_dequantized_reference, test_kimi_k3_cooperative_preactivated_projection, test_latent_moe_tail_fp8_matches_dequantized_oracle_and_replays, test_kimi_k3_mla_gate_*, …). A green run does not verify kernel correctness. The env-gated integrations (SGLANG_K3_AITER_*, all default-off) and the 12-head MLA path are likewise unexercised. Author must run the jit-kernel-unit suite on a gfx950/MI355X runner (and a Kimi-K3 e2e smoke with the flags from the PR body) before merge.

Caution

PR CI is incomplete. PR Test (Base) run 32214735204 fast-failed on base-a-test-cpu (2), so base-a-test-1-gpu-small, base-b-*, base-c-* GPU stages and the NVIDIA jit-kernel-* jobs were skipped / not tested. 2 AMD mi300 shards (12, 13) are still running — AMD is not GREEN yet. The base-a-test-cpu root cause looks like a stale base (see below); rebasing on latest main and rerunning is advised.

Changed files (30): kernels under python/sglang/kernels/ops/kimi_k3/** (new), python/sglang/__init__.py (+36, HIP+env-gated), srt/layers/attention/aiter_backend.py (+16/-10, MLA decode head-pad), srt/models/.../forward_mla_rocm.py (+7/-1, NoPE rope skip), srt/models/kimi_k3.py (+331/-30), tests under test/registered/kernels/ops/kimi_k3/flydsl_ops/** (new).

Executed CI failure attribution: AMD: 4 failures (0 related) · Others: 5 failures (0 related) · plus fast-fail cascades (collapsed) and 2 AMD shards pending.

AMD Executed Failures

Job	Test File	Test Function	Error	Related?	Why
1gpu-small (0), mi300	test/registered/moe/test_torch_compile_moe.py	(module)	Triton HIP 900: op not permitted when stream is capturing → HIPBLAS_STATUS_INTERNAL_ERROR	🟢	MoE/torch.compile path; PR touches only MLA attention + env-gated K3 kernels. Classic AMD graph-capture infra error.
1gpu-small (8), mi300	test/registered/moe/test_fused_moe.py	(module)	AssertionError: Tensor-likes are not close! (also KeyError: /loky-…)	🟢	fused-MoE numerics; PR does not modify fused_moe. Not on any changed code path.
2gpu-large (1), mi300	test/registered/perf/test_bench_one_batch_2gpu.py	(module)	Triton HIP 900 stream capturing / HIPBLAS_STATUS_INTERNAL_ERROR / aiter_ipc DistStoreError wait timeout 600s	🟡	Full-model perf bench on AMD path; symptoms are HIP graph-capture/IPC infra. MLA changes are behavior-preserving for non-Kimi head counts, but confirm after rebase+rerun.
8gpu mi35x disagg (gfx950)	test/registered/amd/disaggregation/test_disaggregation_basic.py	(module)	ConnectionRefused 127.0.0.1:11200 / Internal Server Error	🟡	Server startup crash on gfx950. Env-gated K3 code stays off (flags unset), but this is the only gfx950 job — confirm after rebase+rerun.

Other Executed Failures

Job	Test File	Test Function	Error	Related?	Why
base-a-test-cpu (2)	test/registered/unit/mem_cache/test_hiradix_pp_sync_drain.py	test_pp_batches_write_and_load_counts_once	AttributeError: 'UnifiedRadixCache' object has no attribute 'buffer_pipeline'	🟢	PR touches no mem_cache code. Error at unified_radix_cache.py:2611 vs main's buffer_pipeline at line 242 ⇒ stale base / rebase mismatch, not this diff. This is the root cause that fast-failed the whole Base workflow.
base-c-test-acc-2-npu-a3	test/registered/npu/accuracy/qwen3_5_9b/test_npu_qwen3_5_9b_bf16_1p_gsm8k.py	(module)	exit 255 / container failure	🟢	NPU Qwen3.5 accuracy; different vendor backend. PR is HIP-only.
base-c-test-acc-4-npu-a3	N/A (NPU container)	N/A	Executing the custom container implementation failed	🟢	NPU runner/infra error, unrelated.
multimodal-gen-1-npu-a3	(NPU mm-gen)	N/A	404 … model_index.json for openmoss/MOVA-360p	🟢	ModelScope model-download 404 (infra), unrelated.
stage-a-test-1-gpu-xpu	test/registered/xpu/test_xpu_basic.py	(module)	RuntimeError: Failed to parse benchmark output (prefill_latency=None)	🟢	XPU backend; PR is HIP-only.

Collapsed cascades (not independent failures): jit-kernel-unit/benchmark/b200-test and base-a-test-1-gpu-small → fast-fail of base-a-test-cpu; base-c-test-perf-16-npu-a3 → fast-fail of base-c-test-acc-2/4-npu; wait-for-base-a, wait-for-stage-b-amd-rocm720, pr-test-finish, pr-test-npu-finish, finish are gate/rollup markers.

Details / what to do before merge

• Verify the kernels on real gfx950 (highest priority). Nothing in this run executed the FlyDSL numerical assertions. Run test/registered/kernels/ops/kimi_k3/flydsl_ops/* on an MI355X/gfx950 runner and run a Kimi-K3 e2e smoke with the PR-body flags (SGLANG_K3_AITER_*) so the env-gated integrations and the 12-head MLA decode path are actually exercised.
• Rebase on latest main and rerun. The base-a-test-cpu buffer_pipeline AttributeError indicates the branch base predates/misaligns with the current UnifiedRadixCache; a rebase should clear the Base-workflow fast-fail and let base-a/b/c and NVIDIA jit-kernel stages actually run (in-flight fix search for buffer_pipeline returned no open PR).
• Re-check the AMD 🟡 rows after rebase+rerun. test_bench_one_batch_2gpu and test_disaggregation_basic show HIP graph-capture / IPC / server-startup infra signatures; confirm they reproduce (or clear) once the branch is current. If a high-priority full-signal run is needed despite unrelated early failures, the bypass-fastfail label forces downstream stages (use sparingly — higher CI cost).
• The python/sglang/__init__.py addition is safe for non-AMD backends: it is guarded by torch.version.hip is not None + SGLANG_K3_AITER_M16384_PROFILE, so it cannot explain the CPU/NPU/XPU failures.

---

Generated by amd-bot using Claude Code CLI

[kkHuang-amd]

@amd-bot ci-status

[amd-bot]

@kkHuang-amd

CI Status for PR #35287

Merge verdict: 🔴 Do not merge. PR CI is incomplete (fast-fail cascades skipped downstream stages on Nvidia, AMD, and XPU) and there is one executed failure directly caused by this PR: on Nvidia, test_kimi_k3_bfa_overlap.py crashes in the PR's own kimi_k3.py:forward_qkvbfg_fused (AttributeError: '_kda_group64_weight'). Separately, the PR's core value — the gfx950 FlyDSL kernels — is behind default-off env vars and was not exercised by any PR-CI test, so green elsewhere proves nothing about it.

Caution

Two coverage problems, both matter more than the unrelated red X's below:

1. PR CI is incomplete. The Nvidia root failure fast-failed wait-for-base-b, skipping all base-c-* GPU jobs (H100/H200/B200/B300/GB300/H20). On AMD ROCm 7.2, the two stage-b failures fast-failed wait-for-stage-b, skipping every stage-c-* job — including the gfx950/mi35x stage-c jobs that are this PR's actual target hardware. XPU stage-b was skipped too. These skipped jobs are not tested.
2. The PR's changed code is largely not exercised. All new FlyDSL kernels (SGLANG_K3_AITER_MOE_PREROUTE_FP8, _KDA_GROUP64, _MLA_GATE, _LATENT_TAIL_FP8, _M16384_PROFILE, _B2_FUSIONS) are off by default and no PR test sets them. The 12-head AITER MLA padding path (aiter_backend.py) has no flag but is only selected for the Kimi-K3 gfx950 TP8 shape, which no PR-CI job ran. The new test/registered/kernels/ops/kimi_k3/flydsl_ops/* unit tests do not appear to have executed (the AMD sgl-kernel-unit-test suite was fast-fail skipped). Before merge, run the Kimi-K3 gfx950 suite with the env flags from the PR description.

Changed files (30): core = python/sglang/srt/models/kimi_k3.py (+331/-30), python/sglang/srt/layers/attention/aiter_backend.py (+16/-10, MLA-only), forward_mla_rocm.py (+7/-1, MLA-only), python/sglang/__init__.py (+36, ROCm-gated); rest = new env-gated kernels/ops/kimi_k3/** + new tests.

Executed CI failure attribution: AMD: 2 failures (0 related) · Others (Nvidia/NPU/XPU): 3 executed root failures (1 related). All other red jobs are fast-fail cascades collapsed into their roots below.

Other Executed Failures

Job	Test File	Test Function	Error	Related?	Why
base-b-test-1-gpu-large (4)	test/registered/unit/models/test_kimi_k3_bfa_overlap.py	test_capture_replay_matches_serial	AttributeError: 'types.SimpleNamespace' object has no attribute '_kda_group64_weight'	🔴	Crashes at kimi_k3.py:1989 in forward_qkvbfg_fused — a method this PR modified. New code unconditionally reads self._kda_group64_weight, but this test drives the method on a SimpleNamespace that never sets it. Root cause of the 5 other base-b fast-fail cascades.
base-c-test-perf-8-npu-a3	test/registered/npu/performance/minimax_m2_5/test_npu_minimax_m2_5_w8a8_4p_in64k_out1k_prefix90_50ms.py	(perf assertion)	AssertionError: 51.4 not less than or equal to 51.0	🟢	MiniMax-M2.5 NPU latency threshold (51.4ms vs 51.0ms budget) — different model, no Kimi-K3/MLA path. Perf-threshold miss. Root of perf-2/perf-4 cascades.
stage-a-test-1-gpu-xpu	test/registered/xpu/test_xpu_basic.py	(server startup)	OSError: libsycl.so.8: cannot open shared object file	🟢	XPU runtime library missing on the runner — infra/env, unrelated to the diff.

AMD Executed Failures

Job	Test File	Test Function	Error	Related?	Why
stage-b-test-2-gpu-large (rocm720)	test/registered/perf/test_bench_one_batch_2gpu.py	test_torch_compile_tp2_bs1	AssertionError: -1 not greater than 200	🟢	Model is Llama-3.1-8B (dense, no MLA). The PR's AMD changes (aiter_backend.py, forward_mla_rocm.py) are strictly inside the use_mla path, which Llama never enters. Throughput -1 = bench subprocess produced no output; looks like pre-existing ROCm 7.2 perf/infra flake.
stage-b-8gpu-mi35x-disaggregation (rocm720)	test/registered/amd/disaggregation/test_disaggregation_basic.py	(client requests)	ConnectionRefused 127.0.0.1:11200 / RuntimeError: Internal Server Error	🟢	Model is Qwen/Qwen3-8B (dense, no MLA/Kimi path). Decode server unreachable — pre-existing AMD disagg startup instability, no code path the PR touches.

Details / what to do before merge

• 🔴 Fix the real bug first. In forward_qkvbfg_fused, self._kda_group64_weight is read even when the attribute was never initialized (the diff adds self._kda_group64_weight = None in __init__ but this code path is reachable on objects that skip that init, e.g. the overlap test's SimpleNamespace). Guard with getattr(self, "_kda_group64_weight", None), or update test_kimi_k3_bfa_overlap.py to set the attribute. This is what fast-failed all of Nvidia base-b/base-c.
• Actually exercise the PR. Re-run with the PR-description env set (SGLANG_K3_AITER_MOE_PREROUTE_FP8, _KDA_GROUP64, _MLA_GATE, _M16384_PROFILE, _B2_FUSIONS, SGLANG_K3_FLYDSL_SOURCE=sglang) on gfx950, plus a Kimi-K3 TP8 run to hit the 12-head MLA padding path. Confirm the new flydsl_ops/test_* unit tests actually run.
• Get full vendor signal. After the fix, ensure Nvidia base-c, AMD stage-c (esp. the mi35x/gfx950 jobs), and XPU stage-b complete rather than being fast-fail skipped. For a full-signal run despite the unrelated XPU/NPU/AMD-dense flakes, the bypass-fastfail label + branch update is an option (uses more CI resources — use sparingly).
• 🟢 The Llama/Qwen3/MiniMax/XPU failures need no action from this PR — none touch MLA or Kimi-K3 code; treat as pre-existing/infra and confirm they reproduce on main.

---

Generated by amd-bot using Claude Code CLI

[kkHuang-amd]

@amd-bot ci-status

[amd-bot]

@kkHuang-amd

CI Status for PR #35287

Merge verdict: ❌ Do not merge on "green." PR CI is incomplete and, more importantly, this PR's changed code was never exercised. All 5 new Kimi-K3 flydsl kernel tests register to the AMD MI35x (gfx950) suite stage-b-test-1-gpu-small-amd-mi35x, and that stage was skipped because wait-for-stage-a-amd-rocm720 timed out. No executed failure is attributable to this PR — but that fact is meaningless here because the code paths this PR adds/changes did not run.

Caution

This PR's changed code is not verified by PR CI. The 5 new tests (test_kimi_k3_kda_input_group64, test_kimi_k3_mla_gate_epilogue, test_kimi_k3_moe_preroute_fp8, test_latent_moe_tail_fp8, test_aiter_feature_gates) all target suite stage-b-test-1-gpu-small-amd-mi35x and @skipif get_gfx_runtime() != "gfx950". On this run that suite (stage-b-test-1-gpu-small-amd-mi35x-rocm720) was skipped, and the benchmark (jit-kernel-benchmark-test-amd-rocm720) is still in-progress. The modified production code (kimi_k3.py +331/-30, aiter_backend.py, forward_mla_rocm.py) is only exercised by AMD stage-b/stage-c, which were all skipped. Re-run AMD ROCm 7.2 CI to completion (or on a gfx950/MI35x runner) before merging — a human must confirm the MI35x stage-b suite actually passes.

Changed files: 31 files (+5605/-41) — new python/sglang/kernels/ops/kimi_k3/flydsl/** gfx950 kernels + tests; srt/models/kimi_k3.py (+331/-30), srt/layers/attention/aiter_backend.py (+16/-10), srt/models/deepseek_common/.../forward_mla_rocm.py (+7/-1).

Executed CI failure attribution: AMD: 0 real test failures (stage-b/c skipped, benchmark pending) · NVIDIA: 1 executed failure (0 related) · NPU: 1 executed failure (0 related). Many "failures" are fast-fail cascade / gate rollups, not independent failures.

AMD CI — incomplete, not failed

Job	Test File	Test Function	Error	Related?	Why
wait-for-stage-a-amd-rocm720	N/A	N/A	Timeout waiting for stage-a-amd-rocm720 jobs	🟢	Infra/timeout — jit-kernel-benchmark-test-amd-rocm720 still in-progress; not a test failure. Consequence: skips all stage-b/c.
stage-b-test-1-gpu-small-amd-mi35x-rocm720	(this PR's 5 new tests)	—	SKIPPED (not run)	—	Coverage gap — this is where the PR's kernels would be tested.
call-gate / pr-gate · pr-test-amd-extra-finish	N/A	N/A	gate/finish rollup	🟢	Aggregation jobs reflecting the skipped extra-a suites, not independent failures.

Note: stage-a AMD did pass — jit-kernel-unit-test-amd-rocm720 ✅ and stage-a-test-1-gpu-small-amd-rocm720 ✅ (mi300/gfx942 — does not exercise the gfx950 kernels).

NVIDIA / Other Executed Failures

Job	Test File	Test Function	Error	Related?	Why
call-jit-kernel-tests / jit-kernel-unit-test (step 9)	test/registered/profiling/test_diffusion_stage_profiler_sync.py	test_stage_entry_sync_excludes_previous_stage_tail	AssertionError: producer tail leaked into consumer … assert 155.8 < 100	🟢	Flaky GPU timing-threshold test; unrelated to Kimi-K3/AMD. This single failure fast-fail-cascaded all NVIDIA base-b-* jobs (14 "failure" + 12 "cancelled" all cite root cause jit-kernel-unit-test / wait-for-base-b).
multimodal-gen-test-1-npu-a3	N/A	N/A	log unavailable (BlobNotFound); NPU run still queued	🟢	NPU multimodal path, unrelated to this PR's ROCm changes.

The ~26 NVIDIA base-b-* "failure"/"cancelled" jobs + wait-for-base-b + pr-test-finish are one root cause (the diffusion flake above), collapsed here rather than listed individually.

Details / what to do before merge

• Close the coverage gap (blocking). Re-run PR Test ROCm 7.2 (AMD) so wait-for-stage-a-amd-rocm720 doesn't time out (it was waiting on jit-kernel-benchmark-test-amd-rocm720, still in-progress). Confirm stage-b-test-1-gpu-small-amd-mi35x-rocm720 actually runs and passes — that suite is the only thing exercising the new gfx950 kernels and the @skipif gfx950 tests. If no gfx950/MI35x runner is available in PR CI, the new kernels are effectively untested and a maintainer must run the suite manually on an MI35x box.
• NVIDIA jit flake: re-run call-jit-kernel-tests — the diffusion profiler timing assertion (consumer_ms 155 < 100) is a threshold flake unrelated to this PR; a green rerun should clear the whole base-b cascade.
• NPU: re-run once the NPU queue drains; log is currently unavailable and the run is still queued.
• If full signal is needed urgently, consider the bypass-fastfail label + branch update so the NVIDIA cascade doesn't mask the AMD result (use sparingly — higher CI cost).

---

Generated by amd-bot using Claude Code CLI

[kkHuang-amd]

@amd-bot ci-status

[amd-bot]

@kkHuang-amd

CI Status for PR #35287

Merge verdict: Not safe to merge on "green" yet. NVIDIA is effectively green and the new Kimi-K3 gfx950 kernel unit tests ran and passed on gfx950, but PR CI is incomplete — ROCm 7.2 stage-c and most of the NPU pipeline were fast-fail skipped, so they are untested. None of the executed failures appear caused by this PR (they sit in code this PR does not touch, or are infra/flake), but this PR's runtime features are behind default-off env vars that no PR test sets, so the fused serving paths are not verified end-to-end.

Caution

Coverage gap — the PR's runtime feature paths are not exercised, and downstream CI was fast-fail skipped.

• New kernels have unit coverage: the K3 tests register into stage-b-test-1-gpu-small-amd-mi35x, which ran on gfx950 as stage-b-test-1-gpu-small-amd-mi35x-rocm720 (linux-mi35x-gpu-1) and passed.
• But every runtime integration in kimi_k3.py is gated behind default-off env vars (SGLANG_K3_AITER_MLA_GATE, ..._KDA_GROUP64, ..._B2_FUSIONS, ..._MOE_PREROUTE_FP8, ..._M16384_PROFILE, etc.). No PR-CI test sets them ⇒ the fused e2e paths are green-but-unverified. Author must run a Kimi-K3 gfx950 serving job with those flags on before merge.
• PR Test ROCm 7.2 (AMD): wait-for-stage-b failed ⇒ 11 stage-c jobs skipped (incl. stage-c-*-mi35x-rocm720, which is where Kimi-K3 e2e on gfx950 would run). PR Test (NPU): base-c-test-perf-8-npu-a3 fast-failed ⇒ perf-2/4, acc-16, base-b-16, multimodal skipped. These are not tested — rerun (or bypass-fastfail for a full-signal run) before trusting the pipeline.

Changed files: 31 files, +5605/-41 — mostly new gfx950 FlyDSL kernels + unit tests under python/sglang/kernels/ops/kimi_k3/**; edits to srt/layers/attention/aiter_backend.py (+16/-10), srt/models/deepseek_common/.../forward_mla_rocm.py (+7/-1), srt/models/kimi_k3.py (+331/-30), sglang/__init__.py (+36, HIP-guarded).

Executed CI failure attribution: AMD: 2 executed test failures (0 related) + 1 fast-fail gate · Others: 3 executed failures across NPU/XPU/Xeon (0 related). NVIDIA PR Test: all base-a/b/c jobs passed; run shows "cancelled" only because B300 jobs are disabled (aa3f7667) and pr-test-finish cascaded.

AMD Executed Failures

Job	Test File	Test Function	Error	Related?	Why
stage-b-test-1-gpu-small-amd-rocm720 (mi300, 0)	test/registered/moe/test_torch_compile_moe.py	(torch.compile MoE run)	ValueError: Incorrect number of arguments passed to kernel: … expected [… 'BLOCK_SIZE', 'USE_PDL'] → HIP graph capture abort → server exit -9	🟢 unlikely	Failing kernel _fused_gate_sigmoid_mul_add_kernel lives in kernels/ops/elementwise/elementwise.py (a constexpr USE_PDL default not passed under torch.compile on ROCm) — not touched by this PR; generic MoE test, not Kimi-K3.
stage-b-test-large-8-gpu-mi35x-disaggregation-amd-rocm720	test/registered/amd/disaggregation/test_disaggregation_basic.py	(bench)	ConnectionRefusedError 127.0.0.1:11200 / RuntimeError: Internal Server Error mid-benchmark	🟡 possibly	Runs on gfx950 (PR's target HW) but both prefill+decode servers started and captured CUDA graphs successfully, failing only mid-bench; K3 flags are off, model is not Kimi-K3. Looks like a disagg infra/flake — rerun to confirm.
wait-for-stage-b-amd-rocm720	N/A	N/A	fast-fail gate	—	Cascade of the two rows above; caused 11 stage-c jobs to skip (see caution).

Other Executed Failures

Job	Test File	Test Function	Error	Related?	Why
base-c-test-perf-8-npu-a3	test/registered/npu/performance/minimax_m2_5/test_npu_minimax_m2_5_w8a8_4p_in64k_out1k_prefix90_50ms.py	assert_metrics	AssertionError: 51.9 not less than or equal to 51.0 (perf threshold)	🟢 unlikely	NPU MiniMax-M2.5 perf latency 0.9ms over threshold; PR touches no NPU or MiniMax code. Root cause of the NPU pipeline; perf-2/4, acc-16, base-b-16, multimodal all fast-fail cascade from it.
stage-b-test-1-gpu-xpu	test/registered/attention/test_chunk_gated_delta_rule.py	(chunk gated delta rule)	AssertionError: … padded rows wrote into the state pool (+ torchcodec/libavutil OSErrors)	🟢 unlikely	XPU backend; generic GDN kernel not modified by this PR (PR's KDA changes are gfx950-only and env-gated).
build-test (xeon-gnr, base-b-test-cpu)	test/registered/cpu/test_subblock_sparse_attention.py	(import)	ModuleNotFoundError: No module named 'imageio' + HF 401 GatedRepoError (Llama-3.1-8B)	🟢 unlikely	Missing dep in CPU image + HuggingFace auth — pure infra/env, unrelated to the diff.

Details / what to do before merge

• Verify the env-gated feature paths (highest priority). Green PR CI does not exercise the MLA-gate / KDA / MoE-preroute / M16384 fusions because their env vars default off. Run a Kimi-K3 gfx950 serving + accuracy job with the flag set from the PR description (SGLANG_K3_AITER_*=1) and confirm correctness vs the fallback path.
• Rerun PR Test ROCm 7.2 (AMD) so stage-c (incl. stage-c-*-mi35x-rocm720) actually runs — that is the closest thing to Kimi-K3 e2e coverage on gfx950, and it was skipped by the stage-b fast-fail.
• Confirm the two AMD stage-b failures are pre-existing/flake: re-run the ROCm720 workflow. test_torch_compile_moe.py (USE_PDL) is a torch.compile+triton issue in unmodified elementwise.py; test_disaggregation_basic.py is a mid-bench connection drop after healthy server startup. Neither points at this diff, but a clean rerun removes doubt.
• NPU/XPU/Xeon failures need no action from this PR — MiniMax perf threshold, XPU GDN state-pool assert, and CPU missing-imageio/HF-401 are all unrelated to AMD Kimi-K3 changes.

---

Generated by amd-bot using Claude Code CLI

[kkHuang-amd]

@amd-bot ci-status

[amd-bot]

@kkHuang-amd

CI Status for PR #35287

Merge verdict: 🔴 Do not merge on green — this PR's changed code was not exercised by PR CI. All six new Kimi-K3 kernel tests register to the stage-b-test-1-gpu-small-amd-mi35x (gfx950/MI355x) suite, and that stage was skipped because wait-for-stage-a-amd-rocm720 timed out at 240 min (runner-queue backup) even though the stage-a jobs themselves passed. No executed failure anywhere is caused by this PR — every red job is a pre-existing NPU/MLX/Xeon issue or an expected label/infra gate — but a passing stage-a proves nothing about the 5,605 lines of new gfx950 kernels.

Caution

This PR's changed code is not verified by PR CI. The AMD gate (pr-test-amd-rocm720.yml) reached stage-a (passed), but wait-for-stage-a-amd-rocm720 timed out (infra, not the PR), so all stage-b/stage-c jobs — including stage-b-test-1-gpu-small-amd-mi35x-rocm720, the only job that runs this PR's new tests — were skipped. AMD Extra + Extra were gated off (missing run-ci-extra label). The author must get a clean MI355x stage-b run before merge. Re-run/update the branch (or add bypass-fastfail sparingly to dispatch stages in parallel and avoid the stage-a wait timeout), and confirm the six new test/registered/kernels/ops/kimi_k3/** tests actually execute on gfx950.

Changed files: 31 files, +5,605/-41 — new python/sglang/kernels/ops/kimi_k3/** FlyDSL kernels + tests; models/kimi_k3.py (+331/-30); layers/attention/aiter_backend.py; deepseek_common/.../forward_mla_rocm.py; python/sglang/__init__.py (+36, HIP-only, env-gated).

Executed CI failure attribution: AMD: 0 executed test failures (2 non-test infra/gate jobs, see caution) · Others: 6 executed failures (0 related to this PR)

Other Executed Failures

Job	Test File	Test Function	Error	Related?	Why
base-c-test-perf-8-npu-a3	test/registered/npu/performance/minimax_m2_5/test_npu_minimax_m2_5_w8a8_4p_in64k_out1k_prefix90_50ms.py	throughput run	Server never came up — ConnectionRefused 127.0.0.1:20166, TypeError: float() … not 'NoneType' (log L13137/13190)	🟢	NPU/Ascend MiniMax perf test; PR touches only AMD gfx950 code
base-c-test-acc-2-npu-a3	test/registered/npu/accuracy/qwen3_5_9b/test_npu_qwen3_5_9b_bf16_1p_gsm8k.py	TestNPUQwen3_5_9B_GSM8K	AssertionError: 0.82 not >= 0.82665 (log L21513)	🟢	NPU Qwen3.5-9B GSM8K accuracy dip; no AMD code path
multimodal-gen-test-1-npu-a3	sglang/multimodal_gen/test/server/ascend/test_server_1_npu.py	test_diffusion_generation[ernie/flux/z_image…]	Diffusion testcase check failures; FAILED [code=127] cuda_0.o (log L13903+)	🟢	NPU diffusion server; health-check itself filtered this as non-root-cause
base-b-test-16-npu-a3	N/A	N/A	Fast-fail skip; root cause = base-c-test-perf-8-npu-a3 (log L285)	🟢	Cascade of the NPU perf failure above
stage-a-unit-test-mlx	test/registered/unit/utils/test_diffusion_torch_fallback.py	test_norm_infer_matches_reference (fp16)	AssertionError: Tensor-likes are not equal — 1/128 elem, abs diff 6.1e-5 (log L1155)	🟢	fp16 tolerance flake in diffusion norm fallback; unrelated to Kimi-K3
build-test (xeon-gnr, base-b-test-cpu)	test/registered/cpu/test_subblock_sparse_attention.py	(import)	ModuleNotFoundError: No module named 'imageio' + HF 401 GatedRepoError for Llama-3.1-8B (log L4897/4317)	🟢	Xeon CPU env/dependency + HF auth issue; no AMD code path

(NPU base-c-test-perf-4/-16 also show as failed check-runs but are the same MiniMax perf cluster / fast-fail cascade as perf-8; collapsed into the perf-8 root cause above.)

Details / what to do before merge

• Coverage (the blocker). The six new tests — test_kimi_k3_moe_preroute_fp8.py, test_kimi_k3_kda_input_group64.py, test_kimi_k3_mla_gate_epilogue.py, test_latent_moe_tail_fp8.py, bench_kimi_k3_moe_preroute_fp8.py, test_aiter_feature_gates.py — all call register_amd_ci(suite="stage-b-test-1-gpu-small-amd-mi35x") and are guarded by get_gfx_runtime() == "gfx950". They only run in the MI355x stage-b job, which was skipped. Get that job to run (green stage-a should let stage-b dispatch on rerun; or use bypass-fastfail to skip the stage wait) and confirm the tests execute rather than skip on gfx950.
• Feature flags. Most of the PR's runtime paths are behind default-off env vars (SGLANG_K3_AITER_MLA_GATE, _KDA_GROUP64, _MOE_PREROUTE_FP8, _M16384_PROFILE, etc.). Even once stage-b runs, verify the covering tests actually set these (the gate test does) so the new code is reachable, not just imported.
• Shared __init__.py is safe for other backends. The added block is guarded by torch.version.hip is not None and SGLANG_K3_AITER_M16384_PROFILE, so it is a no-op on NPU/MLX/Xeon/MUSA/XPU — it does not explain any of the non-AMD failures above (confirmed those reached model load / server launch).
• Non-AMD reds need no action from this PR, but the NPU Qwen3.5-9B accuracy dip (0.82 vs 0.827 baseline) and NPU MiniMax server-startup failure are worth flagging to the NPU owners separately.

---

Generated by amd-bot using Claude Code CLI