Skip to content

perf(flydsl): add Kimi-K3 FP8 latent MoE tail - #18

Open
JohnQinAMD wants to merge 1 commit into
perf/kimi-k3-latent-tail-cleanfrom
perf/kimi-k3-latent-tail-fp8-clean
Open

JohnQinAMD wants to merge 1 commit into
perf/kimi-k3-latent-tail-cleanfrom
perf/kimi-k3-latent-tail-fp8-clean

Conversation

@JohnQinAMD

@JohnQinAMD JohnQinAMD commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Summary

Add an incremental rowwise OCP-E4M3 prepack and gfx950 batch-one persistent
kernel for the Kimi-K3 latent-MoE tail. The kernel fuses RMSNorm, FP8-weight
GEMV, BF16 materialization, and the shared-expert add.

This PR is stacked on #17. It keeps FP32 RMS/dot-product
accumulation, FP32 per-row scales, and BF16 activation/output, and fails closed
outside the exact contiguous (1, 3584) -> (1, 7168) gfx950 contract.

Correctness

The real-weight rotating-bank campaign checked 24 cases:

  • worst RRMSE versus BF16: 0.020959;
  • minimum cosine: 0.999780;
  • minimum SNR: 33.573 dB;
  • worst RRMSE versus dequantized FP8: 1.487e-5.

Tests cover support/quantizer/schedule rejection, both numerical oracles, graph
replay, output reuse, and changed input.

Performance

MI355X/gfx950, 92 rotating real weights, 11 trials, 30 graph replays/trial:

Path ms/token
BF16 parent 0.920660
FP8 candidate 0.668757
Saving 0.251903

Speedup is 1.376673x. The component is below the 0.300 ms/token standalone
endpoint floor, so it receives no isolated endpoint credit. In the Wave40
bundle, TPOT improved 12.194182 -> 11.886058 ms and throughput 82.006322 ->
84.132180 tok/s/GPU; GSM8K first 100 was 100/100 with zero invalid and zero
paired changes.

Validation

  • Ruff check/format pass all three changed files.
  • Exact clean current-main GPU suite: 7 passed in the immutable MI355X
    validation image.
  • git diff --check passes.
  • The commit carries DCO and AI-assistance trailers.

Disclosure

Developed with assistance from OpenAI Codex. The author reviewed the precision
contract, tests, schedule, and performance evidence.

Add the incremental gfx950 FP8-weight latent-tail variant while preserving the BF16 parent and exact fail-closed dispatch contract.

Assisted-by: OpenAI Codex
Signed-off-by: Yanyuan Qin <yanyuan.qin@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant