Skip to content

[Kimi-K3] Add AttnRes kernels - #50090

Merged
vllm-bot merged 2 commits into
vllm-project:mainfrom
gau-nernst:k3-attnres
Jul 28, 2026
Merged

vllm-bot merged 2 commits into
vllm-project:mainfrom
gau-nernst:k3-attnres

Conversation

@gau-nernst

@gau-nernst gau-nernst commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Purpose

#50000. Add AttnRes kernels

  • Specialized CUDA C++ kernel for NVIDIA sm100
  • Triton for NVIDIA fallback and AMD

This PR also adds Buildkite CI for K3

Test Plan

# NVIDIA
pytest -v tests/models/kimi_k3/test_attn_res.py

# AMD
pytest -v tests/models/kimi_k3/test_amd_attn_res.py

Test Result


Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Co-authored-by: OpenAI Codex <codex@openai.com>

Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Co-authored-by: OpenAI Codex <codex@openai.com>

Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
@zyongye zyongye added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 28, 2026
@vllm-bot
vllm-bot merged commit 61ac368 into vllm-project:main Jul 28, 2026
98 of 172 checks passed
@gau-nernst
gau-nernst deleted the k3-attnres branch July 28, 2026 07:23
@ZJY0516 ZJY0516 mentioned this pull request Jul 28, 2026
4 tasks
@namgyu-youn

Copy link
Copy Markdown
Contributor

Hi @gau-nernst, question on the SM100 path here.

The kernel indexes layer_res/delta/output with a hard-coded row stride of H, threading real strides only for blocks.

Nothing enforces that: the dispatch asserts stride(-1) == 1 but not stride(0) == hidden_size, and there's no STD_TORCH_CHECK either. On a B200 build, feeding a row-padded prefix (stride(0)=7175, H=7168) gives max|out-ref| = 6.66 against the suite's 8e-2 — no fault, no NaN, and the in-place prefix update is wrong too.

Is packed rows an intended precondition? If the K3 model never passes padded slices, I'll send a STD_TORCH_CHECK so it fails loudly. If they're reachable, the dispatch should decline them and fall back to Triton, which already threads the strides correctly. Happy to do either and hear your thoughts~

@gau-nernst

Copy link
Copy Markdown
Contributor Author

@namgyu-youn Good catch. I think just add STD_TORCH_CHECK is enough, we don't expect the inputs to be strided.

Right now the team is trying to get #50000 merged, so you can open a PR after that is merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build k3 kimi ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants