Support kimi k3 nvfp4 checkpoint by wzhao18 · Pull Request #53132 · vllm-project/vllm

wzhao18 · 2026-08-20T16:17:23Z

Purpose

This PR supports running kimi k3 nvfp4 checkpoint nvidia/Kimi-K3-NVFP4

Test Plan

Kimi K3 GSM8k on 8 x B300

Test Result

TP8:

vllm serve nvidia/Kimi-K3-NVFP4 \
  -tp 8 \
  --load-format fastsafetensors \
  --no-enable-flashinfer-autotune \
  --trust-remote-code \
  --language-model-only \
  --attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}' \
  --kv-cache-dtype fp8 \
  --speculative-config '{"model":"Inferact/Kimi-K3-DSpark","method":"dspark","num_speculative_tokens":4,"rejection_sample_method":"standard"}' \
  --enable-prompt-tokens-details \
  --prefix-match-unit 128

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9591|±  |0.0055|
|     |       |strict-match    |     5|exact_match|↑  |0.9591|±  |0.0055|

TEP8:

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9606|±  |0.0054|
|     |       |strict-match    |     5|exact_match|↑  |0.9598|±  |0.0054|

DCP8:

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.9575|±  |0.0056|
|     |       |strict-match    |     5|exact_match|↑  |0.9575|±  |0.0056|

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>

zyongye · 2026-08-21T04:07:19Z

/ci run

github-actions · 2026-08-21T04:07:37Z

✅ Triggered Buildkite CI #84955 for commit dd72b3df39b2.

DamirLisak · 2026-08-21T12:29:11Z

@wzhao18 support for deepseek v4 nvfp4 checkpoint #52447? 🙏☺️

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Wyett <wyettzeng@gmail.com>

…trim FormatScheme A FP8_PB_WO output width that is not a multiple of 128 (a partial trailing block; e.g. GLM's replicated fused_qkv_a_proj = 2048 + 576 = 2624) is padded up to a block boundary with zeros before the kernel post-load, the GEMM runs on the padded weight, and the output is trimmed back to the logical width with bias added after. This is wei-zhao vllm-project#53132's approach, expressed as a FormatScheme (_Fp8PbWoPartialBlock) on the generic method rather than a per-format class -- post_process pads the weight, apply wraps the kernel to trim the output. A new FormatScheme.apply hook makes this compute-time residue expressible without touching ModelOptLinearMethod. KFp8Block128 sizes the weight_scale by cdiv to match the (padded) block count. No-op for the common block-aligned case. Chosen over a no-pad variant because it handles any width and does not depend on DeepGEMM's undocumented N-alignment tolerance -- it is the logic upstream already merged. Test is a faithful port of vllm-project#53132's test_modelopt_fp8_pb_wo_hides_output_padding for the generic method. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Zhu, Zufang <zufang.zhu@intel.com>

…trim FormatScheme A FP8_PB_WO output width that is not a multiple of 128 (a partial trailing block; e.g. GLM's replicated fused_qkv_a_proj = 2048 + 576 = 2624) is padded up to a block boundary with zeros before the kernel post-load, the GEMM runs on the padded weight, and the output is trimmed back to the logical width with bias added after. This is wei-zhao vllm-project#53132's approach, expressed as a FormatScheme (_Fp8PbWoPartialBlock) on the generic method rather than a per-format class -- post_process pads the weight, apply wraps the kernel to trim the output. A new FormatScheme.apply hook makes this compute-time residue expressible without touching ModelOptLinearMethod. KFp8Block128 sizes the weight_scale by cdiv to match the (padded) block count. No-op for the common block-aligned case. Chosen over a no-pad variant because it handles any width and does not depend on DeepGEMM's undocumented N-alignment tolerance -- it is the logic upstream already merged. Test is a faithful port of vllm-project#53132's test_modelopt_fp8_pb_wo_hides_output_padding for the generic method. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: khushali9 <khushali.desai9@gmail.com>

…trim FormatScheme A FP8_PB_WO output width that is not a multiple of 128 (a partial trailing block; e.g. GLM's replicated fused_qkv_a_proj = 2048 + 576 = 2624) is padded up to a block boundary with zeros before the kernel post-load, the GEMM runs on the padded weight, and the output is trimmed back to the logical width with bias added after. This is wei-zhao vllm-project#53132's approach, expressed as a FormatScheme (_Fp8PbWoPartialBlock) on the generic method rather than a per-format class -- post_process pads the weight, apply wraps the kernel to trim the output. A new FormatScheme.apply hook makes this compute-time residue expressible without touching ModelOptLinearMethod. KFp8Block128 sizes the weight_scale by cdiv to match the (padded) block count. No-op for the common block-aligned case. Chosen over a no-pad variant because it handles any width and does not depend on DeepGEMM's undocumented N-alignment tolerance -- it is the logic upstream already merged. Test is a faithful port of vllm-project#53132's test_modelopt_fp8_pb_wo_hides_output_padding for the generic method. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

The rebase onto vllm-project#53132/Kimi K3 brought in mla_attention.py's _get_kv_b_proj_input_dtype, which imported and isinstance-checked the deleted ModelOptFp8PbWoLinearMethod (a runtime ImportError for MLA + block-FP8 models, caught by mypy). Map it to the generic ModelOptLinearMethod + the block-FP8 weight key (kFp8Static128BlockSym), same absorption as deep_gemm_warmup. Also reword a test docstring (mis-scaled -> wrong scales) to satisfy the typos hook. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com>

…trim FormatScheme A FP8_PB_WO output width that is not a multiple of 128 (a partial trailing block; e.g. GLM's replicated fused_qkv_a_proj = 2048 + 576 = 2624) is padded up to a block boundary with zeros before the kernel post-load, the GEMM runs on the padded weight, and the output is trimmed back to the logical width with bias added after. This is wei-zhao vllm-project#53132's approach, expressed as a FormatScheme (_Fp8PbWoPartialBlock) on the generic method rather than a per-format class -- post_process pads the weight, apply wraps the kernel to trim the output. A new FormatScheme.apply hook makes this compute-time residue expressible without touching ModelOptLinearMethod. KFp8Block128 sizes the weight_scale by cdiv to match the (padded) block count. No-op for the common block-aligned case. Chosen over a no-pad variant because it handles any width and does not depend on DeepGEMM's undocumented N-alignment tolerance -- it is the logic upstream already merged. Test is a faithful port of vllm-project#53132's test_modelopt_fp8_pb_wo_hides_output_padding for the generic method. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

The rebase onto vllm-project#53132/Kimi K3 brought in mla_attention.py's _get_kv_b_proj_input_dtype, which imported and isinstance-checked the deleted ModelOptFp8PbWoLinearMethod (a runtime ImportError for MLA + block-FP8 models, caught by mypy). Map it to the generic ModelOptLinearMethod + the block-FP8 weight key (kFp8Static128BlockSym), same absorption as deep_gemm_warmup. Also reword a test docstring (mis-scaled -> wrong scales) to satisfy the typos hook. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Juhi Mittal <juhim@nvidia.com>

wzhao18 marked this pull request as ready for review August 20, 2026 16:17

wzhao18 requested review from AndreasKaratzas, LucasWilkinson, MatthewBonanni, WoosukKwon, mgoin, pavanimajety, robertgshaw2-redhat, tlrmchlsmth, yewentao256 and zyongye as code owners August 20, 2026 16:17

claude Bot reviewed Aug 20, 2026

View reviewed changes

mergify Bot added quantization kimi k3 nvidia labels Aug 20, 2026

github-project-automation Bot added this to NVIDIA Aug 20, 2026

Support kimi k3 nvfp4 checkpoint

20d2f65

Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>

wzhao18 force-pushed the wzhao/k3-nvfp4 branch from 2eced09 to 20d2f65 Compare August 20, 2026 16:18

Move padding handling for modelopt fp8_pb_wo to linear method

9811b6d

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>

wzhao18 requested review from DarkLight1337, ProExpertProg, houseroad, njhill, youkaichao and ywang96 as code owners August 21, 2026 03:36

mergify Bot added deepseek Related to DeepSeek models mrv2 Model Runner V2 specific labels Aug 21, 2026

wzhao18 force-pushed the wzhao/k3-nvfp4 branch from 6e8ad3b to 9811b6d Compare August 21, 2026 03:37

zyongye approved these changes Aug 21, 2026

View reviewed changes

github-project-automation Bot moved this to Ready in NVIDIA Aug 21, 2026

zyongye enabled auto-merge (squash) August 21, 2026 04:07

Merge branch 'main' into wzhao/k3-nvfp4

dd72b3d

github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 21, 2026

zyongye merged commit f8e0602 into vllm-project:main Aug 21, 2026
136 of 137 checks passed

github-project-automation Bot moved this from Ready to Done in NVIDIA Aug 21, 2026

djramic mentioned this pull request Aug 21, 2026

[ROCm][Test] Use platform FP8 dtype in ModelOpt FP8_PB_WO test #53268

Merged

4 tasks

wyettzeng pushed a commit to wyettzeng/vllm that referenced this pull request Aug 21, 2026

Support kimi k3 nvfp4 checkpoint (vllm-project#53132)

78863a6

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com> Signed-off-by: Wyett <wyettzeng@gmail.com>

jahnclawdmonet mentioned this pull request Aug 28, 2026

[Bug]: DeepGEMM is reported supported on sm_121 (GB10) but faults — support_deep_gemm() accepts the whole 120 capability family #54125

Open

1 task

juhi10071998 mentioned this pull request Aug 31, 2026

[ModelOpt] Redesign the LinearMethod classes using the generic QuantKey-driven method #49381

Merged

am-cohere pushed a commit to am-cohere/vllm that referenced this pull request Sep 1, 2026

Support kimi k3 nvfp4 checkpoint (vllm-project#53132)

ce5793e

Signed-off-by: wzhao18 <wzhao18.sz@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com>

huthvincent mentioned this pull request Sep 2, 2026

[Bugfix][Quantization] Run block kernel post-processing for ModelOpt FP8_PB_WO #48422

Closed

Navigation Menu

Uh oh!

Uh oh!

Support kimi k3 nvfp4 checkpoint - #53132

Support kimi k3 nvfp4 checkpoint#53132
zyongye merged 3 commits into
vllm-project:mainfrom
wzhao18:wzhao/k3-nvfp4

wzhao18 commented Aug 20, 2026 •

edited by github-actions Bot

Loading

Uh oh!

claude Bot left a comment

Uh oh!

zyongye commented Aug 21, 2026

Uh oh!

github-actions Bot commented Aug 21, 2026

Uh oh!

Uh oh!

DamirLisak commented Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

wzhao18 commented Aug 20, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

zyongye commented Aug 21, 2026

Uh oh!

github-actions Bot commented Aug 21, 2026

Uh oh!

Uh oh!

DamirLisak commented Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wzhao18 commented Aug 20, 2026 •

edited by github-actions Bot

Loading