[McZyWu]

Motivation

The Kimi-K3 ModelSlim W4A8 multi-node accuracy job fails after loading its 351 checkpoint shards with:

ValueError: Kimi-K3 MLA K projection must remain GGUF Q4_0

Failed job: https://github.com/sgl-project/sglang/actions/runs/32936865342/job/98168349010

#35314 added split-GGUF K/V support for the Kimi-K3 ExpertPack path, but selected that representation through supports_kimi_k3_quantized_latent_projections. ModelSlim already advertises that capability for Kimi-K3 latent-MoE linear layers, while its MLA checkpoint stores kv_b_proj.weight as a regular dense FLOAT tensor. As a result, model construction removed kv_b_proj, registered unloaded GGUF placeholders, and the post-load Q4_0/Q2_K validation failed.

Modifications

• Add a dedicated supports_kimi_k3_split_gguf_kv_b capability to ExpertPackConfig.
• Select split-GGUF MLA K/V storage only through that format-specific capability.
• Keep supports_kimi_k3_quantized_latent_projections unchanged, preserving ModelSlim latent-MoE quantization and the ExpertPack path added by Support deepseek v4 and kimi k3 on ssd #35314.
• Add a regression test proving that ModelSlim uses dense kv_b_proj and an explicitly split-GGUF configuration uses separate K/V tensors.

Current head: bca476e03b118b474192754705320c565658d7f6.

Checkpoint inspection

The target checkpoints were inspected before the hardware run:

• /home/weights/Kimi-K3-w4a8-int-moe
  • 351 checkpoint shards.
  • All 24 MLA layers contain dense self_attn.kv_b_proj.weight tensors.
  • The ModelSlim quantization description marks those tensors as FLOAT.
  • No k_b_qweight or v_b_qweight tensors are present.
• /home/weights/Kimi-K3-DSpark
  • Five-layer BF16 draft checkpoint.
  • No MLA kv_b or split-GGUF tensors are present.

This confirms that the failing checkpoint must retain the dense ModelSlim MLA path.

Four-node NPU validation

The PR production logic was validated on four Ascend nodes with 16 ranks per node (64 NPU ranks total). The hardware run used commit f956e283e24b33d505bec0bb7173f646a4c9e9c7; the current head only applies Black's one-line formatting to the same helper, with no behavioral change.

Configuration:

• TP64, DP4, DP attention, and DP LM head.
• ModelSlim W4A8 main model and DeepEP auto.
• DSpark draft model, block size 7, Ascend draft attention, and Triton linear-attention verification.
• --mem-fraction-static 0.75, --max-mamba-cache-size 240, --chunked-prefill-size 16384, and --max-running-requests 64.
• The same ModelSlim and DSpark checkpoint paths used by the failing CI case.

Startup and inference results:

Check	Result
Distributed initialization	All 64 TP/EP ranks initialized
Main checkpoint load	351/351 shards loaded in about 181 seconds
Original MLA post-load failure	Not reproduced; zero Kimi-K3 MLA K projection errors
ModelSlim path	Every rank reported Using ModelSlimW8A8Int8
DSpark graph capture	Completed on all ranks; sampled capture time 2.17-2.98 seconds
Server startup	Ready about 353 seconds after launch
Built-in warmup	HTTP 200
External /health	HTTP 200 before and after the benchmark
External completion request	HTTP 200 and entered K3 reasoning/generation
Post-run critical scan	Zero Traceback, ValueError, RuntimeError, MLA projection error, or segfault on every node
Cleanup	Target service processes: 0; NPU process entries: 0 on all four nodes

GSM8K 200

Command shape:

python -m sglang.test.few_shot_gsm8k \
  --num-questions 200 \
  --num-shots 5 \
  --max-new-tokens 512 \
  --parallel 128 \
  --temperature 0 \
  --host 127.0.0.1 \
  --port 30000 \
  --data-path /path/to/gsm8k.jsonl

Results:

Metric	Result
Completed requests	200/200
Accuracy	0.970 (194/200)
Invalid	0.005 (1/200)
Latency	156.517 s
Output throughput	126.421 token/s
Benchmark exit status	0

The result is above the 96.5% 200-question baseline reported in #33465 and consistent with the approximately 98% result reported around #34715.

Compatibility with #35314

The fix keeps #35314's ExpertPack behavior intact:

• The real ExpertPackConfig still has supports_kimi_k3_quantized_latent_projections = True and now explicitly declares supports_kimi_k3_split_gguf_kv_b = True.

• A direct final-head class-level check passed:

  PASS: ExpertPack keeps split GGUF K/V; ModelSlim keeps dense kv_b_proj
  

• Kimi-K3 GGUF regression suite: 9/9 passed.

• ExpertPack runtime suite: 10 passed; 1 CUDA-pinned-memory test skipped as expected on the NPU host.

• The Support deepseek v4 and kimi k3 on ssd #35314 K/V checkpoint mapping, Q4_0/Q2_K post-load checks, forced MLA dispatch, ExpertPack loader/runtime, and CUDA MXFP4 kernels are unchanged by this PR.

The available Ascend hosts do not contain a Kimi-K3 GGUF/ExpertPack checkpoint. In addition, #35314's Kimi ExpertPack runtime is a single-GPU CUDA/RTX 5090 path, so its full E2E cannot run on this NPU cluster. The actual configuration-selection path and all hardware-independent #35314 regressions were exercised instead.

Other checks

• test/registered/expert_pack/test_kimi_k3_gguf.py: 9/9 passed on the final head.
• test/registered/expert_pack/test_expert_pack_runtime.py: 10 passed, 1 skipped on the final head.
• python scripts/lint/check_registered_tests.py: passed.
• python scripts/lint/check_no_bare_pytest_main.py: passed.
• Python syntax compilation for all changed files: passed.
• git diff --check: passed.
• GitHub lint run #33047357734: passed.
• PR is open and GitHub reports it as mergeable.

The remaining required pr-test-finish failure is the repository gate rejecting a PR without the maintainer-controlled run-ci label; its log reports Missing required label 'run-ci'. The manually executed four-node NPU validation above covers the originally failing hardware path.

Validation artifacts

Artifacts are retained on the validation nodes under:

/home/wzy/pr36603-k3-gsm8k200-20260827/

This directory contains the four rank logs, exact launch script, smoke request, gsm8k200.log and status, the direct ExpertPack compatibility check, and both final regression logs/status files.

Checklist

• Format the code with the repository's Black/pre-commit rules.
• Add and run unit/regression tests.
• Run the full four-node ModelSlim + DSpark weight-loading and inference path.
• Run GSM8K 200 accuracy/stress validation.
• Verify Support deepseek v4 and kimi k3 on ssd #35314's hardware-independent ExpertPack path is preserved.
• Documentation is not needed for this internal capability correction.
• Follow the SGLang code-style guidance.

---

CI States

Latest PR Test (Base): ⏳ Run #33136037154
Latest PR Test (Extra): ❌ Run #33136037039
Latest PR Test (AMD ROCm 7.2): ❌ Run #33136037153

[McZyWu]

Four-node Kimi-K3 hardware validation

Validated the production logic now at PR head bca476e03b118b474192754705320c565658d7f6 on 4 Ascend nodes / 64 NPU ranks with the CI ModelSlim checkpoint and the Kimi-K3 DSpark draft checkpoint. The final commit after the hardware run is Black-only formatting of the same capability helper.

Configuration:

• TP64, DP4, DP attention and DP LM head
• ModelSlim W4A8 main model, DeepEP auto
• DSpark draft, block size 7, Ascend draft attention, Triton linear-attention verification
• --enable-shared-experts-attn-tp and --enable-dense-mlp-attn-tp from [bugfix] [NPU] fix transpose batch matmul K*B exceed 65536. #34715
• decode graph batch sizes 11 and 16

Results:

• Regression suite: 9/9 passed.
• Loaded all 351 ModelSlim shards. The original post-load error (Kimi-K3 MLA K projection must remain GGUF Q4_0) did not occur.
• All 64 ranks completed main-model post-load, DSpark initialization, draft verify graph capture, and server warmup.
• /health: HTTP 200; external completion request: HTTP 200.
• GSM8K 5-shot, 200 questions, temperature 0, 128 concurrency:
  • Accuracy: 0.970
  • Invalid: 0.005
  • Latency: 156.517 s
  • Output throughput: 126.421 token/s
• Post-benchmark critical-error scan was zero on every node; the server remained healthy.

This covers the failing weight-loading path and a full multi-node inference/accuracy run.

Compatibility with #35314

• The real ExpertPackConfig retains supports_kimi_k3_quantized_latent_projections = True and now explicitly declares supports_kimi_k3_split_gguf_kv_b = True.
• A direct final-head compatibility check confirms that K3 still selects split GGUF K/V for ExpertPackConfig, while selecting dense kv_b_proj for ModelSlimConfig.
• All 9 Kimi-K3 GGUF mapping, post-processing, shard-order, and capability regression tests pass on the final head.
• The ExpertPack runtime suite also passes: 10 passed and 1 CUDA-pinned-memory test skipped as expected on the NPU host.
• The Support deepseek v4 and kimi k3 on ssd #35314 K/V tensor mapping, Q4_0/Q2_K post-load validation, forced MLA dispatch, and ExpertPack loader/runtime code are unchanged.

The available Ascend hosts do not contain a Kimi-K3 GGUF/ExpertPack checkpoint, and #35314's ExpertPack kernel is a single-GPU CUDA/RTX 5090 path, so its full E2E cannot be run on this NPU cluster. The class-level selection and all hardware-independent #35314 regressions were exercised instead.

[sglang-npu-bot]

/tag-and-rerun-ci