[Kimi] Support kimi-k3

[hnyls2002]

Day-0 support for the Kimi K3 model.

Nvidia Support

Day 0 Cuda 13 image: docker pull lmsysorg/sglang:kimi-k3
Day 0 Cuda 12 image: docker pull lmsysorg/sglang:kimi-k3-cu12

AMD Support

Day 0 image: docker pull lmsysorg/sglang-rocm:rocm720-mi35x-k3-20260727
More details: #32548

Links

Cookbook: https://docs.sglang.io/cookbook/autoregressive/Moonshotai/Kimi-K3
Blog: https://www.lmsys.org/blog/2026-07-27-kimi-k3-day0-support
DSpark: https://huggingface.co/RadixArk/Kimi-K3-DSpark

Consolidated VLM follow-ups

This branch now absorbs #31318, #31581, #31587, #31637, #31638, and #31944. #31932 is superseded by the stronger any(...) assertion already present on this branch.

• avoid redundant Kimi vision projection split/cat copies
• make seeded random-image benchmark requests reproducible across processes
• default the CUDA IPC pool-handle cache on only when CUDA IPC transport is selected; this reuses the existing bounded pool mapping and does not reserve another pool
• keep packed MoonViT grid metadata on CPU and reuse host-precomputed attention metadata
• share the vision RoPE component while selecting the fastest layout-specific path: in-place JIT for contiguous K2.5/K2.7 Q/K and the existing strided-QKV Triton path for K3
• preserve invalid media input as ValueError so the serving layer returns HTTP 400 without logging an expected traceback

PR-before vs PR-after performance

Projection packing microbenchmark, RTX 5090 BF16: 1/2/4/8 image chunks improved from 0.0242/0.0222/0.0231/0.0274 ms to 0.0140/0.0126/0.0126/0.0134 ms (1.73x-2.05x). This isolates the post-projector packing path.

CUDA IPC handle-cache A/B, 4x H100 TP4, Qwen3.6-35B-A3B-FP8 with four seeded random images per request:

Workload	Metric	Before	After	Change
output=1, burst	Request throughput	7.55 req/s	20.43 req/s	+170.6%
output=1, rate=4	Mean TTFT	496.30 ms	97.70 ms	-80.3%
output=64, burst	Request throughput	5.59 req/s	17.17 req/s	+207.2%
output=64, burst	Mean E2E	2007.29 ms	560.14 ms	-72.1%

The cache-off/cache-on runs used the same 512 MiB producer pool and had identical per-GPU memory after warmup (69534 / 68076 / 68058 / 67578 MiB). CUDA IPC transport itself remains opt-in because its configured pool consumes HBM.

Packed-grid metadata profiling on H200 TP8 removed all 12 Memcpy DtoH events and reduced cudaStreamSynchronize from 60 to 48 for the measured request set. End-to-end bench_serving stayed within run-to-run noise, so this is a synchronization cleanup rather than a standalone throughput claim.

MoonViT RoPE microbenchmark on NVIDIA B300 BF16 (16 heads, head dim 72); preparation is performed once per encoder forward and the table measures the per-layer Q/K operation:

Vision tokens	Before: portable	After: shared in-place	Speedup
256	47.58 us	7.60 us	6.26x
1,024	48.14 us	7.53 us	6.39x
4,096	63.55 us	7.84 us	8.10x
8,192	116.23 us	14.52 us	8.01x

The earlier H200 TP8 profile measured launch count 648 -> 108 and relevant GPU time -5.802 ms for the random-image request set; end-to-end serving remained within noise at that workload size.

Validation

• NVIDIA B300: 260 passed, 32 skipped, 18 subtests passed across the affected Kimi K2.5/K3, multimodal transport, media error, dataset, server-args, vision metadata, and vision RoPE tests
• B300 fused RoPE coverage includes 1/480/5660/8360 random-token shapes, BF16/FP16, CUDA graph replay, and K2.5 head-dim 72 parity
• all changed-file pre-commit hooks and git diff --check pass

Rebasing Tracker

Merge 07/25 commits: 19ba402
Merge 07/26 commits: e7eec37
Merge 07/27 commits: 8a263bb
Merge 07/28 first half (to 9c0dbf): 9ef4c95
Merge 07/27 second half: b951c6b
Merge 07/29 first half (to e1f2f9d): b51f61f
Merge 07/29 second half: ac77803
Merge 07/31 (to e1964d): 49b58a0

---

CI States

Latest PR Test (Base): 🚫 Run #30932096634
Latest PR Test (Extra): ✅ Run #30950069784

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[github-actions]

Results for /rerun-test test/registered/models_e2e/test_ministral4_models.py:

🚀 2-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_ministral4_models.py

[ispobock]

/rerun-test test_mxfp4_sm90_cutlass.py

[github-actions]

Results for /rerun-test test_mxfp4_sm90_cutlass.py:

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/unit/layers/quantization/test_mxfp4_sm90_cutlass.py

[BBuf]

/rerun-test test/registered/models_e2e/test_dsa_glm52_hisparse.py

[github-actions]

Results for /rerun-test test/registered/models_e2e/test_dsa_glm52_hisparse.py:

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_dsa_glm52_hisparse.py

[billishyahao]

Hi, I hit the following issue on mi355x:

stdout:
[1/2] /opt/rocm-7.2.0/bin/hipcc -std=c++17 -O2 -fPIC -D__HIP_PLATFORM_AMD__=1 -fno-gpu-rdc --offload-arch=gfx950 -DUSE_ROCM -std=c++20 -O3 -DHIP_FP8_TYPE_E4M3=1 -I/opt/venv/lib/python3.10/site-packages/tvm_ffi/include -I/opt/venv/lib/python3.10/site-packages/tvm_ffi/include -I/opt/rocm-7.2.0/include -I/billhe/sglang-k3-day0/python/sglang/kernels/jit/include -c /root/.cache/tvm-ffi/sgl_kernel_jit_kimi_k3_situ_and_mul_bf16_t_false_926fca12de2a1a77__arch_9.5__tvmffi_0.5.16.dev20260802+g0877a0e2f1/cuda.cu -o cuda_0.o
FAILED: [code=1] cuda_0.o
/opt/rocm-7.2.0/bin/hipcc -std=c++17 -O2 -fPIC -D__HIP_PLATFORM_AMD__=1 -fno-gpu-rdc --offload-arch=gfx950 -DUSE_ROCM -std=c++20 -O3 -DHIP_FP8_TYPE_E4M3=1 -I/opt/venv/lib/python3.10/site-packages/tvm_ffi/include -I/opt/venv/lib/python3.10/site-packages/tvm_ffi/include -I/opt/rocm-7.2.0/include -I/billhe/sglang-k3-day0/python/sglang/kernels/jit/include -c /root/.cache/tvm-ffi/sgl_kernel_jit_kimi_k3_situ_and_mul_bf16_t_false_926fca12de2a1a77__arch_9.5__tvmffi_0.5.16.dev20260802+g0877a0e2f1/cuda.cu -o cuda_0.o
In file included from /root/.cache/tvm-ffi/sgl_kernel_jit_kimi_k3_situ_and_mul_bf16_t_false_926fca12de2a1a77__arch_9.5__tvmffi_0.5.16.dev20260802+g0877a0e2f1/cuda.cu:7:
/billhe/sglang-k3-day0/python/sglang/kernels/jit/csrc/kimi_k3/situ_and_mul.cuh:21:10: fatal error: 'cuda_fp8.h' file not found
   21 | #include <cuda_fp8.h>
      |          ^~~~~~~~~~~~
1 error generated when compiling for gfx950.
failed to execute:/opt/rocm-7.2.0/lib/llvm/bin/clang++  --offload-arch=gfx950  -std=c++17 -O2 -fPIC -D__HIP_PLATFORM_AMD__=1 -fno-gpu-rdc -DUSE_ROCM -std=c++20 -O3 -DHIP_FP8_TYPE_E4M3=1 -I/opt/venv/lib/python3.10/site-packages/tvm_ffi/include -I/opt/venv/lib/python3.10/site-packages/tvm_ffi/include -I/opt/rocm-7.2.0/include -I/billhe/sglang-k3-day0/python/sglang/kernels/jit/include -c -x hip /root/.cache/tvm-ffi/sgl_kernel_jit_kimi_k3_situ_and_mul_bf16_t_false_926fca12de2a1a77__arch_9.5__tvmffi_0.5.16.dev20260802+g0877a0e2f1/cuda.cu -o "cuda_0.o"
ninja: build stopped: subcommand failed.

A quick sample fix is :

diff --git a/python/sglang/srt/layers/activation.py b/python/sglang/srt/layers/activation.py
index 837b766e74..06d1f1e3eb 100644
--- a/python/sglang/srt/layers/activation.py
+++ b/python/sglang/srt/layers/activation.py
@@ -208,6 +208,14 @@ class SituAndMul(MultiPlatformOp):

         return situ_and_mul(x, None, self.beta, self.linear_beta)

+    # MultiPlatformOp.forward_hip defaults to forward_cuda, which JIT-builds
+    # kimi_k3/situ_and_mul.cuh; that header includes <cuda_fp8.h> and so cannot
+    # compile under hipcc. Introduced upstream by fb207b72b0 (#32890)
+    # Falling back to the native formula keeps K3 runnable here.
+    def forward_hip(self, x: torch.Tensor) -> torch.Tensor:
+        return self.forward_native(x)
+
     def forward_cpu(self, x: torch.Tensor) -> torch.Tensor:
         return self.forward_native(x)

cc @HaiShaw

@billishyahao, @RolaoDenthu fixed it in #33447

[BBuf]

/rerun-test test/registered/models_e2e/test_gpt_oss_4gpu_mxfp4.py

[github-actions]

Results for /rerun-test test/registered/models_e2e/test_gpt_oss_4gpu_mxfp4.py:

🚀 4-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_gpt_oss_4gpu_mxfp4.py

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_gpt_oss_4gpu_mxfp4.py