[beyondHJM]

[Feature] Add SSD-backed Expert Pack inference for DeepSeek-V4-Flash and Kimi-K3

Summary

This PR adds an opt-in SSD-backed inference path for DeepSeek-V4-Flash and Kimi-K3. It allows these MoE models to run when their complete weights are larger than GPU VRAM and host RAM combined.

Routed expert weights are stored on SSD and loaded only when selected by the router. The implementation uses:

• an expert-major Expert Pack layout for efficient random access;
• direct I/O into aligned pinned host buffers;
• asynchronous host-to-device transfers; and
• a byte-budgeted LFU/LRU GPU cache for frequently used experts.

The runtime does not prune, merge, skip, replace, or re-quantize experts, and does not reduce Expert Top-K. The default auto, safetensors, and gguf loading paths remain unchanged; SSD execution is enabled only through the explicit expert_pack path.

Supported Scope

ExpertPackModelLoader is selected only with --load-format expert_pack. Official checkpoints loaded through the standard auto or safetensors paths never enter this loader.

The currently validated DeepSeek input is the single-file MXFP4 GGUF used by frob/deepseek-v4-flash-0731:latest. Its routed expert matrices use MXFP4; the large non-routed matrices are mostly Q8_0, router input weights are BF16, and the inventory also contains F32 and I32 normalization or auxiliary tensors. The inspected GGUF contains 129 MXFP4, 512 Q8_0, 43 BF16, 641 F32, and 3 I32 tensors. It contains no FP8 tensors.

The other validated input is the text-only 38-shard KIMI-K3-MXP4-DERISKED-Q2_K-*.gguf, whose routed experts use Q2_K for gate/up and Q3_K for down. The Kimi launcher extracts the official text_config into a flat KimiLinearConfig; the complete multimodal Kimi-K3 checkpoint is not supported by this loader. Other GGUF quantization variants have not been validated and are not currently claimed as supported.

Usage

Install this branch from source before running the examples:

python3 -m pip install -e "python"

DeepSeek-V4-Flash

The tested MXFP4 GGUF is available as frob/deepseek-v4-flash-0731:latest, which was imported from unsloth/DeepSeek-V4-Flash-0731-GGUF. The benchmark used the immutable Ollama model blob with SHA-256 947ac34c08c0e5c5752ac76398f934b3b6b4075cfe915ba43dd5ac754900a4cd.

python3 examples/runtime/deepseek_v4/benchmark_deepseek_5090.py \
  --gguf /path/to/deepseek-v4-flash-0731-mxfp4.gguf \
  --prompt "Explain why the sky appears blue." \
  --max-new-tokens 200

Kimi-K3

Download all 38 Q2_K GGUF shards from Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED. Also place the tokenizer/config files from moonshotai/Kimi-K3 in a sibling directory named kimi-k3-tokenizer.

hf download Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED \
  --include "KIMI-K3-MXP4-DERISKED-Q2_K-*.gguf" \
  --local-dir /path/to/kimi-k3

python3 examples/runtime/kimi_k3/benchmark_kimi_k3_5090.py \
  --gguf /path/to/kimi-k3/KIMI-K3-MXP4-DERISKED-Q2_K-00001-of-00038.gguf \
  --prompt "Explain why the sky appears blue." \
  --max-new-tokens 200

The first shard is only the entrypoint. The launcher automatically discovers and uses all 38 shards in the same directory. Both launchers derive the remaining paths automatically. If the Expert Pack is missing, it is built on the first launch and reused afterward.

Model	First Expert Pack build	Approximate first-run readiness
DeepSeek-V4-Flash	Estimated 5-10 minutes on the benchmark NVMe SSD	8-15 minutes
Kimi-K3	29 minutes 42 seconds measured	35-45 minutes

The 38 Kimi-K3 source shards and generated Expert Pack occupy approximately 1.814 TiB in total, so a 4 TB SSD is recommended for practical headroom.

Correctness

• DeepSeek-V4-Flash produced semantically equivalent answers to Ollama across travel planning, science, history, programming, and mathematics prompts.
• Kimi-K3 matched the 200-token SGLang reference output exactly. All 92 routed layers executed Top-16 experts with fallback_count=0 and io_errors=0.

Performance

The target platform is one RTX 5090 with 32 GB VRAM, 32 GB host RAM, and NVMe SSD storage. The retained measurements were collected on RTX 5090 hosts; host RAM and SSD capacity differed between the two model runs.

DeepSeek-V4-Flash vs Ollama

Five prompts were measured with 200 generated tokens per prompt.

Metric	SGLang	Ollama	Result
Prefill	3.265 token/s	1.457 token/s	2.24x faster
Decode	2.020 token/s	1.354 token/s	1.49x faster

Kimi-K3 vs llama.cpp

The SGLang run used a 91-token prompt and generated 200 tokens.

Metric	SGLang	llama.cpp	Result
Prefill	0.66412 token/s	0.11289 token/s	5.88x faster
Decode	0.44830 token/s	0.1810 token/s	2.48x faster

The llama.cpp decode baseline came from an earlier 50-token run, so the 2.48x decode figure is indicative rather than a strict equal-length comparison.

---

CI States

Latest PR Test (Base): ❌ Run #32837312025
Latest PR Test (Extra): ❌ Run #32837311604
Latest PR Test (AMD ROCm 7.2): ❌ Run #32837311774

[beyondHJM]

/tag-and-rerun-ci

[beyondHJM]

/rerun-failed-ci

[beyondHJM]

/rerun-failed-ci

[beyondHJM]

/rerun-failed-ci

[beyondHJM]

/rerun-failed-ci

[BBuf]

https://github.com/sgl-project/sglang/actions/runs/32837312025/job/97803698788?pr=35314

[merrymercy]

Style comment:

A top-level folder tools/expert_pack just for this pr is not allowed.
Can you move this folder to another place? It can be under scripts or your custom stuff or srt/utils

@beyondHJM @BBuf can you follow up on this?

[cyxlily]

Heads up — this PR accidentally reverted the XPU MHC dispatch that #32166 added to python/sglang/srt/models/deepseek_v4.py‎ (likely a stale-branch merge). Filed #36803 to restore it. Please double check for similar silent reverts in future merges, thanks!

[253844077]

Is there any deployment that implements hybrid inference for DS4.1 across memory, SSD, and GPU — putting Engram in memory, MoE on SSD, and activated experts on GPU? I feel this could be done with just 512GB of memory, a 2TB SSD, and 48GB of VRAM.