[Docs][Cookbook] AMD DeepSeek-V4 FP8 cells are missing SGLANG_DSV4_FP4_EXPERTS=0 and cannot load on a cold cache

[michaelzhang-ai]

Scope: one file, nine added lines, all in docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx. No code, no test, no workflow.

Motivation

All nine AMD DeepSeek-V4 FP8 cookbook cells name an FP8 checkpoint but omit SGLANG_DSV4_FP4_EXPERTS=0, the env var that tells the loader the routed experts are FP8 rather than mxfp4-packed:

Cell	had it before
mi300x | flash | fp8 — low-latency, balanced, high-throughput	no
mi355x | flash | fp8 — low-latency, balanced, high-throughput	no
mi355x | pro | fp8 — low-latency, balanced, high-throughput	no
h200 | flash | fp8 and h200 | pro | fp8 (6 cells)	yes

All nine are marked verified: true. Every other FP8 DSV4 config in the repo sets the var — the MI35x nightly tests (test_deepseek_v4_flash_fp8.py, ..._fp8_tbo.py, test_deepseek_v4_pro_fp8.py), the disaggregation tests, kv_canary/consts.py, the manual test/manual/dsv4/ suite, the NPU perf tests, and scripts/ci/slurm/launch_mi355x.sh, which derives it from PRECISION. The AMD cookbook cells are the only FP8 DSV4 configuration in the tree that leaves it unset.

Why this is latent rather than obviously broken. SGLANG_DSV4_FP4_EXPERTS defaults to True:

    # Set False when using FP4-to-FP8 converted DeepSeek V4 checkpoint.
    SGLANG_DSV4_FP4_EXPERTS = EnvBool(True)

ModelConfig falls back to auto-detection when it is unset:

        if is_deepseek_v4(self.hf_config):
            self.is_fp4_experts = envs.SGLANG_DSV4_FP4_EXPERTS.get()
            if (
                not envs.SGLANG_DSV4_FP4_EXPERTS.is_set()
                or envs.SGLANG_DSV4_FP4_DEQUANT.is_set()
            ):
                from sglang.srt.configs.deepseek_v4 import try_detect_fp4_experts

                detected = try_detect_fp4_experts(self.model_path)
                if detected is not None:
                    self.is_fp4_experts = detected

But try_detect_fp4_experts reads the safetensors header out of the local HF cache and returns None when the repo is not cached — its own docstring says "None when the header isn't readable (HF slug not cached yet, etc.). Caller falls back to user default." find_local_repo_dir returns None for an uncached slug, so on a fresh host detection is skipped and the True default wins. The FP8 checkpoint is then read at mxfp4 shapes.

That is why these cells can be marked verified and still be broken: they work on a machine that already has the weights and fail on a new one, which is exactly the machine a reader following the cookbook is on.

Modifications

Added "SGLANG_DSV4_FP4_EXPERTS=0" to the env list of the nine AMD FP8 cells, matching what the H200 FP8 cells already do. Nothing else changed — no flag, no other env var, no verified value, and no FP4 cell.

Setting it explicitly also removes the cold/warm cache dependence, so the recipe behaves identically on any host instead of depending on whether the checkpoint happens to be cached.

Accuracy Tests

Observed on 8 x MI300X running the mi300x | flash | fp8 | low-latency | single cell verbatim on a cold HF cache — run 32914407497, from the CI job added in #36396. The log contains no Auto-detected DSV4 routed-expert layout line, confirming detection returned None, and then all eight ranks crash during weight load:

File "/sglang-checkout/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 703, in _load_w13
    expert_data.copy_(loaded_weight)
RuntimeError: The size of tensor a (2048) must match the size of tensor b (4096) at non-singleton dimension 1

The factor of two is the mxfp4 packing: two 4-bit experts per byte against FP8's one. Other ranks report the same mismatch at their own shard shapes (128 vs 256, 128 vs 32). The server then exits with code -9 and never serves a request.

Static validation:

• pre-commit run --files docs/src/snippets/configs/deepseek-ai/deepseek-v4.jsx — all hooks pass
• re-audited every cell in the file by parsing it: all 15 FP8 cells now carry the var, and no non-FP8 cell was touched
• the diff is nine added lines, one per cell
• the Mintlify preview build renders it: the deployed page carries 21 occurrences of SGLANG_DSV4_FP4_EXPERTS against production's 12, a delta of exactly the nine patched cells

Not verified: I could not re-run the MI355X Flash/Pro FP8 cells, so those six are fixed by the same reasoning rather than by observation. The reasoning is checkpoint-level, not architecture-level — the MI35x nightly tests set the same var against the same checkpoints and pass — but if an AMD reviewer wants to confirm one MI355X cell before merging, that seems worth doing.

Worth a follow-up conversation separately: try_detect_fp4_experts silently falling back to a default that hard-crashes at weight load is a poor failure mode. Loading an FP8 checkpoint at mxfp4 shapes is detectable at config time, and a clear error naming the env var would be better than a shape mismatch eight stack frames into the MoE weight loader. This PR does not attempt that.

Speed Tests and Profiling

No performance impact; this makes the recipes load at all on a fresh host.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): ❌ Run #34829299271
Latest PR Test (Extra): ❌ Run #34829298608
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
lmsysorg	🟢 Ready	View Preview	Aug 26, 2026, 1:23 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.