Serve lossless MXFP4-CSF DeepSeek-V4-Flash and Vision-Exp checkpoints

[voipmonitor]

Load lossless MXFP4-CSF checkpoints of DeepSeek-V4-Flash and DeepSeek-V4-Flash-Vision-Exp

The MXFP4-CSF reader so far knew the DeepSeek-V4.1-Flash family only. This adds the deepseek_v4_flash family, so the lossless CSF containers of DeepSeek-V4-Flash (0731) and DeepSeek-V4-Flash-Vision-Exp serve with compressed expert scales that stay compressed in GPU memory:

• vllm/models/deepseek_v4/mxfp4_csf.py: the family reads the compressed scales of the 256 routed MXFP4 experts in each of the 43 target layers by DeepSeek's native tensor names, at TP1/TP2/TP4/TP8;
• routed experts keep MXFP8 activations like the native B12X MXFP4 path (VLLM_B12X_MOE_FP4_FORCE_A16=1 still selects BF16);
• attention, dense and shared-expert weights keep the source block-FP8 configuration; MTP/DSpark draft experts, the vision tower, the aligner and the image embeddings load unchanged;
• expert parallelism (including MegaMoE) is rejected, because those experts would bypass the compressed-scale reader.

Checkpoints (built with trellis-quant, byte-exact restore verified for every tensor): local-inference-lab/DeepSeek-V4-Flash-0731-lossless-CSF (166.89 → 159.44 GB) and local-inference-lab/DeepSeek-V4-Flash-Vision-Exp-lossless-CSF (167.82 → 160.37 GB).

Tests

• tests/quantization/test_mxfp4_csf.py: 65 passed in the beta image on CPU (family detection, tensor names, TP shards, EP rejection, config rewrite).
• ruff check / format clean.
• GPU (2x RTX PRO 6000 Max-Q, dk ds4-flash profile parameters, this loader as a Python overlay on the beta image): DeepSeek-V4-Flash-0731 lossless CSF serves at TP2; model memory 77.29 GiB vs 80.64 GiB for the original, KV cache 1,853,797 vs 1,308,256 tokens (+41.7%); greedy answers coherent (they diverge from the original's after a few words, as two original runs can: serving runs non-deterministic MoE output; the b12x MXFP4-CSF expert output is bit-exact against native in tests/moe/test_mxfp4_csf.py). DeepSeek-V4.1-Flash CSF (TP4) served with the same overlay (shared loader files): KV 12,951,057 vs 9,566,426 tokens, prefill -1.3/-1.7% at 8K/32K, decode -1% to -3.5% depending on concurrency. Speed numbers for DS4-Flash and the Vision variant follow in a comment.

Fragment vllm-mxfp4-csf-ds4-flash. AI-assisted (Claude Code); reviewed and run by the submitter.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[voipmonitor]

GPU A/B with the dk profile parameters (2x RTX PRO 6000 Max-Q 325 W per server, beta image f71cfb7b with the merged beta b12x/vLLM code as overlays; both arms up at once on separate GPU pairs, measured one at a time; decode = aggregate output tok/s at context 0, two 30 s runs; prefill = one 8K / 32K prompt):

DeepSeek-V4-Flash-0731, TP2, DSpark 5, 8 slots

	KV tokens	C1	C4	C8	prefill 8K	prefill 32K
original	1,308,256	200 / 225	452 / 436	660 / 664	11,042	11,132
lossless CSF	1,853,797 (+41.7%)	206 / 200	457 / 439	667 / 647	11,120	11,374

Per verification step (tok/s divided by the measured acceptance length): C1 -1.6%, C4 -2.0%, C8 -0.2%; raw C1 differs by the acceptance noise of single-request runs.

DeepSeek-V4-Flash-Vision-Exp, TP2, DSpark 3, 4 slots (local CSF serving directory; the Hub upload of this checkpoint is still pending)

	KV tokens	C1	C2	C4	prefill 8K	prefill 32K
original	1,306,708	179 / 182	304 / 288	416 / 420	8,284	9,526
lossless CSF	1,850,842 (+41.6%)	188 / 188	315 / 276	430 / 432	8,169	9,247

Per verification step: C1 -1.7%, C2 -2.7%, C4 +0.8%; prefill -1.4% / -2.9%.

Model memory per GPU: DS4-Flash 77.29 vs 80.64 GiB, Vision 77.73 vs 81.08 GiB. Greedy answers are coherent with both checkpoints and diverge after a few words, as two runs of serving with non-deterministic MoE output can; the b12x MXFP4-CSF expert output is bit-exact against native in deterministic mode (tests/moe/test_mxfp4_csf.py).

[voipmonitor]

Update: the DeepSeek-V4-Flash-Vision-Exp lossless CSF checkpoint is now on the Hub (local-inference-lab/DeepSeek-V4-Flash-Vision-Exp-lossless-CSF @ 73a48c14), and blackwell-llm-docker #123 makes it the ds4-vision default.