[dsv4] Trigger MHC prenorm prewarm at weight-load time with rank sync

[YAMY1234]

Motivation

#27986 made the DSV4 MHC prenorm prewarm trigger lazily inside the first forward that carries tokens — per rank, uncoordinated. On wide-EP disaggregated prefill this is fatal: 0-token EP peers still launch deep_gemm.fp8_fp4_mega_moe every layer and wait for the compiling rank inside the kernel's NVLink barrier, which device-traps after 180 s. The 23-bucket burst (one DeepGEMM tf32_hc_prenorm_gemm + one TileLang big-fuse variant per bucket) measures 159–166 s on a cold JIT cache on GB200, so any >~15 s trigger stagger between ranks blows the budget:

DeepGEMM NVLink barrier timeout (180s): rank=2, counter=550, signal=7, target=8, phase=1, sign=0, tag=1
Assertion failed: .../deep_gemm/include/deep_gemm/comm/barrier.cuh:76
→ tvm.error.InternalError: CUDA driver error (sm100_fp8_fp4_mega_moe.hpp): 719 (CUDA_ERROR_LAUNCH_FAILED)

Reproduces deterministically on GB200 disagg 1p1d dep8-dep16 (8k1k); currently worked around by SGLANG_DSV4_MHC_PREWARM=0. Single-node dep4 setups pass only because their triggers happen to align (skew ≪ 180 s) — verified by a synthetic 4-rank repro where a single 200 s host stall reproduces the exact barrier-timeout signature.

Modifications

• Trigger the same prewarm from the tail of DeepseekV4ForCausalLM.load_weights on every rank — before the memory pool, CUDA-graph capture, and any forward — then get_tp_group().barrier() so no rank proceeds while a peer is still compiling, and torch.cuda.empty_cache() (this now runs before init_memory_pool, so the multi-GB transient prewarm buffers must not skew KV-pool sizing).
• Keep the in-forward trigger as a fallback for entry points that bypass load_weights (e.g. kernel unit tests); add prewarm duration + rank-sync logs.
• Remove the orphaned prewarm_mhc_token_counts/_buckets pair (added by perf(dsv4): add MHC token-count prewarm #25810, removed by refactor(dsv4): route MHC prenorm through DeepGEMM wrapper #26238, resurrected without a caller by the [DeepSeek-V4] Add mhc_fused_post_pre kernel #25976 merge).

Accuracy Tests

No kernel/model math changes — the prewarm replays the existing _mhc_pre_impl path with dummy inputs at load time (same code the lazy trigger ran). Server responses validated on DSV4-Flash (TP4+DP4+deepep) and DSV4-Pro (disagg dep8/dep16).

Speed Tests and Profiling

A/B on GB200 disagg 2-node dep8 prefill + 4-node dep16 decode — same recipe, container, and base commit; the only difference is this patch:

	without fix	with fix
prewarm trigger	1 rank, mid-serving	all 8 prefill ranks at load (153–173 s, rank sync +0.0 s)
NVLink barrier timeouts	16 (fields match production crash)	0
sa-bench (isl 8192 / osl 1024, conc 8)	crash, no results	completes, 80/80 requests, 464 tok/s output

Startup cost is unchanged (~160 s one-shot on a cold JIT cache; ~0 s warm — same as the lazy trigger paid, just moved off the serving path and aligned across ranks).

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

---

CI States

Latest PR Test (Base): ❌ Run #28634631368
Latest PR Test (Extra): ❌ Run #28634631251

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[Fridge003]

/rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp8_h200.py test/registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py test_mhc_kernels.py

[github-actions]

Results for /rerun-test test/registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py test/registered/models_e2e/test_deepseek_v4_flash_fp8_h200.py test/registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py test_mhc_kernels.py:

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp4_megamoe_b200.py
cd test/ && python3 registered/cp/test_deepseek_v4_flash_fp4_b200_cp.py

🚀 8-gpu-h200 (1 test): ✅ View workflow run

cd test/ && python3 registered/models_e2e/test_deepseek_v4_flash_fp8_h200.py

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/kernels/test_mhc_kernels.py