[Feature] Add DWDP (Distributed Weight Data Parallelism) for MoE prefill

[yhyang201]

WIP: This PR is in early development. The code is subject to significant changes as we iterate on the implementation and benchmarking.

Summary

Add DWDP as a new parallelism strategy for MoE prefill. Instead of EP all-to-all token dispatch, each rank prefetches peer expert weights via NVLink P2P and computes all experts locally, eliminating all cross-rank synchronization.

Based on the DWDP paper (arXiv 2604.01621), ported from TensorRT-LLM's implementation using CUDA VMM composite virtual addresses.

Key Design

• CUDA VMM transport: alloc shareable handles, exchange via pidfd (x86_64) or fabric (aarch64), create composite VA per weight
• Double-buffered weight prefetch: layer l+1 remote experts prefetched during layer l MoE compute
• Zero sync: auto-enables dp_attention, dp_lm_head, SCHEDULER_SKIP_ALL_GATHER; idle ranks skip forward entirely

Usage

# PD disaggregated prefill (recommended)
python -m sglang.launch_server --tp 4 --dwdp-size 4 \
  --disaggregation-mode prefill ...

# Standalone
python -m sglang.launch_server --tp 4 --dwdp-size 4 ...

Results (4x B200, gpt-oss-120b, prefill-only)

DWDP4 vs DEP4 (EP4 + allgather), same total tokens:

MNT	ISL=4K	ISL=8K	ISL=16K	ISL=32K
16K	1.16x	1.37x	1.18x	-
32K	1.30x	1.45x	1.48x	1.92x

bench_serving at saturation (CONC=128, ISL=8K): DWDP4 506K tok/s vs DEP4 329K tok/s (1.54x).

---

CI States

Latest PR Test (Base): ❌ Run #29770352071
Latest PR Test (Extra): ✅ Run #29799031259

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[yhyang201]

/rerun-test test_disaggregation_dwdp_gpt_oss.py test_disaggregation_dwdp_mimo.py

[github-actions]

Results for /rerun-test test_disaggregation_dwdp_gpt_oss.py test_disaggregation_dwdp_mimo.py:

🚀 4-gpu-b200 (1 test): ❌ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_dwdp_gpt_oss.py

🚀 8-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_dwdp_mimo.py

[yhyang201]

/rerun-test test_disaggregation_dwdp_gpt_oss.py test_disaggregation_dwdp_mimo.py

[github-actions]

Results for /rerun-test test_disaggregation_dwdp_gpt_oss.py test_disaggregation_dwdp_mimo.py:

🚀 4-gpu-b200 (1 test): ✅ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_dwdp_gpt_oss.py

🚀 8-gpu-b200 (1 test): ❌ View workflow run

cd test/ && python3 registered/disaggregation/test_disaggregation_dwdp_mimo.py

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[Fridge003]

We might add a document later for DWDP