[AMD] Speed up DSV4 MoE weight loading from mmap views

[bingxche]

Summary

DeepSeek-V4-Pro TP8 model loading was bottlenecked by 140,544 small MoE H2D copies per rank. Rank-local tensor views retained much larger safetensors mmap-backed storage, causing TP0/TP7 to spend 27-32 minutes in H2D while the other ranks took about 4 minutes.

This PR optionally copies only oversized or non-contiguous CPU weight views into independent contiguous storage immediately before H2D. It is disabled by default via SGLANG_MOE_COPY_WEIGHT_VIEWS_BEFORE_H2D and enabled in selected AMD DSV4, Qwen, and GPT-OSS validation jobs.

Results

• model loading: ~35m -> 6m20s (5.6x faster)
• edge-rank H2D copy p50: 41-45 ms -> ~5.4 ms
• CPU copy overhead: ~10s wall time
• no observed regression: GSM8K 0.9484; bs=32 throughput 1565.09 tok/s vs. 1563.15 baseline

Additional MI300 nightly validation

The branch was synchronized through b61cb5f9, including the ROCm AITER update used by the failing scheduled nightlies. All four previously timing-out large-model paths passed with the opt-in enabled:

• Qwen3.5-397B BF16, ROCm 7.0: 2879s load followed by file timeout -> 1488s load, accuracy passed, 1.93x faster.
• Qwen3.5-397B BF16, ROCm 7.2: 69/94 shards after 3310s, then timeout -> 94/94 in 1465s, accuracy passed, more than 2.3x faster to completion.
• Qwen3-235B BF16: 87/118 shards after 3492s, then timeout -> 852s load, test passed, more than 4.1x faster to completion.
• Qwen3-235B FP8: 913s -> 509s in the same runs, 1.79x faster.
• GPT-OSS-20B BF16: 545s -> 70s, 7.8x faster.
• GPT-OSS-120B BF16: two launch attempts failed to finish within 900s each -> 433s in the same runs, more than 2.1x faster.

Validation:

Before

After

CPU tests cover zero-offset views, nonzero-offset views, non-contiguous tensors, and tensors that already have exact storage.

---

CI States

Latest PR Test (Base): ✅ Run #30602173081
Latest PR Test (Extra): ❌ Run #30602172994

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[bingxche]

@amd-bot ci-status

[amd-bot]

@bingxche

CI Status for PR #32315

Merge verdict: PR CI is incomplete — do not read "green" as verified. This is an AMD-targeted PR, but every AMD stage-b/stage-c test was skipped because wait-for-stage-a-amd timed out (a runner-queue infra issue, not this PR — see below). Of the 4 executed test/infra failures (XPU, NPU, NVIDIA, CPU), none touch this PR's code paths — the change is default-off (SGLANG_MOE_COMPACT_WEIGHT_LOAD=False) and confined to MoE weight loading. The new helper's unit test passed on CPU, but the PR's actual value (env-gated compact load + AMD DSV4-Pro accuracy/perf) is nightly-only and was not exercised by PR CI.

Caution

This PR's real value is not verified by PR CI. The env-gated integration path (_maybe_compact_loaded_weight behind SGLANG_MOE_COMPACT_WEIGHT_LOAD=1) and the ROCm 7.2 DeepSeek-V4-Pro accuracy/perf validation run only in the nightly suite, which does not run on PR CI. On top of that, the AMD PR pipeline's stage-b/stage-c jobs were all skipped (wait-for-stage-a-amd timed out at 240min while stage-a-test-1-gpu-small-amd sat in the runner queue ~8h, then eventually passed at 19:22 UTC). Before merge, confirm the attached full ROCm 7.2 DSV4-Pro nightly dispatch is green (author links runs/30073792821), and consider re-running AMD PR CI so downstream stages actually execute.

Changed files: .github/workflows/nightly-test-amd-rocm720.yml (+2), python/sglang/srt/environ.py (+3), python/sglang/srt/layers/moe/fused_moe_triton/layer.py (+24), test/registered/unit/layers/moe/test_compact_weight_load.py (+52)

Executed CI failure attribution: AMD: 0 executed test failures (all stage-b/c skipped — not tested) · Others: 4 executed failures (0 related to this PR)

AMD CI — incomplete (no downstream tests ran)

Job	Test File	Test Function	Error	Related?	Why
wait-for-stage-a-amd	N/A	N/A	Timeout waiting for stage-a-amd jobs (240×60s poll)	🟢	Infra/runner-queue: stage-a-amd queued ~8h; wait caps at 4h. stage-a-amd itself PASSED at 19:22 UTC. All AMD stage-b/c skipped as a result.

Other Executed Failures

Job	Test File	Test Function	Error	Related?	Why
stage-b-test-1-gpu-xpu	test/registered/spec/eagle/test_spec_eagle_parity.py	(suite error)	RuntimeError: level_zero backend failed: 20 (UR_RESULT_ERROR_DEVICE_LOST)	🟢	XPU device-lost hardware fault; PR touches no XPU/eagle code.
stage-b-test-4-npu-a3	N/A (pip install)	N/A	ProtocolError: Connection broken: IncompleteRead → "custom container failed, contact runner admin"	🟢	NPU network/container infra failure during dependency install.
base-c-test-8-gpu-h20 (0)	N/A (setup)	N/A	git clone human-eval: CONNECT tunnel failed, response 502 (exit 128)	🟢	Network/proxy infra failure cloning eval harness. (base-c-4-gpu-h100 + wait-for-base-b cancelled = fast-fail cascade from this.)
build-test (xeon-gnr, base-b-test-cpu)	test/registered/chunked_prefill/test_mm_chunked_embedding_unit.py	test_by_item_list_matches_tensor, test_full_list_matches_tensor, test_full_matches_by_item	3 failed, 2 passed	🟢	Multimodal chunked-embedding path; unrelated to the default-off MoE weight-load change.

Coverage — what actually ran

• ✅ New unit test passed: test_compact_weight_load.py (4/4) ran on base-a-test-cpu (0) — verifies the pure _make_loaded_weight_compact helper (zero/nonzero-offset views, non-contiguous, already-compact).
• ❌ Integration path NOT run on PR CI: _maybe_compact_loaded_weight is only active when SGLANG_MOE_COMPACT_WEIGHT_LOAD=1, which the diff sets only in the ROCm 7.2 DSV4-Pro nightly job. No PR-CI test sets that flag, so the real weight-loading behavior + GSM8K/perf gates are unverified by this run.
• ❌ AMD backend signal missing: the ROCm test stages that would at least exercise MoE loading on AMD were all skipped.

What to do before merge

• Confirm the full ROCm 7.2 DSV4-Pro nightly dispatch is green — this is the only place the env-gated path + accuracy/perf are validated (author links runs/30073792821).
• Re-run AMD PR CI (or update the branch) so stage-a-test-1-gpu-small-amd isn't stuck behind the runner queue and downstream AMD stages actually execute; the timeout was infra, not a code failure.
• The 4 executed failures (XPU device-lost, NPU network, NVIDIA 502, CPU mm-chunked-embedding) are unrelated to this PR — no action needed on this PR, though the CPU test_mm_chunked_embedding_unit.py failure looks like a separate real issue worth a rerun to confirm flaky vs. broken-on-main.

---

Generated by amd-bot using Claude Code CLI

[yctseng0211]

stage-b-test-1-gpu-large-amd (linux-mi300-1gpu-sglang, 1)
will be resolved by #32862