Fix Qwen3 MoE double-reduce when DP attention + EP + reduce_scatterv (#23729)

[ByronHsu]

Motivation

PR #22642 introduced should_use_dp_reduce_scatterv() which fuses the post-MoE all-reduce with the dp_scatter into a single reduce_scatterv inside LayerCommunicator. The qwen2_moe.py forward path was patched to skip the explicit tensor_model_parallel_all_reduce when the fast path is active — but qwen3_moe.py was missed.

As a result, Qwen3 MoE models running with DP attention + EP (e.g. --tp 2 --dp 2 --ep 2 --enable-dp-attention, no --moe-a2a-backend) double-reduce the MoE output: once explicitly via moe_expert_parallel_all_reduce (and moe_tensor_model_parallel_all_reduce) in forward_normal, then again inside reduce_scatterv from the communicator. The output is silently corrupted; the model still produces fluent text but logprobs differ from a tp-only baseline by 0.5–2 nats.

Reported as #23729.

Reproduction (no DP attention vs DP+EP, same model, same prompt, temperature 0)

# Server A: tp-only baseline
python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 \
  --tp-size 2 --trust-remote-code --mem-fraction-static 0.8 --port 30000

# Server D: tp + dp + ep + dp_attention (the broken combination)
python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B-Instruct-2507 \
  --tp-size 2 --dp-size 2 --ep-size 2 --enable-dp-attention \
  --trust-remote-code --mem-fraction-static 0.8 \
  --chunked-prefill-size 16384 --cuda-graph-max-bs 128 --port 30000

curl -sS -X POST http://127.0.0.1:30000/generate \
  -H 'Content-Type: application/json' \
  -d '{"text":"Write a science fiction for me please.","sampling_params":{"temperature":0,"max_new_tokens":100,"top_p":1.0},"return_logprob":true,"logprob_start_len":0}'

variant	common-prefix vs A	mean |Δlp|	max |Δlp|	A and D text
before fix	2 / 100	1.291	2.032	diverge
after fix	100 / 100	0.027	0.279	identical

v0.5.9 is also clean (76/100 prefix, max |Δlp|=0.147 — pure float drift), confirming the regression was introduced after that release.

A and D agreement is now within normal float drift, matching the level of agreement between any of the (TP-only, DP-attn-only, EP-only) configurations on this model.

Modifications

In Qwen3MoeSparseMoeBlock.forward_normal, mirror the qwen2_moe.py guard onto both the EP and TP all-reduce branches: skip them when should_use_dp_reduce_scatterv() returns True (the communicator now performs the reduce inside reduce_scatterv).

-        if self.ep_size > 1 and not should_allreduce_fusion:
+        if (
+            self.ep_size > 1
+            and not should_allreduce_fusion
+            and not should_use_dp_reduce_scatterv()
+        ):
             final_hidden_states = moe_expert_parallel_all_reduce(final_hidden_states)

         if (
             self.tp_size > 1
             and not should_allreduce_fusion
             and not use_reduce_scatter
             and not should_use_flashinfer_cutlass_moe_fp4_allgather()
+            and not should_use_dp_reduce_scatterv()
         ):
             final_hidden_states = moe_tensor_model_parallel_all_reduce(...)

Accuracy

The fix only deletes redundant collectives along the affected branch; numerics on every other path are identical. Validated above on Qwen3-30B-A3B-Instruct-2507 in the broken configuration; configurations that didn't trigger should_use_dp_reduce_scatterv() (TP-only, DP-attn alone, EP alone) are unaffected.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy.
• Follow the SGLang code style guidance.

cc @YAMY1234 (PR #22642 author)

Fixes #23729

[gemini-code-assist]

Code Review

This pull request integrates the should_use_dp_reduce_scatterv check into the qwen3_moe model's forward pass to conditionally skip all-reduce operations. The review feedback suggests also incorporating the use_reduce_scatter flag check within the expert parallel block to ensure consistency with the tensor parallel logic and prevent redundant reductions.

[gemini-code-assist]

The expert parallel (EP) all-reduce block should also check the use_reduce_scatter flag, mirroring the logic used in the tensor parallel (TP) block at line 345. This ensures that redundant EP all-reduces are skipped when the communicator is configured to perform a reduce-scatter operation instead of an all-reduce.

Suggested change

	if (
	self.ep_size > 1
	and not should_allreduce_fusion
	and not should_use_dp_reduce_scatterv()
	):
	if (
	self.ep_size > 1
	and not should_allreduce_fusion
	and not use_reduce_scatter
	and not should_use_dp_reduce_scatterv()
	):