[kkHuang-amd]

Motivation

In the DeepSeek-V4 DP-attention TP-MoE decode path (MAX_LEN padding, no EP,
tp_size == attn_dp_size), the MoE combine previously did the MoE-internal
post-experts all_reduce (aiter cross_device_reduce_2stage) followed by
dp_scatter. An all_reduce moves ~2× the bytes of a reduce_scatter.

This PR replaces that with an equal-chunk reduce_scatter for the MAX_LEN
decode combine, using aiter's custom (symmetric-memory) kernel on ROCm, so both
halves of the DP-MoE gather/combine are aiter custom kernels
(allgather_vec for the gather + reduce_scatter_first_dim for the combine)
instead of falling back to a generic RCCL collective.

This is the decode-side complement to the already-shipped SUM_LEN gatherv pair
(all_gatherv + reduce_scatterv, SGLANG_DP_USE_GATHERV), which only applies
to prefill — decode under SGLANG_USE_ROCM700A=0 is MAX_LEN, where gatherv is
inactive. Together: prefill uses reduce_scatterv (SUM_LEN), decode uses the
equal-chunk reduce_scatter (MAX_LEN), so neither phase pays the full all_reduce
traffic.

Modifications

• environ.py: add SGLANG_DP_USE_REDUCE_SCATTER. Platform-agnostic
  (aiter custom kernel on ROCm, RCCL elsewhere) with a platform-conditional
  default: ON for HIP, OFF otherwise — implemented via a lazy callable default
  (_default_hip, no torch import at module load) plus generic callable-default
  support in EnvField.get() (backward compatible: non-callable defaults
  unchanged).
• distributed/parallel_state.py: GroupCoordinator.reduce_scatter_tensor
  now tries an aiter custom reduce_scatter first (equal-chunk, size-bounded via
  should_custom_ar, CUDA-graph capture aware), else falls back to RCCL. Gated by
  SGLANG_DP_USE_REDUCE_SCATTER. Mirrors the existing _all_gather_into_tensor
  custom all-gather path.
• layers/dp_attention.py: is_dp_gatherv_active() now also requires SUM_LEN
  padding (not _DpGatheredBufferWrapper.is_dp_max_padding()). The gatherv pair is
  only valid under SUM_LEN; this keeps callers without a ForwardBatch (e.g.
  dp_reduce_scatter_tensor) consistent and lets the MAX_LEN decode combine route
  to the equal-chunk reduce_scatter.
• models/deepseek_v4.py: in the MAX_LEN decode combine, skip the
  MoE-internal post-experts all_reduce (use_reduce_scatter=True) and call
  dp_reduce_scatter_tensor (routes to the equal-chunk reduce_scatter / aiter,
  since gatherv is inactive under MAX_LEN). Renamed locals for clarity
  (_use_gatherv_pair → _use_reduce_scatterv, _use_aiter_rs →
  _use_reduce_scatter).

Gated, falls back to RCCL when the shape/size/topology is unsupported, and only
activates on the validated path (MAX_LEN decode, no EP, tp == dp).

Test commands

Server (HIP/MI300, tp8 + dp8 dp-attention; SGLANG_DP_USE_REDUCE_SCATTER
defaults ON on ROCm):

SGLANG_USE_ROCM700A=0 \
SGLANG_DP_USE_GATHERV=1 \
SGLANG_SHARED_EXPERT_TP1=1 \
SGLANG_DP_SHARED_EXPERT_LOCAL=1 \
SGLANG_USE_AITER=1 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro --trust-remote-code \
  --tp 8 --dp 8 --enable-dp-attention --enable-prefill-delayer \
  --attention-backend dsv4 --kv-cache-dtype fp8_e4m3 \
  --disable-radix-cache --disable-shared-experts-fusion \
  --page-size 256 --mem-fraction-static 0.90 --swa-full-tokens-ratio 0.15 \
  --cuda-graph-max-bs 512 --max-running-requests 512 \
  --chunked-prefill-size 65536 --port 8000

A/B: feature ON is the default on ROCm; force OFF with
SGLANG_DP_USE_REDUCE_SCATTER=0 (decode combine reverts to all_reduce + dp_scatter).

Accuracy (gsm8k 5-shot, via lm-eval):

lm_eval --model local-completions \
  --model_args model=deepseek-ai/DeepSeek-V4-Pro,base_url=http://localhost:8000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False \
  --tasks gsm8k --num_fewshot 5

Speed (c512, 8k/1k and 1k/1k):

# 8k/1k
python3 -m sglang.bench_serving --backend sglang-oai \
  --base-url http://127.0.0.1:8000 --model deepseek-ai/DeepSeek-V4-Pro \
  --dataset-name random --random-input-len 8192 --random-output-len 1024 \
  --random-range-ratio 1.0 --num-prompts 4096 --max-concurrency 512 \
  --request-rate inf --warmup-requests 1024

# 1k/1k
python3 -m sglang.bench_serving --backend sglang-oai \
  --base-url http://127.0.0.1:8000 --model deepseek-ai/DeepSeek-V4-Pro \
  --dataset-name random --random-input-len 1024 --random-output-len 1024 \
  --random-range-ratio 1.0 --num-prompts 4096 --max-concurrency 512 \
  --request-rate inf --warmup-requests 1024

Accuracy Tests

gsm8k 5-shot (DeepSeek-V4-Pro, tp8 + dp8 dp-attention, feature ON):

metric	value
flexible-extract	0.9545
strict-match	0.9553

Matches the baseline (~0.95); no correctness regression.

Cross-platform validation — NVIDIA B200 (CUDA / NCCL path)

SGLANG_DP_USE_REDUCE_SCATTER is platform-agnostic: on non-HIP it defaults OFF
and routes to the generic NCCL reduce_scatter (not the aiter custom kernel).
Validated on 8× NVIDIA B200 that explicitly enabling the feature is
numerically correct (DeepSeek-V4-Pro, tp8 + dp8 dp-attention, flashinfer_mxfp4
MoE, feature ON).

gsm8k 5-shot (lm-eval, feature ON):

metric	value
flexible-extract	0.9530
strict-match	0.9538

Matches the baseline (~0.95); no correctness regression on the CUDA path.

Commands (B200; on CUDA the collectives default OFF, so enable explicitly — A/B by
setting SGLANG_DP_USE_REDUCE_SCATTER=0):

SGLANG_DP_USE_REDUCE_SCATTER=1 \
SGLANG_DP_USE_GATHERV=1 \
SGLANG_JIT_DEEPGEMM_PRECOMPILE=0 \
SGLANG_OPT_SWA_SPLIT_LEAF_ON_INSERT=1 \
SGLANG_OPT_USE_JIT_NORM=1 \
SGLANG_OPT_USE_JIT_INDEXER_METADATA=1 \
SGLANG_OPT_USE_TOPK_V2=1 \
SGLANG_OPT_USE_CUSTOM_ALL_REDUCE_V2=1 \
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V4-Pro --trust-remote-code \
  --tp 8 --dp 8 --enable-dp-attention \
  --disable-radix-cache --max-running-requests 256 \
  --mem-fraction-static 0.90 --swa-full-tokens-ratio 0.1 \
  --moe-runner-backend flashinfer_mxfp4 --chunked-prefill-size 8192 \
  --disable-flashinfer-autotune --port 8000

lm_eval --model local-completions \
  --model_args model=deepseek-ai/DeepSeek-V4-Pro,base_url=http://localhost:8000/v1/completions,num_concurrent=64,max_retries=3,tokenized_requests=False \
  --tasks gsm8k --num_fewshot 5

Speed Tests and Profiling

c512, random ratio 1.0, num-prompts=4096, warmup=1024, tp8 + dp8
dp-attention, SGLANG_USE_ROCM700A=0, SGLANG_DP_USE_GATHERV=1,
SGLANG_SHARED_EXPERT_TP1=1, SGLANG_DP_SHARED_EXPERT_LOCAL=1,
--chunked-prefill-size 65536. Same-session A/B, reduce_scatter OFF vs ON:

workload	OFF (tok/s)	ON (tok/s)	Δ
8k/1k	37,171	37,701	+1.4%
1k/1k	18,026	18,666	+3.5%

Decode TPOT improves (the combine moves ~½ the bytes); the win grows on the more
decode-bound 1k/1k workload, as expected for a decode-combine optimization.

Trace verification (decode, MAX_LEN): the per-layer combine kernel is now
aiter reduce_scatter_first_dim (×61); cross_device_reduce_2stage drops to 0;
the gather stays aiter allgather_vec. So both gather and combine are aiter custom
kernels.

Checklist

• Format your code according to pre-commit.
• Provide accuracy and speed benchmark results.
• Add unit tests.
• Update documentation.

---

CI States

Latest PR Test (Base): ❌ Run #28071472700
Latest PR Test (Extra): ❌ Run #28071472574

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[HaiShaw]

/tag-and-rerun-ci

[HaiShaw]

@amd-bot ci-status

[amd-bot]

@HaiShaw

CI Status for PR #29103

Merge verdict: ⚠️ Do not merge on a green-CI basis yet — the change is effectively untested by PR CI. All 4 executed failures are pre-existing/infra issues unrelated to this PR (Qwen3-8B PD-disagg, NPU perf, CPU bench/packaging), but the PR's core new functionality (DSv4 aiter reduce-scatter decode) lives behind code paths that only run in nightly AMD suites, which PR CI never executes. AMD PR CI is also incomplete: the stage-b 8-GPU disagg job failed, cancelling wait-for-stage-b-amd so downstream AMD stages were not reached.

Caution

This PR's changed code is not exercised by any PR-CI test. The new _use_reduce_scatter path in deepseek_v4.py is gated by SGLANG_DP_USE_REDUCE_SCATTER (default ON for ROCm/HIP) but only triggers for DeepSeek-V4 + DP attention (tp==attn_dp) + MAX_LEN decode. The only AMD tests that exercise that combo are registered nightly=True:

• test/registered/amd/test_deepseek_v4_pro_fp4.py → suite nightly-amd-8-gpu-mi35x-deepseek-v4-pro
• test/registered/amd/accuracy/mi30x/test_deepseek_v32_dp_eval_amd.py → suite nightly-amd-accuracy-8-gpu-deepseek-v32-dp

The dp_attention.py is_dp_gatherv_active() change is additionally gated behind SGLANG_DP_USE_GATHERV (default off). Green PR CI does NOT verify this PR. Before merge, run the nightly DSv4 suites above (8-GPU MI35x) with the default HIP setting, and ideally an A/B with SGLANG_DP_USE_REDUCE_SCATTER=0 vs 1 to confirm numerical parity + the claimed ~2x combine-traffic reduction.

Changed files: python/sglang/srt/distributed/parallel_state.py (+49/-0), python/sglang/srt/environ.py (+32/-3), python/sglang/srt/layers/dp_attention.py (+10/-2), python/sglang/srt/models/deepseek_v4.py (+33/-4)

Executed CI failure attribution: AMD: 1 failure (0 related) · Others: 3 failures (0 related) · plus AMD pipeline incomplete (stage-c not reached). Aggregator/*-finish/*-gate/wait-for-* jobs are fast-fail cascades of the rows below.

AMD Executed Failures

Job	Test File	Test Function	Error	Related?	Why
stage-b-test-large-8-gpu-mi35x-disaggregation-amd	test/registered/amd/disaggregation/test_disaggregation_basic.py	TestDisaggregationAccuracy.test_gsm8k	KeyError: 'answer' after KVTransferError … Aborted by AbortReq / client-side disconnects	🟢	Test uses Qwen3-8B PD-disaggregation; PR touches only the DSv4 DP-MoE combine path, which this test never reaches. KV-transfer abort is a disagg-transport issue.

wait-for-stage-b-amd was cancelled and downstream AMD stage(s) not reached because the disagg job above failed → AMD CI signal is incomplete.

Other Executed Failures

Job	Test File	Test Function	Error	Related?	Why
stage-b-test-1-npu-a2 (0)	test/registered/ascend/basic_function/quant/test_npu_w8a8_quantization.py	w8a8 throughput check	AssertionError: 448.95 not >= 700 (perf threshold)	🟢	NPU/Ascend w8a8 perf threshold; PR is ROCm/HIP-only and changes no NPU code path.
build-test (xeon-gnr, base-b-test-cpu)	test/registered/bench_fn/test_benchmark_datasets_api.py	test_* (returncode 1≠2)	AssertionError: 1 != 2 + IntegrityError: model.safetensors mismatch	🟢	CPU bench-API test + a model-download integrity failure (infra); unrelated to GPU MoE combine code.
base-a-test-cpu (3)	test/registered/unit/test_bench_long_context.py	import-time (module load)	ModuleNotFoundError: No module named 'sglang.bench_serving'	🟢	CPU packaging/import issue in a benchmark helper; no relation to the diff.

Details / what to do before merge

• Coverage (top priority): the diff's value is invisible to PR CI. Trigger the two nightly DSv4 AMD suites (8-GPU MI35x) on this branch and confirm pass. Add an A/B run toggling SGLANG_DP_USE_REDUCE_SCATTER to validate output parity, since this changes the MoE-combine collective (all_reduce + dp_scatter → equal-chunk reduce_scatter) and a wrong skip of the MoE-internal all_reduce would silently corrupt logits, not crash.
• AMD completeness: re-run AMD CI so stage-c/downstream actually execute; the current run stopped at stage-b due to the unrelated Qwen3-8B disagg failure. For full signal despite that unrelated failure, the bypass-fastfail label is an option (uses more CI resources — use sparingly).
• The 4 executed failures need no action from this author — they are pre-existing/infra (NPU perf, CPU import/integrity, Qwen3 disagg KV-transfer). Worth a quick check that the disagg KV-transfer abort isn't a broader AMD-disagg regression, but it's not this PR's responsibility.

---

Generated by amd-bot using Claude Code CLI

[HaiShaw]

@kkHuang-amd please add follow-up PR for test cases.