[yctseng0211]

Motivation

RCCL	DDA blocks	Total tok/s	Median TTFT (s)	Median TPOT (ms)	Median ITL (ms)	P99 ITL (ms)
2.30.7 bundled, default	256	7730	22.47	52.64	30.59	40.14
2.30.7 bundled, MAXBLOCKS=64	64	8135	22.71	48.64	27.75	31.21
2.32.3 latest, default	96	8247	21.36	49.04	28.67	36.78
2.32.3 latest, MAXBLOCKS=64	64	8173	22.23	48.85	27.94	31.40

• The bundled 2.30.7 (2.30.7-HEAD:7da4302) uses a fixed 256 DDA fabric blocks. Capping at 64 gives +5.2% throughput, -9.3% median ITL and -22% p99 ITL.
• 2.32.3 (rocm-systems develop @ 4ca8623) defaults to 96 blocks here. That alone gives the best throughput (+6.7%) and TTFT (-5.0%) and -6.3% median ITL, but p99 ITL improves by only 8.4%. The first run in the fresh container was discarded as JIT warm-up.
• On 2.32.3, MAXBLOCKS=64 still trades about 4% TTFT for -15% p99 ITL and -2.5% median ITL; throughput is flat.
• At TP4, MAXBLOCKS=64 makes no difference (all metrics within +/-0.4%).

Background (4GPUs)
We first found this regression with RCCL 2.30.7. At conc=64 (8192 in / 1024 out), median ITL went from 40.59 ms on stock 2.30.4 to 50.85 ms (+25%), although throughput improved by 6.8%. That 2.30.7 was built by us from rocm-systems @ 44704e4 against the ROCm 10 SDK in rocm/sgl-dev:v0.5.19-rocm10-mi45x-20260913. Root cause: RCCL_DDA_FABRIC_MAXBLOCKS defaults to 192, so the DDA fabric barrier issues 768 flag writes per all-reduce (96 on 2.30.4). Capping it at 64 brought median ITL to 38.63 ms (-4.8% vs stock) and throughput to +20.4% vs stock.

Modifications

Pin rccl to 1c0ab425abda9d1fd80fa977c253556b14eb140b
ROCm/rocm-systems#14254 fixes the random hang issue. The current image is built from that PR's head; we will rebuild from main once the PR is merged.

Accuracy Tests

Speed Tests and Profiling

unset NCCL_MIN_NCHANNELS NCCL_MAX_NCHANNELS   
NCCL_DEBUG=INFO HSA_ENABLE_COREDUMP=0 \
NCCL_IGNORE_CPU_AFFINITY=1 NCCL_CUMEM_ENABLE=1 NCCL_MNNVL_ENABLE=1 HSA_HOTSWAP_DISABLE=1 \
SGLANG_DEFAULT_THINKING=1 SGLANG_DSV4_REASONING_EFFORT=max \
SGLANG_HACK_FLASHMLA_BACKEND=unified_kv_triton AITER_BF16_FP8_MOE_BOUND=0 AITER_FORCE_A8W4=1 \
SGLANG_OPT_FUSE_MHC_POST_PRE=0 ENABLE_CK=0 SGLANG_USE_AITER=1 SGLANG_USE_ROCM700A=0 \
[RCCL_DDA_FABRIC_MAXBLOCKS=64] \
python3 -m sglang.launch_server --model-path /models/DeepSeek-V4-Flash/ --trust-remote-code --tp 8 \
  --attention-backend dsv4 --page-size 256 --mem-fraction-static 0.8 --swa-full-tokens-ratio 0.15 \
  --tool-call-parser deepseekv4 --reasoning-parser deepseek-v4 \
  --chunked-prefill-size 2048 --max-prefill-tokens 2048 --cuda-graph-max-bs-decode 256 \
  --max-running-requests 256 --kv-cache-dtype fp8_e4m3 --port 30000 --disable-custom-all-reduce

python3 -m sglang.bench_serving --backend sglang --port 30000 --dataset-name random \
  --random-range-ratio 1.0 --warmup-requests 1 --flush-cache \
  --random-input-len 8192 --random-output-len 1024 --num-prompts 320 --max-concurrency 64 \
  --output-file bench_conc64.jsonl

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ✅ Run #40494526127
Latest PR Test (Extra): ❌ Run #40494526148
Latest PR Test (AMD ROCm 10): ❌ Run #40494526163

[sogalin]

LGTM, specific for ROCm10.1