Skip to content

feat: support TP>1 Domino rollout for DFlash V2 - #37069

Merged
Qiaolin-Yu merged 18 commits into
sgl-project:mainfrom
jianuo-huang:feat/domino-tensor-parallel-pr36899
Sep 12, 2026
Merged

Qiaolin-Yu merged 18 commits into
sgl-project:mainfrom
jianuo-huang:feat/domino-tensor-parallel-pr36899

Conversation

@jianuo-huang

@jianuo-huang jianuo-huang commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Summary

Extend the optimized Domino rollout from PR #36899 to tensor parallel (TP>1), including CUDA Graph capture and eager fallback paths.

  • Compute first-token argmax from vocab-sharded LM-head logits with deterministic global tie handling.
  • Add a full-vocabulary TP gather path for small logical tensors.
  • Add a compact global candidate-pool path for larger batches/graphs: local top-k, TP all-gather, and candidate base-logit all-reduce.
  • Validate LM-head and embedding shard metadata, padded vocab layout, TP rank, and unsupported added-vocab layouts.
  • Keep B=1 on full gather and use compact candidate pooling for B>1 CUDA-graph buckets.

Correctness and validation

  • SGLang nightly DLC unit test: 29 passed, 35 subtests passed.
  • TP2 fake-shard tests compare both full-gather and compact paths against the TP1 full-vocabulary Domino oracle with exact token parity.
  • Real 2-GPU TP2 Qwen3-8B service loaded the Domino checkpoint, captured decode graph buckets (bs=1 and bs=2), and completed serving successfully.
  • Real 2-GPU TP2 Qwen3.6-27B service loaded both DFlash and Domino checkpoints, captured CUDA Graphs, and completed the full benchmark.

Qwen3.6-27B benchmark

Setting: TP2, BF16, FA3 full attention, block size 16, thinking enabled, max output 4096, greedy decoding, one measured run per cell. Workloads are GSM8K-128, HumanEval-164, and MT-Bench-80.

Workload Concurrency DFlash tok/s Domino tok/s Throughput gain DFlash accept Domino accept Accept gain
GSM8K 1 340.25 437.40 +28.6% 6.777 9.287 +37.0%
GSM8K 8 1,855.85 2,314.51 +24.7% 6.860 9.304 +35.6%
GSM8K 32 3,094.09 3,794.29 +22.6% 6.840 9.286 +35.7%
HumanEval 1 363.60 399.53 +9.9% 6.422 7.407 +15.3%
HumanEval 8 2,048.48 2,206.18 +7.7% 6.451 7.433 +15.2%
HumanEval 32 3,452.99 3,733.11 +8.1% 6.496 7.422 +14.3%
MT-Bench 1 254.40 279.24 +9.8% 4.971 5.927 +19.2%
MT-Bench 8 1,442.02 1,517.15 +5.2% 5.027 5.966 +18.7%
MT-Bench 32 2,076.32 2,387.72 +15.0% 4.946 5.963 +20.5%

All 18 DFlash/Domino workload-concurrency cells completed successfully. Domino improves acceptance length and throughput in every measured cell.

Scope

This PR validates TP2 on the Qwen3 and Qwen3.6-27B model families, including eager and CUDA Graph paths. TP4/TP8, quantized LM-heads, and added-vocabulary shards are not claimed by this PR.

Related: #36899


CI States

Latest PR Test (Base): ✅ Run #34536644674
Latest PR Test (Extra): ❌ Run #34536644475
Latest PR Test (AMD ROCm 10): ❌ Run #34536644685

@Qiaolin-Yu Qiaolin-Yu self-assigned this Sep 9, 2026
@Qiaolin-Yu
Qiaolin-Yu enabled auto-merge (squash) September 11, 2026 22:49
@Qiaolin-Yu
Qiaolin-Yu disabled auto-merge September 12, 2026 01:50
@Qiaolin-Yu
Qiaolin-Yu merged commit e91c948 into sgl-project:main Sep 12, 2026
230 of 253 checks passed
@rwang5203 rwang5203 mentioned this pull request Sep 15, 2026
74 tasks
mqhc2020 pushed a commit to mqhc2020/sglang that referenced this pull request Sep 15, 2026
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants