feat: add optimized Domino rollout to DFlash V2 - #36899
Merged
Qiaolin-Yu merged 16 commits intoSep 9, 2026
Merged
Qiaolin-Yu merged 16 commits into
Qiaolin-Yu merged 16 commits into
Conversation
jianuo-huang
requested review from
Qiaolin-Yu,
Ying1123,
hnyls2002 and
merrymercy
as code owners
August 28, 2026 18:39
jianuo-huang
force-pushed
the
feat/domino-dflash2-upstream-rebased
branch
from
August 28, 2026 19:21
f081461 to
d63bc4c
Compare
Preallocate candidate-pool tensors for CUDA graph-safe rollout and keep the ATen GRUCell feedback path allocation-free.
This reverts commit 2fcc8e5.
Qiaolin-Yu
force-pushed
the
feat/domino-dflash2-upstream-rebased
branch
from
September 9, 2026 00:12
8669c17 to
ddaf5cb
Compare
Qiaolin-Yu
approved these changes
Sep 9, 2026
pllimax
added a commit
to pllimax/sglang
that referenced
this pull request
Sep 10, 2026
* origin/main: (27 commits) [Simulator] Give the OFFLINE/BLOCKING comparison tolerances real headroom (sgl-project#38732) [Config] msgspec.Struct for the config tier (sgl-project#38753) [AMD] ci: move the miles nightlies from rocm700 to rocm10 (sgl-project#37495) [Config] One writer for the declaration stash; no exception to the write seal (sgl-project#38752) docker(xpu): drop redundant setvars.sh from torch_memory_saver RUN (sgl-project#38665) [XPU][Fix] Pack device-pointer tables as uint64 to avoid 64-bit address overflow (sgl-project#35051) [CI] Temporarily disable GB300 tests (sgl-project#38770) [diffusion] feat: spill large tensors over shared memory like numpy arrays (sgl-project#38656) [diffusion] refactor: refactor utility ownership and document helper placement (sgl-project#38699) [NPU]Support GLM5.2 and FP8 DSA&Indexer kvcache for 950 (sgl-project#38250) [CI] Answer unrecognized slash commands instead of skipping silently (sgl-project#38736) [AMD] Parallelize aiter spec-decode KV index building over token blocks (sgl-project#37659) [DSv4] Integrate TRT-LLM DSv4 Attention for SM100/103 (sgl-project#30805) Add Opt-In for GLM-5.3 Flash breakable prefill CUDA graphs (sgl-project#38522) [CI] Install helion 1.4.0 for the KDA Helion kernel tests (sgl-project#38688) [Rust] Gate health on startup warmup completion (sgl-project#37994) [HiCache] Replace skip_lock_node_ids with a segment lock protocol (sgl-project#36848) feat: add optimized Domino rollout to DFlash V2 (sgl-project#36899) [CI] Add /run-full-ci and /run-extra-ci slash commands (sgl-project#38734) [Model] Support GLM-5.3 Flash NVFP4 loading (sgl-project#38621) ...
Leoyzen
pushed a commit
to Leoyzen/sglang
that referenced
this pull request
Sep 10, 2026
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com> (cherry picked from commit a84ffd1)
74 tasks
mqhc2020
pushed a commit
to mqhc2020/sglang
that referenced
this pull request
Sep 15, 2026
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds optimized Domino projector support to DFlash V2 for the public
Qwen3-8B-Domino-b16checkpoint. It is based on the latest upstreammainand supersedes the earlier Domino PRs #31328 and #36860.The target verification, continuous-prefix acceptance, bonus-token handling,
and KV-cache commit paths remain unchanged.
Included implementation
two-layer SiLU correction projector.
realized proposal back into the GRU.
fallback when graphs are unavailable.
GRUCellfor one-token feedback updates to avoid repeated cuDNNweight packing.
correction head.
K=0remains the full-vocabulary reference path.Scope
correctness boundary.
verification semantics.
in PR feat: support TP>1 Domino rollout for DFlash V2 #37069.
Validation
Unit and registered tests
Focused tests:
The focused suite covers config parsing, strict Domino projector weight loading,
shape validation, full-chain rollout parity, candidate-pool boundaries,
batched-vs-single-row parity, eager fallback, and CUDA Graph replay.
Serving validation setting
We validated the Qwen3-8B DFlash/Domino serving path with the following fixed setting:
Qwen/Qwen3-8BQwen3-8B-DFlash-b16andQwen3-8B-Domino-b16temperature=0,top_p=1,top_k=1max_running_requests=32,cuda_graph_max_bs=32,mem_fraction_static=0.70,page_size=1The serving validation completed successfully for both DFlash and Domino on this setting.
Numerical throughput and acceptance-length comparisons are reported separately in PR #37069.
Changes
This branch is rebased directly onto the current upstream
main; no unrelatedbenchmark or training scripts are included.
CI States
Latest PR Test (Base): ❌ Run #34294845632
Latest PR Test (Extra): ❌ Run #34294845436
Latest PR Test (AMD ROCm 10): ❌ Run #34294845773