Skip to content

feat: add optimized Domino rollout to DFlash V2 - #36899

Merged
Qiaolin-Yu merged 16 commits into
sgl-project:mainfrom
jianuo-huang:feat/domino-dflash2-upstream-rebased
Sep 9, 2026
Merged

Qiaolin-Yu merged 16 commits into
sgl-project:mainfrom
jianuo-huang:feat/domino-dflash2-upstream-rebased

Conversation

@jianuo-huang

@jianuo-huang jianuo-huang commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR adds optimized Domino projector support to DFlash V2 for the public
Qwen3-8B-Domino-b16 checkpoint. It is based on the latest upstream
main and supersedes the earlier Domino PRs #31328 and #36860.

The target verification, continuous-prefix acceptance, bonus-token handling,
and KV-cache commit paths remain unchanged.

Included implementation

  • Parse, validate, instantiate, and load the bias-free prefix GRU and the
    two-layer SiLU correction projector.
  • Generate the 15-token Domino proposal chain for block size 16, feeding each
    realized proposal back into the GRU.
  • Precompute all base LM-head logits in one step-major projection.
  • Capture the complete rollout in the DFlash draft CUDA Graph, with eager
    fallback when graphs are unavailable.
  • Use ATen GRUCell for one-token feedback updates to avoid repeated cuDNN
    weight packing.
  • Add a request-local, block-shared K=2048 candidate vocabulary pool for the
    correction head. K=0 remains the full-vocabulary reference path.

Scope

  • BF16, CUDA, TP=1, Qwen3-8B, DFlash block size 16.
  • Greedy decoding path; target verification remains full-vocabulary and is the
    correctness boundary.
  • The candidate pool is a draft-side approximation; it does not alter target
    verification semantics.
  • Non-greedy sampling and tensor-parallel Domino are covered by follow-up work
    in PR feat: support TP>1 Domino rollout for DFlash V2 #37069.

Validation

Unit and registered tests

Focused tests:

PYTHONPATH=python python3 -m pytest -q \
  test/registered/unit/spec/test_dflash_domino.py

CUDA_VISIBLE_DEVICES=0 PYTHONPATH=python3 -m pytest -q -s \
  test/registered/spec/dflash/test_dflash_domino.py

The focused suite covers config parsing, strict Domino projector weight loading,
shape validation, full-chain rollout parity, candidate-pool boundaries,
batched-vs-single-row parity, eager fallback, and CUDA Graph replay.

Serving validation setting

We validated the Qwen3-8B DFlash/Domino serving path with the following fixed setting:

  • Target: Qwen/Qwen3-8B
  • Drafts: Qwen3-8B-DFlash-b16 and Qwen3-8B-Domino-b16
  • CUDA, BF16, TP=1, FA3 attention backend
  • DFlash block size: 16 (15 proposed tokens)
  • Domino candidate pool: K=2048
  • Greedy sampling: temperature=0, top_p=1, top_k=1
  • Qwen3 tokenizer chat template, thinking disabled
  • Workloads: GSM8K-128, HumanEval-164, MT-Bench-80
  • Concurrency: 1, 8, 32; maximum generation length: 2048
  • Server: max_running_requests=32, cuda_graph_max_bs=32,
    mem_fraction_static=0.70, page_size=1
  • Warmup: 3 requests, warmup maximum generation 1024, first request skipped
  • One measured run per configuration, timeout 3600 seconds

The serving validation completed successfully for both DFlash and Domino on this setting.
Numerical throughput and acceptance-length comparisons are reported separately in PR #37069.

Changes

This branch is rebased directly onto the current upstream main; no unrelated
benchmark or training scripts are included.


CI States

Latest PR Test (Base): ❌ Run #34294845632
Latest PR Test (Extra): ❌ Run #34294845436
Latest PR Test (AMD ROCm 10): ❌ Run #34294845773

@jianuo-huang
jianuo-huang force-pushed the feat/domino-dflash2-upstream-rebased branch from f081461 to d63bc4c Compare August 28, 2026 19:21
Preallocate candidate-pool tensors for CUDA graph-safe rollout and keep the ATen GRUCell feedback path allocation-free.
@Qiaolin-Yu Qiaolin-Yu self-assigned this Sep 8, 2026
@Qiaolin-Yu Qiaolin-Yu changed the title feat: add optimized Domino rollout to DFlash V2 (rebased) feat: add optimized Domino rollout to DFlash V2 Sep 8, 2026
@Qiaolin-Yu
Qiaolin-Yu force-pushed the feat/domino-dflash2-upstream-rebased branch from 8669c17 to ddaf5cb Compare September 9, 2026 00:12
@Qiaolin-Yu
Qiaolin-Yu merged commit a84ffd1 into sgl-project:main Sep 9, 2026
222 of 257 checks passed
pllimax added a commit to pllimax/sglang that referenced this pull request Sep 10, 2026
* origin/main: (27 commits)
  [Simulator] Give the OFFLINE/BLOCKING comparison tolerances real headroom (sgl-project#38732)
  [Config] msgspec.Struct for the config tier (sgl-project#38753)
  [AMD] ci: move the miles nightlies from rocm700 to rocm10 (sgl-project#37495)
  [Config] One writer for the declaration stash; no exception to the write seal (sgl-project#38752)
  docker(xpu): drop redundant setvars.sh from torch_memory_saver RUN (sgl-project#38665)
  [XPU][Fix] Pack device-pointer tables as uint64 to avoid 64-bit address overflow (sgl-project#35051)
  [CI] Temporarily disable GB300 tests (sgl-project#38770)
  [diffusion] feat: spill large tensors over shared memory like numpy arrays (sgl-project#38656)
  [diffusion] refactor: refactor utility ownership and document helper placement (sgl-project#38699)
  [NPU]Support GLM5.2 and FP8 DSA&Indexer kvcache for 950 (sgl-project#38250)
  [CI] Answer unrecognized slash commands instead of skipping silently (sgl-project#38736)
  [AMD] Parallelize aiter spec-decode KV index building over token blocks (sgl-project#37659)
  [DSv4] Integrate TRT-LLM DSv4 Attention for SM100/103 (sgl-project#30805)
  Add Opt-In for GLM-5.3 Flash breakable prefill CUDA graphs (sgl-project#38522)
  [CI] Install helion 1.4.0 for the KDA Helion kernel tests (sgl-project#38688)
  [Rust] Gate health on startup warmup completion (sgl-project#37994)
  [HiCache] Replace skip_lock_node_ids with a segment lock protocol (sgl-project#36848)
  feat: add optimized Domino rollout to DFlash V2 (sgl-project#36899)
  [CI] Add /run-full-ci and /run-extra-ci slash commands (sgl-project#38734)
  [Model] Support GLM-5.3 Flash NVFP4 loading (sgl-project#38621)
  ...
Leoyzen pushed a commit to Leoyzen/sglang that referenced this pull request Sep 10, 2026
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
(cherry picked from commit a84ffd1)
@rwang5203 rwang5203 mentioned this pull request Sep 15, 2026
74 tasks
mqhc2020 pushed a commit to mqhc2020/sglang that referenced this pull request Sep 15, 2026
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants