Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
56a1572
[Spec] Add LiLiCorr: a candidate-lattice reranker for DFlash drafts
mrusanovsky Sep 1, 2026
fee4ff0
[Spec] LiLiCorr: address review feedback
mrusanovsky Sep 7, 2026
ce3d0e5
[Spec] LiLiCorr: sample the draft commit at T>0
mrusanovsky Sep 8, 2026
b22e740
[Spec] LiLiCorr: ruff-format the sampled commit
mrusanovsky Sep 9, 2026
7bf872f
[Spec] LiLiCorr: gate the sampled commit on the same device condition…
mrusanovsky Sep 9, 2026
3fba252
[Spec] LiLiCorr: move the registered tests into the kind/subsystem ta…
mrusanovsky Sep 9, 2026
6d7c37d
[Spec] LiLiCorr: take the device gate on the eager seam too
mrusanovsky Sep 14, 2026
1c2ee8a
[Spec] LiLiCorr: register the kernel tests to a suite that exists
mrusanovsky Sep 14, 2026
cfd26a7
Merge branch 'main' into lilicorr-support
nvpohanh Sep 15, 2026
c86e9ad
Merge branch 'main' into lilicorr-support
nvpohanh Sep 15, 2026
3735622
Merge branch 'main' into lilicorr-support
nvpohanh Sep 17, 2026
9a205d6
Merge branch 'main' into lilicorr-support
nvpohanh Sep 17, 2026
b3dddb8
Merge branch 'main' into lilicorr-support
nvpohanh Sep 18, 2026
e9359cd
[Spec] LiLiCorr: one module, fewer comments, tests in the right tree
mrusanovsky Sep 18, 2026
d03cf81
[Spec] LiLiCorr: drop torch.compile, keep the CUDA-graph fold
mrusanovsky Sep 18, 2026
83d8fa3
[Spec] LiLiCorr: condense the worker-stub test docstring too
mrusanovsky Sep 18, 2026
7e52d04
[Spec] LiLiCorr: close two docstring backticks the comment pass left …
mrusanovsky Sep 18, 2026
d249b5a
[Spec] LiLiCorr: skip the E2E test instead of failing when no drafter…
mrusanovsky Sep 21, 2026
675387b
Merge branch 'main' into lilicorr-support
mrusanovsky Sep 21, 2026
4ef3ef7
[spec] Reuse DFlash2's selector walk for the LiLiCorr commit
mrusanovsky Sep 22, 2026
5da5de0
[spec] Trim LiLiCorr comments and docstrings to the load-bearing floor
mrusanovsky Sep 22, 2026
2858437
[spec] Tighten LiLiCorr diagnostic messages
mrusanovsky Sep 22, 2026
33a23c0
[spec] Remove test_lilicorr.py and collapse onto one commit entry point
mrusanovsky Sep 22, 2026
ae2c705
[spec] Trim LiLiCorr comments and redundant kernel tests
kpham-sgl Sep 27, 2026
7db2f1e
[spec] Move LiLiCorr env vars into envs and register its kernels
kpham-sgl Sep 27, 2026
889d209
[spec] Move LiLiCorr kernel tests to extra-a
kpham-sgl Sep 27, 2026
ec0c8cd
[spec] Drop redundant LiLiCorr comments and docstrings
kpham-sgl Sep 27, 2026
1d56d95
[spec] Read LiLiCorr env at worker init; move conv check to DFlash lo…
kpham-sgl Sep 27, 2026
86541a2
[spec] Take LiLiCorr candidates from DFlash's candidate_topk
mrusanovsky Sep 27, 2026
a4f78fe
[spec] Compute the candidate log-partition without an fp32 copy of th…
mrusanovsky Sep 27, 2026
4a9d476
[spec] Drop the device-side row check in per_request_last_row
mrusanovsky Sep 27, 2026
f25d961
[spec] Pass the LiLiCorr attention bias to SDPA as a broadcast
mrusanovsky Sep 27, 2026
0987e1c
[spec] Take the candidate top-k and log-partition from one fused pass
mrusanovsky Sep 27, 2026
99a38c8
[spec] Trim LiLiCorr comments and docstrings
mrusanovsky Sep 27, 2026
ddabbde
[spec] Let LiLiCorr serve fewer draft tokens than its trained block
mrusanovsky Sep 27, 2026
f892d14
[spec] Move the conv coverage check back onto LiLiCorrDraftModel
kpham-sgl Sep 27, 2026
46af387
Merge remote-tracking branch 'origin/main' into kpham/pr37462-cleanup
kpham-sgl Sep 27, 2026
46ea22e
[spec] Use model_runner.tp_rank after the ps record was retired on main
kpham-sgl Sep 27, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions docs/docs/advanced_features/speculative_decoding.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -594,6 +594,40 @@ response = client.chat.completions.create(
print(response.choices[0].message.content)
```

### LiLiCorr: reranked DFlash drafts

A DFlash drafter is trained on per-position marginals rather than on the joint block distribution, so its drafted tokens are individually plausible yet jointly incoherent. **LiLiCorr** ([arXiv:2608.20530](https://arxiv.org/abs/2608.20530)) keeps the top-`k` candidates per block position, scores the whole lattice jointly with a small two-layer transformer, and commits a path through it greedily instead of taking the per-slot argmax. Verify is unchanged, so outputs remain distributionally identical to the target model's and only acceptance length moves.

LiLiCorr is **not a separate `--speculative-algorithm`**: it is a property of the draft checkpoint, the same way the DFlash candidate selector is. Serve it with `--speculative-algorithm DFLASH` and a checkpoint whose config declares `architectures: ["LiLiCorrDraftModel"]` and carries the `lilicorr_*` geometry fields inside `dflash_config`. The head is then built, loaded and folded into the draft CUDA graph automatically.

No LiLiCorr drafter is published yet, so the draft path below is a placeholder.

```bash Command
python3 -m sglang.launch_server \
--model Qwen/Qwen3-8B \
--attention-backend fa3 \
--speculative-algorithm DFLASH \
--speculative-draft-model-path <your-lilicorr-draft-model>
```

**Use a backend that opts out of the host `seq_lens` mirror.** `AttentionBackend.needs_cpu_seq_lens` defaults to `True`; `fa3`, `triton`, `trtllm_mha`, `trtllm_mla`, `flashmla` and `dsa` set it `False`, while `FlashInferAttnBackend` inherits the default. Under a backend that does not opt out the worker publishes `seq_lens_cpu` every step, which is a device-to-host sync per block. The flag is OR-ed over the target and draft runners, so both have to opt out. This is a property of DFlash decoding rather than of LiLiCorr, but it dominates the throughput of a 16-slot block. Backends also differ in attention numerics, so acceptance measured on different backends does not belong in one table.

A checkpoint carrying head weights but declaring `architectures: ["DFlashDraftModel"]` loads as plain DFlash and ignores the head, which reads as a small acceptance loss rather than an error. The reverse, declaring `LiLiCorrDraftModel` without complete head weights, is refused at load.

#### Sampling the committed path

Above temperature 0 the target samples, but a deterministic draft is paid `p(argmax)` per slot because verify is handed a point mass. `SGLANG_ENABLE_LILICORR_SAMPLING=1` draws each slot from `softmax(scores / T)` over that slot's candidates at the request's own temperature and publishes the proposal it drew from, so rejection sampling is paid the overlap of the two distributions instead. Verify stays exact. Greedy requests sharing the batch take the argmax and report a one-hot proposal.

Off by default; with it unset the captured body is `select` itself, so temperature-0 output is unchanged bit for bit. On Qwen3-8B it is worth about 6% acceptance and 6% end-to-end at temperature 1, at under 1% of the per-block wall.

**Only enable it on the folded path.** The eager fallback that serves the steps the draft CUDA graph cannot is correct but costs a large fraction of throughput, and because acceptance is identical either way that regression shows up only in tokens/s. `build_lilicorr_draft_sampler` warns with a reason when it declines to fold, and the worker warns once if a decode step lands on the eager head.

#### Drafters with grouped convolutions

LiLiCorr composes with the grouped dynamic convolutions that wrap each DFlash sublayer, and serving one needs no extra flag: the convolution belongs to the DFlash backbone, so `LiLiCorrDraftModel` inherits the builder and picks it up from `conv_kernel_size` and `conv_group_size` in `dflash_config`.

Both keys default to `0`, so a checkpoint whose config lost them builds no convolution modules and the backbone loader then drops every convolution tensor it cannot resolve. That is correct for rotary caches and the worst case here, because the draft would serve as its convolution-free parent at a lower but entirely believable acceptance length. The loader therefore refuses any checkpoint whose convolution tensors and built modules disagree, in either direction.

---

## Standalone Speculative Decoding (Small Draft Model)
Expand Down
2 changes: 2 additions & 0 deletions python/sglang/kernels/ops/speculative/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@
("topk1", "draft_topk1_postprocess"),
("ragged_verify_kernels", "pad_verify_lens_to_bucket"),
("ragged_verify_kernels", "build_qo_indptr"),
("lilicorr", "lilicorr_topk_lse"),
("lilicorr", "lilicorr_sample_path"),
("reject_sampling", "chain_speculative_sampling_triton"),
("row_argmax", "row_argmax"),
("reconstruct_tree", "reconstruct_indices_from_tree_mask_triton"),
Expand Down
235 changes: 235 additions & 0 deletions python/sglang/kernels/ops/speculative/lilicorr.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,235 @@
from __future__ import annotations

from typing import Tuple

import torch
import torch.nn.functional as F
import triton
import triton.language as tl

from sglang.kernels.ops.speculative.dflash import selector_walk_triton

# Launch geometry tuned on H100 for a ~150k vocabulary; TILE * TPP is the load block.
_TILE = 1024
_TILES_PER_PROGRAM = 8
_NUM_WARPS = 4

# One Triton lane group in the selector walk; the config refuses a wider head.
MAX_FUSED_CANDIDATE_TOPK = 16

_NEG = -3.0e38


@triton.jit
def _tiled_scan(
lp, # [N, V] logits (any float dtype)
tmax, # [N, T] fp32 per-tile max
pm, # [N, P] fp32 online-softmax running max
ps, # [N, P] fp32 online-softmax running sum
V,
T,
stride_lp_n,
stride_tmax_n,
stride_p_n,
TILE: tl.constexpr,
TPP: tl.constexpr,
):
neg = -3.0e38
row = tl.program_id(0)
grp = tl.program_id(1)

t0 = grp * TPP
tile_off = t0 + tl.arange(0, TPP)
offs = tile_off[:, None] * TILE + tl.arange(0, TILE)[None, :]
mask = (tile_off[:, None] < T) & (offs < V)

x = tl.load(lp + row * stride_lp_n + offs, mask=mask, other=neg)
x = x.to(tl.float32)

tm = tl.max(x, axis=1)
tl.store(tmax + row * stride_tmax_n + tile_off, tm, mask=tile_off < T)

m = tl.max(tm, axis=0)
s = tl.sum(tl.exp(x - m), axis=0)
s = tl.sum(s, axis=0)
tl.store(pm + row * stride_p_n + grp, m)
tl.store(ps + row * stride_p_n + grp, s)


@triton.jit
def _tiled_select(
lp, # [N, V] logits
tids, # [N, KT] int64 candidate tile ids (ascending)
ov, # [N, K] fp32 out vals (descending)
oi, # [N, K] int64 out ids
V,
stride_lp_n,
stride_tid_n,
stride_ov_n,
stride_oi_n,
TILE: tl.constexpr,
KT: tl.constexpr,
K: tl.constexpr,
NSEL: tl.constexpr,
):
neg = -3.0e38
row = tl.program_id(0)

tid = tl.load(tids + row * stride_tid_n + tl.arange(0, KT))
offs2 = tid[:, None] * TILE + tl.arange(0, TILE)[None, :]
mask2 = offs2 < V
x2 = tl.load(lp + row * stride_lp_n + offs2, mask=mask2, other=neg)

x = tl.reshape(x2.to(tl.float32), (NSEL,))
ids = tl.reshape(offs2, (NSEL,))
lane = tl.arange(0, NSEL)

for j in range(K):
mv = tl.max(x, axis=0)
mp = tl.argmax(x, axis=0)
mid = tl.sum(tl.where(lane == mp, ids, 0), axis=0)
tl.store(ov + row * stride_ov_n + j, mv)
tl.store(oi + row * stride_oi_n + j, mid)
x = tl.where(lane == mp, neg, x)


def _combine_lse(pm: torch.Tensor, ps: torch.Tensor) -> torch.Tensor:
gm = pm.max(dim=-1, keepdim=True).values
s = (ps * (pm - gm).exp()).sum(dim=-1)
return gm.squeeze(-1) + s.log()


def _topk_lse_torch(
logits: torch.Tensor, k: int
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
vals, ids = torch.topk(logits, k, dim=-1)
rowmax = vals[:, 0:1] # topk is sorted descending
sumexp = (logits - rowmax).exp().sum(dim=-1, dtype=torch.float32)
lse = rowmax.squeeze(-1).to(torch.float32) + sumexp.log()
return vals.float(), ids.to(torch.int64), lse


def lilicorr_topk_lse(

@kpham-sgl kpham-sgl Sep 27, 2026 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reuse DFlash2's compute_candidates (radix top-k + TP gather) and add only the lse, instead of a new top-k kernel and TP combine. It also keeps quantized lm_head support, which the eager path here breaks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done for the structure: compute_candidates body is now candidate_topk() in models/dflash.py and LiLiCorr calls it, so there is one projection (through quant_method), one TP gather, and our TP combine is gone.

One difference: with with_partition=True the top-k and lse come from a single fused pass instead of radix top-k plus a separate logsumexp. The second pass re-reads the [N, V] logits, which dominates at batch: 143 vs 680 us per step at c=32 (56 vs 69 us at c=1; H100, bf16, V=151,936, CUDA graph), and about -1.5% output throughput end to end at c=32. The selector keeps _radix_topk.

logits: torch.Tensor, k: int
) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
# Tile pre-selection is exact: a top-k element's tile has a max at least as large,
# so it is among the k largest-max tiles.
if not logits.is_cuda:
return _topk_lse_torch(logits, k)

logits = logits.contiguous()
n, V = int(logits.shape[0]), int(logits.shape[1])
K = int(k)
T = (V + _TILE - 1) // _TILE
if min(K, T) & (min(K, T) - 1):
return _topk_lse_torch(logits, k)
P = (T + _TILES_PER_PROGRAM - 1) // _TILES_PER_PROGRAM
device = logits.device

tmax = torch.full((n, T), _NEG, dtype=torch.float32, device=device)
pm = torch.full((n, P), _NEG, dtype=torch.float32, device=device)
ps = torch.zeros((n, P), dtype=torch.float32, device=device)
_tiled_scan[(n, P)](
logits,
tmax,
pm,
ps,
V,
T,
logits.stride(0),
tmax.stride(0),
pm.stride(0),
TILE=_TILE,
TPP=_TILES_PER_PROGRAM,
num_warps=_NUM_WARPS,
)

kt = min(K, T)
tids = torch.topk(tmax, kt, dim=-1).indices
Comment thread
nvpohanh marked this conversation as resolved.
Resolved
tids = torch.sort(tids, dim=-1).values.to(torch.int64)

ov = torch.empty((n, K), dtype=torch.float32, device=device)
oi = torch.empty((n, K), dtype=torch.int64, device=device)
_tiled_select[(n,)](
logits,
tids,
ov,
oi,
V,
logits.stride(0),
tids.stride(0),
ov.stride(0),
oi.stride(0),
TILE=_TILE,
KT=kt,
K=K,
NSEL=kt * _TILE,
num_warps=_NUM_WARPS,
)
return ov, oi, _combine_lse(pm, ps)


def _lattice_scores(log_start: torch.Tensor, log_pair: torch.Tensor) -> torch.Tensor:
# The walk reads only scores[:, 0, 0, :] at slot 0, so broadcasting is sound.
topk = int(log_start.shape[-1])
start = log_start.float()[:, None, None, :].expand(-1, 1, topk, topk)
return torch.cat([start, log_pair.float()], dim=1)


def _selector_walk_torch(
*,
candidate_ids: torch.Tensor,
scores: torch.Tensor,
uniforms: torch.Tensor,
temperatures: torch.Tensor,
greedy_mask: torch.Tensor,
) -> Tuple[torch.Tensor, torch.Tensor]:
_, num_slots, topk = candidate_ids.shape
temps = temperatures.view(-1, 1).to(torch.float32)
greedy = greedy_mask.view(-1)
previous = torch.zeros(greedy.shape[0], dtype=torch.int64, device=scores.device)
tokens, q_rows = [], []
for slot in range(num_slots):
node = torch.gather(
scores[:, slot].float(), 1, previous.view(-1, 1, 1).expand(-1, 1, topk)
).squeeze(1)
probs = torch.softmax(node / temps, dim=-1)
sampled = (
uniforms[:, slot : slot + 1]
.ge(probs.cumsum(dim=-1))
.sum(dim=-1)
.clamp_max(topk - 1)
)
previous = torch.where(greedy, node.argmax(dim=-1), sampled)
q_rows.append(
torch.where(
greedy.unsqueeze(-1),
F.one_hot(previous, topk).to(torch.float32),
probs,
)
)
tokens.append(
torch.gather(candidate_ids[:, slot], 1, previous.view(-1, 1)).squeeze(1)
)
return torch.stack(tokens, dim=-1).to(torch.int64), torch.stack(q_rows, dim=1)


def lilicorr_sample_path(
log_start: torch.Tensor,
log_pair: torch.Tensor,
candidate_tokens: torch.Tensor,
uniforms: torch.Tensor,
temperatures: torch.Tensor,
greedy_mask: torch.Tensor,
) -> Tuple[torch.Tensor, torch.Tensor]:
# No log-prob prior in the proposal: the head was trained without one. Greedy rows
# report a point mass so min(1, p/q) stays the right acceptance test.
scores = _lattice_scores(log_start, log_pair)
walk = selector_walk_triton if scores.is_cuda else _selector_walk_torch
return walk(
candidate_ids=candidate_tokens,
scores=scores,
uniforms=uniforms,
temperatures=temperatures,
greedy_mask=greedy_mask,
)
2 changes: 2 additions & 0 deletions python/sglang/srt/environ.py
Original file line number Diff line number Diff line change
Expand Up @@ -1366,6 +1366,8 @@ class Envs:
SGLANG_OPT_FUSED_KDA_VERIFY = EnvBool(False)
# A/B: keep the DFLASH draft greedy head eager (not folded in-graph).
SGLANG_DFLASH_EAGER_DRAFT_SAMPLER = EnvBool(False)
SGLANG_ENABLE_LILICORR_SAMPLING = EnvBool(False)
SGLANG_LILICORR_REQUIRE_SAMPLING = EnvBool(False)
SGLANG_RAGGED_VERIFY_MODE = EnvStr("static")
SGLANG_TEST_RAGGED_VERIFY_FORCE_UNIFORM_CAPTURE = EnvBool(False)
# Skip draft_extend while adaptive spec is at steps=0 (drafting disabled).
Expand Down
Loading
Loading