[Spec] LiLiCorr: named head MLPs and quantized head linears by mrusanovsky · Pull Request #42057 · sgl-project/sglang

mrusanovsky · 2026-10-01T12:14:26Z

Motivation

#37462 added the LiLiCorr draft head. Its
checkpoints name the head's MLP weights by nn.Sequential index (feature_mlp.1,
layers.N.mlp.0). ModelOpt now exports descriptive names instead (feature_mlp.up_proj,
layers.N.mlp.down_proj, see NVIDIA/Model-Optimizer#2601),
the layout vLLM loads, so one checkpoint serves on both engines.

This PR:

loads the new names (no checkpoint with the old ones is published);
loads quantized LiLiCorr drafters (e.g. ModelOpt NVFP4). On main they fail to load: the head's
linears and the DFlash fc are plain nn.Linear that ignore quant_config, so packed weights
and their scales have nowhere to go.

Modifications

models/lilicorr.py

Names: the head's MLPs were nn.Sequential; they are now up_proj/down_proj modules, with the
input LayerNorm as a separate feature_norm.
Quantization: the head's linears were nn.Linear; they are now ReplicatedLinear with
quant_config and a module prefix, so a checkpoint's quantized layers and its exclude list are
honored. Two kinds stay nn.Linear: factor_input_proj/out_head/in_head, whose raw weights
are fused into inference buffers at load, and feature_mlp, whose five inputs fit no quantized
kernel.
Inputs to the head's linears are flattened to 2D, which quantized linear methods require.
Loading check: a missing input_scale is allowed only for NVFP4 W4A16, which registers one it
never reads.
An NVFP4 head that runs on Marlin (W4A16, or W4A4 when --fp4-gemm-backend resolves to
marlin, the default below SM100) is rejected at tp_size > 1: every rank runs the replicated
head, and Marlin's atomic-add reduce does not give the same draft on each rank.
Dtype and sizes are read from slot_embedding and input_size/output_size, which stay valid
when a weight is packed to uint8.

models/dflash.py

Only some DFlash drafts gave their layers a module prefix and built fc with quant_config. A
supports_quantization class flag, set by LiLiCorr, now opts a draft into that path; other DFlash
drafts are unchanged.

speculative/lilicorr_utils.py

Reads sizes from input_size/output_size, for the same reason as above.
At tp_size > 1 the head is folded into the draft CUDA graph instead of running eager. Its
pre-projected token table is replicated on every rank, as DFlash2's selector codebooks are: each
rank projects its vocab shard and the shards are all-gathered once at load, so candidates, which
carry global ids, index it directly and the per-step path is the same as at tp_size 1.

Accuracy Tests

Qwen3-8B target, one H100, gsm8k (1319), greedy, fa3, draft_length 15, one server at a time on
the node.

draft	tree	concurrency 1: acceptance length	tok/s	concurrency 4: acceptance length	tok/s
bf16, old names	main	6.9953	730.8	6.9995	2613
bf16, new names	this PR	6.9953, bit-identical	731.0	6.9762	2607
W4A16 NVFP4	main	fails to load: `fc.weight shape mismatch`
W4A16 NVFP4	this PR	6.9672	716.5	6.9678	2495
bf16, target `--quantization fp8`	main	6.9822	953.1	6.9662	3031
bf16, target `--quantization fp8`	this PR	6.9812	951.9	6.9449	3053

bf16 rows: at concurrency 1 the renamed checkpoint gives the same tokens as main. Above that, 2D inputs
change bf16 rounding in the head; acceptance moves by ±0.02 either way (concurrency 2 / 4 / 8:
main 6.992 / 7.000 / 6.994, this PR 7.005 / 6.976 / 6.981).
W4A16: the newly supported case, a drafter in ModelOpt's W4A16_NVFP4 format. Main stops at
the first packed weight, fc (uint8 (4096, 10240) against the (4096, 20480) nn.Linear).
--quantization fp8 rows: sglang gives the draft the target's quantization by default. On main
that reached only the draft's decoder layers; it now also reaches the head and fc, with acceptance within 0.001.
tp_size 2, bf16, eager head (main) vs head in the draft graph (this PR): concurrency 1
7.0196 / 1016 tok/s vs 7.0211 / 1040 tok/s; concurrency 4 6.9783 / 3144 vs 6.9860 / 3284;
concurrency 64 6.9439 / 14998 vs 6.9684 / 15745. Target --quantization fp8, concurrency 1:
main 7.0174, this PR 7.0203. W4A16 is rejected at load.
tp_size 4, bf16, this PR: concurrency 1 7.0154 / 1322 tok/s; concurrency 64 6.9784 / 19305.

Each row is one run of ModelOpt's examples/specdec_bench (acceptance length is its
Block_Weighted_AL), with concurrency C:

python run.py --engine SGLANG --model_dir Qwen/Qwen3-8B --tokenizer Qwen/Qwen3-8B \
  --speculative_algorithm DFLASH --draft_model_dir $DRAFT --draft_length 15 \
  --dataset gsm8k --dataset_path gsm8k --num_requests 1319 --output_length 2048 \
  --temperature 0.0 --concurrency $C --tp_size 1 --runtime_params runtime.yaml
# runtime.yaml: chat_template_args: {enable_thinking: false}
#               engine_args: {attention_backend: fa3}   # plus quantization: fp8 for the fp8 rows

It starts sgl.Engine with the equivalent of:

python -m sglang.launch_server --model-path Qwen/Qwen3-8B --tp-size 1 \
  --speculative-algorithm DFLASH --speculative-draft-model-path $DRAFT \
  --speculative-num-steps 15 --speculative-eagle-topk 1 --speculative-num-draft-tokens 16 \
  --attention-backend fa3 --mem-fraction-static 0.8 --disable-radix-cache \
  --max-running-requests $C --cuda-graph-max-bs-decode $C --cuda-graph-max-bs-prefill $C
  # plus --quantization fp8 for the fp8 rows

The drafts are not published. All three are one trained Qwen3-8B LiLiCorr draft: as trained,
with the head's MLP keys renamed, and packed by a short script into ModelOpt's W4A16_NVFP4
format.

Checklist

Format your code according to Format code with pre-commit
Provide accuracy and speed benchmark results

CI States

Latest PR Test (Base): ❌ Run #37250210108
Latest PR Test (Extra): ❌ Run #37250210119
Latest PR Test (AMD ROCm 10): ❌ Run #37250209913

Name the head's MLPs up_proj/down_proj with a separate feature_norm, the layout ModelOpt now exports, and keep loading the previous nn.Sequential names. Route the head's projections through quant_config so a quantized head loads; factor_input_proj/out_head/in_head stay dense since their raw weights are fused into inference buffers, and feature_mlp since its five inputs fit no quantized kernel. Give every DFlash draft's layers their module prefix and a quant-aware fc, so exclude lists resolve and a quantized fc loads for any draft.

nvpohanh · 2026-10-01T12:36:07Z

cc @elvischenv @rystewart-nvidia for vis
@kpham-sgl for review

nvpohanh

[by Claude Code] Reviewed at 9855341: two inline findings on the quantized LiLiCorr head path.

…ide W4A16 Quantized linear methods take [tokens, features] input, and pass_hidden is a non-contiguous slice once the batch exceeds one request. Only NVFP4 W4A16 registers an input_scale it never reads, so only it may skip that tensor.

kpham-sgl · 2026-10-01T16:51:48Z

Please remove this newly added test. Add any important tests back to test_dflash_lilicorr.py

Removed. test_dflash_lilicorr.py is an end-to-end server test that stays disabled until a
checkpoint is published, so I did not add unit tests to it.

kpham-sgl · 2026-10-01T16:52:23Z

            for name, weight in weights:
                stripped = name[len("model.") :] if name.startswith("model.") else name
                if stripped.startswith("lilicorr."):
+                    # Older exports index the head MLPs as nn.Sequential.


Do we really need to support this mapping

No, no LiLiCorr checkpoint with the old names is published, so I removed the mapping.

kpham-sgl · 2026-10-01T16:53:57Z

Seems like all the changes in this file just remove a special path for self.is_nemotron_35_draft. Please confirm if this is intended @mrusanovsky

That was wider than needed. With a target --quantization fp8, which the draft
inherits, every DFlash draft's fc became FP8 too. Now only drafts that set
supports_quantization (LiLiCorr) take that path; other DFlash drafts are unchanged from main.

kpham-sgl · 2026-10-01T16:54:44Z

+    return out.view(*x.shape[:-1], -1)
+
+
+class LiLiCorrMLP(nn.Module):


Q: is LiliCorr module not intended to be sharded under TP?

Correct, it is not sharded. The head is replicated on every rank, as on main, where it was
nn.Linear; this PR only swaps in ReplicatedLinear so it can be quantized. At tp_size > 1,
LiLiCorr drafting also falls back to eager mode, as on main, and this PR does not change that.
One addition: a W4A16 head now fails at load when tp_size > 1, because its Marlin kernel does not
give bit-identical results on every rank, which a replicated head needs.

At tp_size > 1,
LiLiCorr drafting also falls back to eager mode, as on main, and this PR does not change that.

We should fix this

Done: at tp_size > 1 the LiLiCorr head now runs inside the draft CUDA graph, as at TP=1, instead of falling back to eager. Its token table is replicated on every rank (built per vocab shard and all-gathered once at load), as DFlash2's selector codebooks are, so the per-step path is the same as at TP=1. Verified at TP=2 and TP=4: same acceptance, greedy and sampled.

…uantization support

nvpohanh

[by Claude Code] Re-reviewed at 5c6efdb. The two earlier inline findings are fixed at this head. New findings: two inline comments, plus the two PR-level comments below.

[by Claude Code] Severity: style | Confidence: High

Issue

The description no longer matches the code. It says old nn.Sequential names "are renamed at load", but e201f48 removed that mapping, so a checkpoint with feature_mlp.1 or layers.N.mlp.0 now fails check_head_weight_coverage. It also says every DFlash draft now gets layer prefixes and a quant-aware fc, but 9e96f49 limits that to LiLiCorr and Nemotron 3.5 drafts. It does not mention that a W4A16 head now requires tp_size 1.

Fix

Update Motivation item 1 and the models/lilicorr.py and models/dflash.py bullets, and add the W4A16 TP limit.

[by Claude Code] Severity: style | Confidence: High

Issue

The accuracy and speed table names the target, GPU, and dataset, but gives no commands. A reviewer cannot rerun a row or see which speculative flags "draft_length 15" means.

Fix

Add the exact server launch command for each row (the --speculative-* flags, plus --quantization fp8 for the fp8 rows) and the gsm8k eval command. The drafts are not published, so also say how the bf16 and W4A16 drafts were produced, for example the ModelOpt export command.

nvpohanh · 2026-10-02T15:35:20Z

/rerun-test test/registered/e2e/models/test_nvidia_nemotron_3_nano.py test/registered/core/test_basic_sanity_dflash.py test/registered/e2e/models/test_glm53_flash_b200.py test/registered/spec/dflash/test_dflash.py test/registered/e2e/speculative/test_dflash_domino.py test/registered/spec/test_gemma4_dflash_31b_extra.py test/registered/spec/test_spec_mixed_chunk.py test/registered/spec/dflash/test_muse_glimmer_dflash_assistant_gsm8k.py test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py test/registered/unit/spec/test_dflash_logits.py test/registered/unit/spec/test_dflash_domino.py test/registered/kernels/ops/speculative/test_lilicorr_cuda.py test/registered/unit/spec/test_dflash_overlap_hostsync.py test/registered/unit/spec/test_dflash_extra_buffer_lazy.py test/registered/kernels/ops/speculative/test_dflash_domino.py test/registered/unit/spec/test_oot_dflash_hooks.py test/registered/unit/model_executor/model_runner_components/test_spec_aux_hidden_state.py test/registered/unit/layer_boundary/test_glm5_next_dflash_capture.py

github-actions · 2026-10-02T15:35:46Z

Results for /rerun-test test/registered/e2e/models/test_nvidia_nemotron_3_nano.py test/registered/core/test_basic_sanity_dflash.py test/registered/e2e/models/test_glm53_flash_b200.py test/registered/spec/dflash/test_dflash.py test/registered/e2e/speculative/test_dflash_domino.py test/registered/spec/test_gemma4_dflash_31b_extra.py test/registered/spec/test_spec_mixed_chunk.py test/registered/spec/dflash/test_muse_glimmer_dflash_assistant_gsm8k.py test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py test/registered/unit/spec/test_dflash_logits.py test/registered/unit/spec/test_dflash_domino.py test/registered/kernels/ops/speculative/test_lilicorr_cuda.py test/registered/unit/spec/test_dflash_overlap_hostsync.py test/registered/unit/spec/test_dflash_extra_buffer_lazy.py test/registered/kernels/ops/speculative/test_dflash_domino.py test/registered/unit/spec/test_oot_dflash_hooks.py test/registered/unit/model_executor/model_runner_components/test_spec_aux_hidden_state.py test/registered/unit/layer_boundary/test_glm5_next_dflash_capture.py:

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_nvidia_nemotron_3_nano.py
cd test/ && python3 registered/e2e/models/test_glm53_flash_b200.py

🚀 1-gpu-5090 (4 tests): ✅ View workflow run

cd test/ && python3 registered/core/test_basic_sanity_dflash.py
cd test/ && python3 registered/spec/dflash/test_dflash.py
cd test/ && python3 registered/unit/spec/test_dflash_overlap_hostsync.py
cd test/ && python3 registered/unit/spec/test_dflash_extra_buffer_lazy.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/speculative/test_dflash_domino.py
cd test/ && python3 registered/spec/test_gemma4_dflash_31b_extra.py

🚀 1-gpu-h100 (5 tests): ✅ View workflow run

cd test/ && python3 registered/spec/test_spec_mixed_chunk.py
cd test/ && python3 registered/spec/dflash/test_muse_glimmer_dflash_assistant_gsm8k.py
cd test/ && python3 registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py
cd test/ && python3 registered/kernels/ops/speculative/test_lilicorr_cuda.py
cd test/ && python3 registered/kernels/ops/speculative/test_dflash_domino.py

🚀 ubuntu-latest (5 tests): ✅ View workflow run

cd test/ && python3 registered/unit/spec/test_dflash_logits.py
cd test/ && python3 registered/unit/spec/test_dflash_domino.py
cd test/ && python3 registered/unit/spec/test_oot_dflash_hooks.py
cd test/ && python3 registered/unit/model_executor/model_runner_components/test_spec_aux_hidden_state.py
cd test/ && python3 registered/unit/layer_boundary/test_glm5_next_dflash_capture.py

nvpohanh · 2026-10-05T01:08:32Z

/rerun-test test/registered/e2e/models/test_nvidia_nemotron_3_nano.py test/registered/core/test_basic_sanity_dflash.py test/registered/e2e/models/test_glm53_flash_b200.py test/registered/spec/dflash/test_dflash.py test/registered/e2e/speculative/test_dflash_domino.py test/registered/spec/test_gemma4_dflash_31b_extra.py test/registered/spec/test_spec_mixed_chunk.py test/registered/spec/dflash/test_muse_glimmer_dflash_assistant_gsm8k.py test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py test/registered/unit/spec/test_dflash_logits.py test/registered/unit/spec/test_dflash_domino.py test/registered/kernels/ops/speculative/test_lilicorr_cuda.py test/registered/unit/spec/test_dflash_overlap_hostsync.py test/registered/unit/spec/test_dflash_extra_buffer_lazy.py test/registered/kernels/ops/speculative/test_dflash_domino.py test/registered/unit/spec/test_oot_dflash_hooks.py test/registered/unit/model_executor/model_runner_components/test_spec_aux_hidden_state.py test/registered/unit/layer_boundary/test_glm5_next_dflash_capture.py

github-actions · 2026-10-05T01:08:58Z

Results for /rerun-test test/registered/e2e/models/test_nvidia_nemotron_3_nano.py test/registered/core/test_basic_sanity_dflash.py test/registered/e2e/models/test_glm53_flash_b200.py test/registered/spec/dflash/test_dflash.py test/registered/e2e/speculative/test_dflash_domino.py test/registered/spec/test_gemma4_dflash_31b_extra.py test/registered/spec/test_spec_mixed_chunk.py test/registered/spec/dflash/test_muse_glimmer_dflash_assistant_gsm8k.py test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py test/registered/unit/spec/test_dflash_logits.py test/registered/unit/spec/test_dflash_domino.py test/registered/kernels/ops/speculative/test_lilicorr_cuda.py test/registered/unit/spec/test_dflash_overlap_hostsync.py test/registered/unit/spec/test_dflash_extra_buffer_lazy.py test/registered/kernels/ops/speculative/test_dflash_domino.py test/registered/unit/spec/test_oot_dflash_hooks.py test/registered/unit/model_executor/model_runner_components/test_spec_aux_hidden_state.py test/registered/unit/layer_boundary/test_glm5_next_dflash_capture.py:

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_nvidia_nemotron_3_nano.py
cd test/ && python3 registered/e2e/models/test_glm53_flash_b200.py

🚀 1-gpu-5090 (4 tests): ✅ View workflow run

cd test/ && python3 registered/core/test_basic_sanity_dflash.py
cd test/ && python3 registered/spec/dflash/test_dflash.py
cd test/ && python3 registered/unit/spec/test_dflash_overlap_hostsync.py
cd test/ && python3 registered/unit/spec/test_dflash_extra_buffer_lazy.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/speculative/test_dflash_domino.py
cd test/ && python3 registered/spec/test_gemma4_dflash_31b_extra.py

🚀 1-gpu-h100 (5 tests): ❌ View workflow run

cd test/ && python3 registered/spec/test_spec_mixed_chunk.py
cd test/ && python3 registered/spec/dflash/test_muse_glimmer_dflash_assistant_gsm8k.py
cd test/ && python3 registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py
cd test/ && python3 registered/kernels/ops/speculative/test_lilicorr_cuda.py
cd test/ && python3 registered/kernels/ops/speculative/test_dflash_domino.py

🚀 ubuntu-latest (5 tests): ✅ View workflow run

cd test/ && python3 registered/unit/spec/test_dflash_logits.py
cd test/ && python3 registered/unit/spec/test_dflash_domino.py
cd test/ && python3 registered/unit/spec/test_oot_dflash_hooks.py
cd test/ && python3 registered/unit/model_executor/model_runner_components/test_spec_aux_hidden_state.py
cd test/ && python3 registered/unit/layer_boundary/test_glm5_next_dflash_capture.py

nvpohanh · 2026-10-05T02:16:35Z

/rerun-test test/registered/e2e/models/test_nvidia_nemotron_3_nano.py test/registered/core/test_basic_sanity_dflash.py test/registered/e2e/models/test_glm53_flash_b200.py test/registered/spec/dflash/test_dflash.py test/registered/e2e/speculative/test_dflash_domino.py test/registered/spec/test_gemma4_dflash_31b_extra.py test/registered/spec/test_spec_mixed_chunk.py test/registered/spec/dflash/test_muse_glimmer_dflash_assistant_gsm8k.py test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py test/registered/unit/spec/test_dflash_logits.py test/registered/unit/spec/test_dflash_domino.py test/registered/kernels/ops/speculative/test_lilicorr_cuda.py test/registered/unit/spec/test_dflash_overlap_hostsync.py test/registered/unit/spec/test_dflash_extra_buffer_lazy.py test/registered/kernels/ops/speculative/test_dflash_domino.py test/registered/unit/spec/test_oot_dflash_hooks.py test/registered/unit/model_executor/model_runner_components/test_spec_aux_hidden_state.py test/registered/unit/layer_boundary/test_glm5_next_dflash_capture.py

github-actions · 2026-10-05T02:17:04Z

Results for /rerun-test test/registered/e2e/models/test_nvidia_nemotron_3_nano.py test/registered/core/test_basic_sanity_dflash.py test/registered/e2e/models/test_glm53_flash_b200.py test/registered/spec/dflash/test_dflash.py test/registered/e2e/speculative/test_dflash_domino.py test/registered/spec/test_gemma4_dflash_31b_extra.py test/registered/spec/test_spec_mixed_chunk.py test/registered/spec/dflash/test_muse_glimmer_dflash_assistant_gsm8k.py test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py test/registered/unit/spec/test_dflash_logits.py test/registered/unit/spec/test_dflash_domino.py test/registered/kernels/ops/speculative/test_lilicorr_cuda.py test/registered/unit/spec/test_dflash_overlap_hostsync.py test/registered/unit/spec/test_dflash_extra_buffer_lazy.py test/registered/kernels/ops/speculative/test_dflash_domino.py test/registered/unit/spec/test_oot_dflash_hooks.py test/registered/unit/model_executor/model_runner_components/test_spec_aux_hidden_state.py test/registered/unit/layer_boundary/test_glm5_next_dflash_capture.py:

🚀 4-gpu-b200 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/models/test_nvidia_nemotron_3_nano.py
cd test/ && python3 registered/e2e/models/test_glm53_flash_b200.py

🚀 1-gpu-5090 (4 tests): ✅ View workflow run

cd test/ && python3 registered/core/test_basic_sanity_dflash.py
cd test/ && python3 registered/spec/dflash/test_dflash.py
cd test/ && python3 registered/unit/spec/test_dflash_overlap_hostsync.py
cd test/ && python3 registered/unit/spec/test_dflash_extra_buffer_lazy.py

🚀 2-gpu-h100 (2 tests): ✅ View workflow run

cd test/ && python3 registered/e2e/speculative/test_dflash_domino.py
cd test/ && python3 registered/spec/test_gemma4_dflash_31b_extra.py

🚀 1-gpu-h100 (5 tests): ❌ View workflow run

cd test/ && python3 registered/spec/test_spec_mixed_chunk.py
cd test/ && python3 registered/spec/dflash/test_muse_glimmer_dflash_assistant_gsm8k.py
cd test/ && python3 registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py
cd test/ && python3 registered/kernels/ops/speculative/test_lilicorr_cuda.py
cd test/ && python3 registered/kernels/ops/speculative/test_dflash_domino.py

🚀 ubuntu-latest (5 tests): ✅ View workflow run

cd test/ && python3 registered/unit/spec/test_dflash_logits.py
cd test/ && python3 registered/unit/spec/test_dflash_domino.py
cd test/ && python3 registered/unit/spec/test_oot_dflash_hooks.py
cd test/ && python3 registered/unit/model_executor/model_runner_components/test_spec_aux_hidden_state.py
cd test/ && python3 registered/unit/layer_boundary/test_glm5_next_dflash_capture.py

nvpohanh · 2026-10-05T08:16:01Z

test_pcg_with_speculative_decoding_dflash.py is not a breakage caused by this PR. It is a known failure.

mrusanovsky requested review from Qiaolin-Yu, Ying1123, hnyls2002 and merrymercy as code owners October 1, 2026 12:14

nvpohanh reviewed Oct 1, 2026

View reviewed changes

Comment thread python/sglang/srt/models/lilicorr.py Outdated

Comment thread python/sglang/srt/models/lilicorr.py Outdated

github-actions Bot added the quant LLM Quantization label Oct 1, 2026

Merge branch 'main' into lilicorr-named-quant-head

569a0b2

kpham-sgl reviewed Oct 1, 2026

View reviewed changes

mrusanovsky added 3 commits October 1, 2026 12:29

[Spec] LiLiCorr: drop the old head-name mapping and the head unit test

e201f48

[Spec] DFlash: keep main's fc and layer prefixes for drafts without q…

9e96f49

…uantization support

[Spec] LiLiCorr: reject a W4A16 head at tp_size > 1

5c6efdb

nvpohanh reviewed Oct 2, 2026

View reviewed changes

Comment thread python/sglang/srt/models/lilicorr.py

Comment thread python/sglang/srt/models/lilicorr.py Outdated

[Spec] LiLiCorr: reject any NVFP4 Marlin head at tp_size > 1

bfeae57

mrusanovsky and others added 3 commits October 3, 2026 06:58

[Spec] LiLiCorr: fold the head into the draft cuda graph at tp_size > 1

4623904

Merge branch 'main' into lilicorr-named-quant-head

161db1e

Merge branch 'main' into lilicorr-named-quant-head

e792adb

kpham-sgl approved these changes Oct 5, 2026

View reviewed changes

kpham-sgl merged commit 26a1377 into sgl-project:main Oct 5, 2026
100 of 116 checks passed

twu3202 mentioned this pull request Oct 5, 2026

[Spec] Build DFlash draft layers under their checkpoint names and refuse tensors the draft would drop #40884

Open

3 of 5 tasks

bingxche mentioned this pull request Oct 5, 2026

[CI Monitor] Daily Report - 2026-10-05 bingxche/sglang-ci-bot#216

Open

bingxche mentioned this pull request Oct 6, 2026

[CI Monitor] Daily Report - 2026-10-06 bingxche/sglang-ci-bot#217

Open

		return out.view(*x.shape[:-1], -1)


		class LiLiCorrMLP(nn.Module):

Conversation

mrusanovsky commented Oct 1, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Accuracy Tests

Checklist

CI States

Uh oh!

nvpohanh commented Oct 1, 2026

Uh oh!

nvpohanh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nvpohanh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nvpohanh commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvpohanh commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvpohanh commented Oct 5, 2026

Uh oh!

github-actions Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nvpohanh commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mrusanovsky commented Oct 1, 2026 •

edited by github-actions Bot

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 5, 2026 •

edited

Loading

github-actions Bot commented Oct 5, 2026 •

edited

Loading