spec: support speculators-format checkpoints for DSpark by wjinxu · Pull Request #26275 · ggml-org/llama.cpp

wjinxu · 2026-07-29T13:08:40Z

Overview

Follow-up to #25173. This adds support for DSpark drafts exported in the speculators format (SpecForge / RedHat), which vLLM gained in vllm-project/vllm#47093.

Format differences

These checkpoints differ from the dense DeepSpec checkpoints in same ways:

The speculators config uses sample_from_anchor to declare the block layout: true selects anchor-first, matching dense DeepSpec where every slot predicts a token; false selects DFlash-style 1+N infilling, where the anchor slot contains the bonus token and 1 + n_max slots produce n_max draft tokens. A missing field defaults to false, since legacy exports only use the 1+N layout.

A checkpoint may use a pruned draft output vocabulary (draft_vocab_size < vocab_size) together with a draft-to-target (d2t) remapping table.

Verification

makora-ai/gemma4-26b-a4b-dspark

category       base_avg_pred_t/s  spec_avg_pred_t/s  decode_speedup  base_avg_latency  spec_avg_latency  latency_speedup  accept_rate
  -------------  -----------------  -----------------  --------------  ----------------  ----------------  ---------------  -----------
  coding         82.59              145.53             1.76x           32.130s           18.790s           1.71x            0.5777
  humanities     83.68              89.31              1.07x           16.266s           17.955s           0.91x            0.2816
  math           83.87              96.88              1.16x           13.976s           14.484s           0.96x            0.3016
  qa             84.36              89.16              1.06x           10.334s           9.125s            1.13x            0.3173
  rag            83.29              104.72             1.26x           11.497s           10.121s           1.14x            0.3347
  reasoning      83.83              88.92              1.06x           15.457s           17.113s           0.90x            0.2790
  stem           83.95              89.88              1.07x           13.436s           16.299s           0.82x            0.2823
  writing        81.62              81.47              1.00x           32.122s           31.942s           1.01x            0.2379
  multilingual   84.05              91.50              1.09x           11.673s           10.677s           1.09x            0.3052
  summarization  84.51              92.09              1.09x           6.422s            5.352s            1.20x            0.2873
  roleplay       83.12              83.53              1.00x           33.010s           32.700s           1.01x            0.2678
  overall        83.53              95.73              1.15x           17.848s           16.778s           1.06x            0.3079

RedHatAI/gemma-4-31B-it-speculator.dspark

Comparison: baseline=baseline.json speculative=dspark.json
category       base_avg_pred_t/s  spec_avg_pred_t/s  decode_speedup  base_avg_latency  spec_avg_latency  latency_speedup  accept_rate
-------------  -----------------  -----------------  --------------  ----------------  ----------------  ---------------  -----------
coding         14.56              45.15              3.10x           121.646s          40.026s           3.04x            0.4527     
humanities     14.69              29.65              2.02x           56.045s           24.973s           2.24x            0.2536     
math           14.71              33.19              2.26x           44.351s           18.708s           2.37x            0.2829     
qa             14.78              30.67              2.08x           26.889s           13.033s           2.06x            0.2420     
rag            14.60              36.29              2.49x           46.995s           21.810s           2.15x            0.2871     
reasoning      14.71              29.17              1.98x           43.860s           20.688s           2.12x            0.2279     
stem           14.71              29.23              1.99x           39.184s           17.862s           2.19x            0.2294     
writing        14.40              31.40              2.18x           152.613s          74.782s           2.04x            0.2514     
multilingual   14.65              31.12              2.12x           69.500s           32.204s           2.16x            0.2646     
summarization  14.73              28.12              1.91x           33.126s           17.515s           1.89x            0.2150     
roleplay       14.60              29.71              2.04x           128.897s          66.507s           1.94x            0.2397     
overall        14.65              32.15              2.19x           69.373s           31.646s           2.19x            0.2720

qwen3.8-27B-dspark

Comparison: baseline=baseline.json speculative=dspark.json
category       base_avg_pred_t/s  spec_avg_pred_t/s  decode_speedup  base_avg_latency  spec_avg_latency  latency_speedup  accept_rate
  -------------  -----------------  -----------------  --------------  ----------------  ----------------  ---------------  -----------
  coding         30.69              64.01              2.09x           117.807s          58.013s           2.03x            0.2974
  humanities     30.72              56.07              1.83x           80.424s           27.638s           2.91x            0.2541
  math           30.76              62.44              2.03x           62.089s           17.632s           3.52x            0.2702
  qa             30.83              54.68              1.77x           35.403s           28.461s           1.24x            0.2545
  rag            30.82              70.27              2.28x           23.925s           10.134s           2.36x            0.3510
  reasoning      30.71              56.35              1.83x           74.643s           21.051s           3.55x            0.2617
  stem           30.71              56.21              1.83x           72.119s           20.270s           3.56x            0.2614
  writing        30.61              58.19              1.90x           111.408s          65.433s           1.70x            0.2410
  multilingual   30.77              61.50              2.00x           41.699s           28.240s           1.48x            0.3058
  summarization  30.85              56.27              1.82x           9.774s            3.635s            2.69x            0.2610
  roleplay       30.72              56.53              1.84x           94.722s           43.616s           2.17x            0.2830
  overall        30.74              59.32              1.93x           65.819s           29.466s           2.23x            0.2723

Additional information

Testing and feedback are welcome.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes

wjinxu · 2026-08-02T04:43:05Z

Will it support this DSpark drafter?

Yes, I’ve tested this checkpoint, but it didn’t perform very well. Its acceptance rate was roughly the same as dflash’s.

ruixiang63 · 2026-08-04T12:34:31Z

        }
        mask_token_id = llama_vocab_mask(llama_model_get_vocab(model_dft));

+        if (is_dspark) {


same here. Feel it is not necessary. Would be better to keep the code change minimal and simple.

ruixiang63 · 2026-08-06T14:37:49Z

What is the difference between this PR and #25549? Which one should we review first?

wjinxu · 2026-08-06T14:47:33Z

I completely understand what you mean. How about we handle this one first? Since this isn't about adding a new backbone but adapting to another training framework, I think it should be relatively simpler.

wjinxu · 2026-08-06T14:47:57Z

As for how to adapt the backbone going forward — Qwen 3.8 may be released in a few days — I think waiting to see how the community chooses could be a good approach. If the community goes with a separate backbone, we can adjust accordingly in time.

ruixiang63 · 2026-08-11T16:37:53Z

can you rebase this PR? @wjinxu Now I have some bandwidth to review the details more closely.

wjinxu · 2026-08-11T16:50:31Z

can you rebase this PR? @wjinxu Now I have some bandwidth to review the details more closely.

Done. Thanks very much.

ruixiang63 · 2026-08-15T00:30:25Z

+        if "aux_hidden_state_layer_ids" in self.hparams:
+            self.hparams.setdefault("dflash_config", {
+                "mask_token_id": self.hparams.get("mask_token_id"),
+                "target_layer_ids": [i - 1 for i in self.hparams["aux_hidden_state_layer_ids"]],


Can you confirm it is i-1 instead of i or i+1?

Confirmed: the base class adds +1 when writing, so -1 passes the aux ids through unchanged, matching EAGLE3.

ruixiang63 · 2026-08-15T01:03:36Z

        ggml_tensor * w1_prev = ggml_get_rows(ctx0, w1, prev);   // [R, n_blocks]
-        ggml_tensor * bias    = ggml_mul_mat(ctx0, w2, w1_prev); // [n_vocab, n_blocks]
+        ggml_tensor * bias    = ggml_mul_mat(ctx0, w2, w1_prev); // [n_vocab_draft, n_blocks]
+        if (model.d2t) {


I saw 1+Nd2t mapping for DSpark. Can you confirm this is correct?

Intentional: logits must be in target-vocab space (sampler + argmax feeding markov_w1, which is keyed by target ids); the per-position w2 bias is draft-vocab-only, so it's zero-scattered onto the -inf base. Greedy output byte-identical, acceptance unchanged.

ruixiang63 · 2026-08-15T01:04:41Z

    cur = build_lora_mm(output, cur, output_s);
+
+    // reduced-draft-vocab exports: scatter the draft logits to the target vocabulary via d2t
+    if (model.d2t) {


The first d2t mapping is here. And the next N mapping is in build_dspark_markov_head.

ruixiang63 · 2026-08-15T01:08:43Z

Once the questions in comments are resolved, could you take another look? @CISC

wjinxu · 2026-08-15T18:31:27Z

Sorry — a previous bug in the sampling decision caused errors when I used qwen3.8-27B-dspark. That is fixed now, and the acceptance rate looks normal.

I am currently running SpeedBench (tools/server/bench/speed-bench). I will update the description with the results when it finishes.

Sorry again for the noise.

Speculators-format DSpark drafts (e.g. SpecForge exports for the Gemma-4-26B-A4B target) differ from the dense DeepSpec checkpoints in three ways: - the config nests the backbone hparams under transformer_layer_config and gives the extract layers as aux_hidden_state_layer_ids - the block is the DFlash 1+N fill-in layout: the anchor slot is a bonus token, not a prediction slot. Written as dflash.bonus_anchor; such drafts build the block and read the mask positions exactly like DFlash (n_max drafts from a 1+n_max block), only the Markov/confidence sampling comes from DSpark - the draft output vocab may be reduced (draft_vocab_size < vocab_size) with a d2t remap table. The converter expands lm_head/markov_w2 back to the full vocab and synthesizes an lm_head bias of -1e9 on the rows the draft cannot produce, so the runtime needs no d2t remapping. Such drafts ship their own (now optional) token_embd/output tensors instead of sharing the target's Verified against gemma4-26b-a4b-dspark: greedy outputs are byte-identical with and without the draft; acceptance 0.46, mean draft len 3.7 (n_max 6). Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable 5

One class now covers every DSpark variant. What used to pick the class is a single flag, because the arch name turns out to be the only thing that separates the two families: SpecForge also exports a flat schema that carries no speculators_* fields yet still uses the 1+N bonus-anchor block, so keying on those fields would silently mis-read its drafts. Also rename i0 to i_first_pred in the draft read loop and the Markov head, and give the head a real bonus_anchor bool instead of testing i0 > 0. Converting the Qwen3-8B DeepSpec draft and both gemma-4 speculators drafts produces byte-identical GGUFs. The one behaviour change is that the markov_head_type check now also covers the DeepSpec checkpoints, which previously skipped it. Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Opus 5

- rename bonus_anchor to sample_from_anchor (GGUF key and code), matching the checkpoint config field; absent key still means anchor-first - rework the reduced draft vocab to match EAGLE3: d2t is written as I64 absolute target ids and the logits are scattered at runtime, instead of expanding lm_head/markov_w2 and synthesizing an output bias at conversion - move the t2d skip to modify_tensors, like EAGLE3 - drop _is_specforge: the arch name only picks the sample_from_anchor default, embed/lm_head sharing is decided by the draft vocab size - deduplicate the tok_embd create_tensor left behind by the rebase Verified with the RedHat gemma-4-31b speculator draft: greedy output is byte-identical with and without the draft; acceptance 0.26 (n_max 7). Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable 5

Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable

* origin/master: (86 commits) ui: enforce alphabetical enum member ordering (ggml-org#27272) ui: Refactor Built-In Tools naming (Server/Browser) (ggml-org#27271) ci: more optimizations (ggml-org#26983) doc: document MCP stdio servers and CORS defaults in the server README [no release] [no ci] (ggml-org#26847) server: save processed mtmd chunks as placeholder (ggml-org#27278) mtmd: use sha256 for input hashing (ggml-org#27274) vocab : support integer tokenizer scores (ggml-org#27260) mtmd : skip thumbnail for non-tiled LFM2 images (ggml-org#27246) cuda : skip UMA override for HIP builds (ggml-org#27083) vendor: move hash to vendor (ggml-org#27262) ci : push release tag explicitly in release.yml (ggml-org#27261) ui: move get_datetime tool to frontend (ggml-org#27255) ci : reduce builds in build-xcframework.sh (ggml-org#27252) model: support speculators-format checkpoints for DSpark (ggml-org#26275) ui: add browser get_info tool (ggml-org#27251) ci : restore release.yml check during make-release.yml (ggml-org#27247) mtmd: harden preprocessor_granite (ggml-org#27235) ci : allow make-release to target a specific commit (ggml-org#27234) ci : make release workflows use a deploy key (ggml-org#27229) convert: add @ModelBase.example (ggml-org#27208) ...

) * dspark: support speculators-format checkpoints (SpecForge exports) Speculators-format DSpark drafts (e.g. SpecForge exports for the Gemma-4-26B-A4B target) differ from the dense DeepSpec checkpoints in three ways: - the config nests the backbone hparams under transformer_layer_config and gives the extract layers as aux_hidden_state_layer_ids - the block is the DFlash 1+N fill-in layout: the anchor slot is a bonus token, not a prediction slot. Written as dflash.bonus_anchor; such drafts build the block and read the mask positions exactly like DFlash (n_max drafts from a 1+n_max block), only the Markov/confidence sampling comes from DSpark - the draft output vocab may be reduced (draft_vocab_size < vocab_size) with a d2t remap table. The converter expands lm_head/markov_w2 back to the full vocab and synthesizes an lm_head bias of -1e9 on the rows the draft cannot produce, so the runtime needs no d2t remapping. Such drafts ship their own (now optional) token_embd/output tensors instead of sharing the target's Verified against gemma4-26b-a4b-dspark: greedy outputs are byte-identical with and without the draft; acceptance 0.46, mean draft len 3.7 (n_max 6). Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable 5 * dspark: fold the speculators draft class into DSparkModel One class now covers every DSpark variant. What used to pick the class is a single flag, because the arch name turns out to be the only thing that separates the two families: SpecForge also exports a flat schema that carries no speculators_* fields yet still uses the 1+N bonus-anchor block, so keying on those fields would silently mis-read its drafts. Also rename i0 to i_first_pred in the draft read loop and the Markov head, and give the head a real bonus_anchor bool instead of testing i0 > 0. Converting the Qwen3-8B DeepSpec draft and both gemma-4 speculators drafts produces byte-identical GGUFs. The one behaviour change is that the markov_head_type check now also covers the DeepSpec checkpoints, which previously skipped it. Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Opus 5 * dspark: address review comments - rename bonus_anchor to sample_from_anchor (GGUF key and code), matching the checkpoint config field; absent key still means anchor-first - rework the reduced draft vocab to match EAGLE3: d2t is written as I64 absolute target ids and the logits are scattered at runtime, instead of expanding lm_head/markov_w2 and synthesizing an output bias at conversion - move the t2d skip to modify_tensors, like EAGLE3 - drop _is_specforge: the arch name only picks the sample_from_anchor default, embed/lm_head sharing is decided by the draft vocab size - deduplicate the tok_embd create_tensor left behind by the rebase Verified with the RedHat gemma-4-31b speculator draft: greedy output is byte-identical with and without the draft; acceptance 0.26 (n_max 7). Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable 5 * dspark: fold the sample_from_anchor read into the block_size block * dspark: fix flake8 continuation indent * clean up * dspark: key the sample_from_anchor default off the export format Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable * dspark: drop t2d in filter_tensors Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable * dspark: map model.lm_head instead of bypassing the dflash prefix Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable --------- Co-authored-by: desovo7 <942845546@qq.com> Co-authored-by: ruixiang63 <wangruixiang07@outlook.com>

github-actions Bot added documentation Improvements or additions to documentation model Model specific conversion labels Jul 29, 2026

wjinxu force-pushed the dspark-speculators branch from 083c33f to 77e9a31 Compare July 29, 2026 13:19

wjinxu marked this pull request as ready for review July 30, 2026 06:18

wjinxu requested review from a team and CISC as code owners July 30, 2026 06:18

wjinxu mentioned this pull request Jul 30, 2026

spec: support DSpark for Gemma4 #25549

Open

ruixiang63 self-requested a review July 30, 2026 10:35

ruixiang63 self-assigned this Jul 31, 2026

wjinxu force-pushed the dspark-speculators branch from 77e9a31 to e107846 Compare August 3, 2026 09:07

ruixiang63 reviewed Aug 4, 2026

View reviewed changes

wjinxu force-pushed the dspark-speculators branch from f806441 to 72f87af Compare August 11, 2026 16:48

ruixiang63 requested changes Aug 11, 2026

View reviewed changes

wjinxu changed the title ~~dspark: support speculators-format checkpoints~~ spec: support speculators-format checkpoints Aug 14, 2026

ruixiang63 reviewed Aug 15, 2026

View reviewed changes

ruixiang63 approved these changes Aug 15, 2026

View reviewed changes

CISC reviewed Aug 16, 2026

View reviewed changes

Comment thread conversion/qwen.py Outdated

Comment thread conversion/qwen.py Outdated

ruixiang63 changed the title ~~spec: support speculators-format checkpoints~~ spec: support speculators-format checkpoints for DSpark Aug 16, 2026

wjinxu and others added 3 commits August 17, 2026 16:45

wjinxu and others added 5 commits August 17, 2026 16:46

dspark: fold the sample_from_anchor read into the block_size block

75e6c0b

dspark: fix flake8 continuation indent

38e4f32

clean up

b464558

dspark: key the sample_from_anchor default off the export format

03aabc0

Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable

dspark: drop t2d in filter_tensors

d11902e

Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable

wjinxu force-pushed the dspark-speculators branch from 6012af5 to d11902e Compare August 17, 2026 08:52

dspark: map model.lm_head instead of bypassing the dflash prefix

64ea235

Co-authored-by: desovo7 <942845546@qq.com> Assisted-by: Claude Fable

CISC approved these changes Aug 17, 2026

View reviewed changes

ruixiang63 merged commit 9cd719a into ggml-org:master Aug 17, 2026
27 of 29 checks passed

wjinxu deleted the dspark-speculators branch August 18, 2026 05:42

Uh oh!

Conversation

wjinxu commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Format differences

Verification

Additional information

Requirements

Uh oh!

wjinxu commented Aug 2, 2026

Uh oh!

Uh oh!

ruixiang63 Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

wjinxu Aug 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ruixiang63 commented Aug 6, 2026

Uh oh!

wjinxu commented Aug 6, 2026

Uh oh!

wjinxu commented Aug 6, 2026

Uh oh!

ruixiang63 commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wjinxu commented Aug 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ruixiang63 Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

wjinxu Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

ruixiang63 Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

wjinxu Aug 15, 2026

Choose a reason for hiding this comment

Uh oh!

ruixiang63 Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ruixiang63 commented Aug 15, 2026

Uh oh!

wjinxu commented Aug 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wjinxu commented Jul 29, 2026 •

edited

Loading

ruixiang63 commented Aug 11, 2026 •

edited

Loading

ruixiang63 Aug 15, 2026 •

edited

Loading

wjinxu commented Aug 15, 2026 •

edited

Loading