Skip to content

feat(models): add MTP drafters for Gemma 4 QAT models - #1269

Merged
gabrielste1n merged 5 commits into
OpenWhispr:mainfrom
xAlcahest:feat/gemma4-mtp-drafters
Jul 22, 2026
Merged

feat(models): add MTP drafters for Gemma 4 QAT models#1269
gabrielste1n merged 5 commits into
OpenWhispr:mainfrom
xAlcahest:feat/gemma4-mtp-drafters

Conversation

@xAlcahest

@xAlcahest xAlcahest commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Second half of the Gemma 4 work, stacked on #1268 (merge that one first; until then the diff shows its commit too). The bundled llama-server b9763 already supports MTP speculative decoding for Gemma 4: generic MTP (ggml-org/llama.cpp#22673), Gemma 4 MTP (ggml-org/llama.cpp#23398) and the E2B/E4B assistant loaders (ggml-org/llama.cpp#24282) all predate the b9763 cut, so no binary bump is involved. This adds optional draftHfRepo/draftFileName/draftSizeBytes fields to the four QAT registry entries pointing at unsloth's published drafter GGUFs (57MB for E2B/E4B, 240-280MB for 26B/31B), downloads the drafter alongside the main GGUF, and starts llama-server with --model-draft --spec-type draft-mtp --spec-draft-n-max 3 when the drafter is on disk. When it is not (including everyone who already pulled the QAT models from #1268), the server starts with today's exact args, so a working setup cannot regress.

Scope and numbers: E2B/E4B are the combination upstream validated in ggml-org/llama.cpp#24282; the 31B and 26B MoE drafters are published but sit outside that PR's stated scope, and upstream saw no MTP speedup on the MoE (ggml-org/llama.cpp#23398), so expect little or nothing on the 26B. Measured here (Linux, CPU, the bundled b9763, E2B QAT + drafter, the app's exact request shape): draft acceptance 0.89 and 82 vs 31 tok/s against the same model without the drafter, with identical output; a longer thinking-enabled run gave 0.70 acceptance at 67 vs 29 tok/s. Upstream's own mobile test reported ~0.48 acceptance, so the gain varies with workload. Field testing also surfaced stale user-installed Vulkan binaries (older than the pin) that reject the MTP args at parse time, so the server start now degrades in order (GPU with MTP, GPU alone, CPU with MTP, plain CPU): a live GPU beats speculation, and the worst case is exactly today's behavior.

Fixes #730

Changes

  • src/models/modelRegistryData.json: drafter field triplets on the four QAT entries, size labels updated to the combined totals (17.9, 14.7, 5.2 and 3.4GB)
  • src/models/ModelRegistry.ts: optional draftHfRepo, draftFileName and draftSizeBytes on ModelDefinition
  • src/helpers/modelManagerBridge.js: dual download with monotonic combined progress, disk precheck covering both files, drafter failure or cancel keeps the main model usable, deleteModel removes the drafter too, resolveDraftPath gates on declared + on disk + over 1MB
  • src/helpers/llamaServer.js: appends the three MTP flags when draftModelPath is set, restarts when drafter presence changes for the same model, and degrades stale binaries down a GPU+MTP, GPU, CPU+MTP, CPU ladder
  • src/helpers/ipcHandlers.js: the manual llama-server-start handler mirrors the same drafter resolution
  • test/helpers/gemmaMtpRegistry.test.js, test/helpers/gemmaMtpDrafter.test.js: thirteen tests covering the registry triplets, URL shape, fallback gate, delete, restart decisions and the degrade ladder

@xAlcahest
xAlcahest marked this pull request as draft July 21, 2026 14:17
@xAlcahest
xAlcahest marked this pull request as ready for review July 21, 2026 15:45
@xAlcahest
xAlcahest requested a review from gabrielste1n July 21, 2026 15:47

@gabrielste1n gabrielste1n left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Brought up to date with main (resolved registry size-label conflicts in favor of the combined main+drafter sizes, which the new registry test enforces) and extracted the repeated draft-path start-options block into a shared helper. Fallback ladder, opportunistic drafter download, and tests all look solid — 678 tests passing locally with lint/typecheck clean.

@gabrielste1n gabrielste1n left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Brought up to date with main (resolved registry size-label conflicts in favor of the combined main+drafter sizes, which the new registry test enforces) and extracted the repeated draft-path start-options block into a shared helper. Fallback ladder, opportunistic drafter download, and tests all look solid — 678 tests passing locally with lint/typecheck clean.

@gabrielste1n
gabrielste1n merged commit 860c33b into OpenWhispr:main Jul 22, 2026
9 checks passed
@gabrielste1n
gabrielste1n deleted the feat/gemma4-mtp-drafters branch July 22, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add latest Gemma 4 models with MTP drafters

2 participants