feat(models): add MTP drafters for Gemma 4 QAT models - #1269
Conversation
gabrielste1n
left a comment
There was a problem hiding this comment.
Brought up to date with main (resolved registry size-label conflicts in favor of the combined main+drafter sizes, which the new registry test enforces) and extracted the repeated draft-path start-options block into a shared helper. Fallback ladder, opportunistic drafter download, and tests all look solid — 678 tests passing locally with lint/typecheck clean.
gabrielste1n
left a comment
There was a problem hiding this comment.
Brought up to date with main (resolved registry size-label conflicts in favor of the combined main+drafter sizes, which the new registry test enforces) and extracted the repeated draft-path start-options block into a shared helper. Fallback ladder, opportunistic drafter download, and tests all look solid — 678 tests passing locally with lint/typecheck clean.
Summary
Second half of the Gemma 4 work, stacked on #1268 (merge that one first; until then the diff shows its commit too). The bundled llama-server b9763 already supports MTP speculative decoding for Gemma 4: generic MTP (ggml-org/llama.cpp#22673), Gemma 4 MTP (ggml-org/llama.cpp#23398) and the E2B/E4B assistant loaders (ggml-org/llama.cpp#24282) all predate the b9763 cut, so no binary bump is involved. This adds optional draftHfRepo/draftFileName/draftSizeBytes fields to the four QAT registry entries pointing at unsloth's published drafter GGUFs (57MB for E2B/E4B, 240-280MB for 26B/31B), downloads the drafter alongside the main GGUF, and starts llama-server with
--model-draft --spec-type draft-mtp --spec-draft-n-max 3when the drafter is on disk. When it is not (including everyone who already pulled the QAT models from #1268), the server starts with today's exact args, so a working setup cannot regress.Scope and numbers: E2B/E4B are the combination upstream validated in ggml-org/llama.cpp#24282; the 31B and 26B MoE drafters are published but sit outside that PR's stated scope, and upstream saw no MTP speedup on the MoE (ggml-org/llama.cpp#23398), so expect little or nothing on the 26B. Measured here (Linux, CPU, the bundled b9763, E2B QAT + drafter, the app's exact request shape): draft acceptance 0.89 and 82 vs 31 tok/s against the same model without the drafter, with identical output; a longer thinking-enabled run gave 0.70 acceptance at 67 vs 29 tok/s. Upstream's own mobile test reported ~0.48 acceptance, so the gain varies with workload. Field testing also surfaced stale user-installed Vulkan binaries (older than the pin) that reject the MTP args at parse time, so the server start now degrades in order (GPU with MTP, GPU alone, CPU with MTP, plain CPU): a live GPU beats speculation, and the worst case is exactly today's behavior.
Fixes #730
Changes