feat(training): wire MTP drafter sources to the real Gemma-4 assistant heads - #9256
Conversation
…t heads
MTP_DRAFTER_BASE pointed at placeholder repos (google/gemma-4-E2B-mtp etc. — do
not exist). Google actually ships the official MTP head with the `-it-assistant`
checkpoint of each Gemma 4 size (a gemma4-assistant-arch 4-block NextN drafter),
loadable via `-md <drafter>.gguf --spec-type draft-mtp` after upstream
convert_hf_to_gguf.py — no distillation / H200 needed, the weights already exist.
Repoint the per-tier map to google/gemma-4-{E2B,E4B,12B,31B}-it-assistant (all
exist on HF) and note the ready GGUF conversions (amaranus/..., cortexist/...,
Janvitos/...). Validated: the E2B drafter gives 1.29x decode speedup on Apple M4
Max Metal (plugins/plugin-local-inference/docs/gemma4-assistant-fork-port-plan.md).
Makes the publish pipeline stage the real, validated drafter — release-shaped
once the fork gains gemma4-assistant arch support (the port in flight).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
…support (#9268) Points the fork submodule at the gemma4-assistant port (validated: loads Google's Gemma-4 MTP drafter + ~1.1x decode speedup via --spec-type draft-mtp on M4 Max Metal). bcae29e65 (prior gitlink) is a clean ancestor, so this is a fast-forward that adds the metal-tbq attn-score fix + the gemma4-assistant arch. Tracks fork PR elizaOS/llama.cpp#32; re-point to the merged commit once that lands. This is the runtime half of the Gemma-4 MTP drafter work — with this, the fused engine built from the fork can load mtp/drafter-<tier>.gguf (the amaranus/Google gemma-4-E2B-it-assistant head, wired in #9256) and run separate-drafter MTP. Co-authored-by: Shaw <shawgotbags@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 participant
#2 of the Gemma-4 MTP drafter work (the drafter is validated; this points the publish pipeline at it).
MTP_DRAFTER_BASEinmodel_registry.pypointed at placeholder repos (google/gemma-4-E2B-mtpetc. — these don't exist). The real drafter is the official MTP head Google ships with the-it-assistantcheckpoint of each Gemma 4 size (agemma4-assistant-arch 4-block NextN drafter), loaded via-md <drafter>.gguf --spec-type draft-mtpafter upstreamconvert_hf_to_gguf.py.google/gemma-4-{E2B,E4B,12B,31B}-it-assistant(all verified HTTP 200 on HF).amaranus/Gemma-4-{E2B,E4B}-it-qat-assistant-MTP-Q8_0-GGUF,{cortexist,Janvitos}/gemma-4-12B-it-assistant-MTP-GGUF.Validated: the E2B drafter gives a 1.29× decode speedup against the eliza-1-2b target on Apple M4 Max Metal (via upstream llama.cpp; see the port plan).
This makes the publish pipeline stage the real, validated drafter — the bundle is release-shaped the moment the fork gains
gemma4-assistantarch support (the port is in flight in #1).🤖 Generated with Claude Code