feat(local-inference): Gemma-4 cutover finish — vision + kokoro 417× + gemma4-assistant MTP + stale-model safety - #11517
Merged
Conversation
…me — stale-partial discard + bounded sha re-fetch The HF hub re-publishes bundle files under stable names (real incident: bundles/2b/text/eliza-1-2b-128k.gguf went qwen35 1211MB -> gemma4 4737MB). Before this fix the downloader resumed any .part unconditionally, so a stale partial from the old content got gemma4 bytes Range-appended onto qwen35 bytes; the corrupt blob only died at the final sha256 gate with an unrecoverable 'SHA256 mismatch' job failure after gigabytes of transfer. - .part files now carry a sha256 sidecar (<staging>.part.expected) recording the content hash they were started against; a resume is only allowed when it matches the current manifest sha. Stale/unknown partials are discarded and fetched fresh from byte 0. - A completed transfer that fails the sha gate is deleted and re-fetched from scratch once (SHA_MISMATCH_MAX_ATTEMPTS=2) before failing, so a stale CDN edge or mid-flight re-publish self-heals; the wrong bytes never survive under the final name. - Stale completed files on disk (wrong sha vs fresh manifest) were already discarded; now logged via [Downloader] warn. New real-path tests (real Downloader, real fs/hash/registry; Range-aware fetch fixture that behaves like the HF CDN): stale completed file re-pull, stale-partial discard (old-sha sidecar + no-sidecar), genuine resume preserved, bounded re-fetch success, persistent-mismatch failure leaves no wrong-content file or staging residue. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…macOS via vm_stat os.freemem() on macOS maps to Mach 'Pages free' only, which the kernel keeps near zero by design; that tripped the arbiter's critical-pressure gate and refused every non-text capability on healthy machines. Count free+inactive+speculative+purgeable pages instead, mirroring Linux MemAvailable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ION + kokoro Metal perf root-cause - vision: real gemma4-2b + mmproj-2b through the fused libelizainference arbiter path (ModelType.IMAGE_DESCRIPTION) correctly describes the test image (red circle, blue square, doubled "HELLO") in 9.6s — the "1536!=2048 mismatch" was a stale qwen35 local file, not a bug; current HF gemma4 (embd 1536) matches mmproj-2b (proj 1536). - kokoro: profiled RTF~64x root cause — iSTFTNet generator forward = 261s (87% of 302s synth), STFT itself only 66ms; conv stack is CPU-bound, needs Metal dispatch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… record Live-verified matrix of every text weight on HF elizaos/eliza-1 (gemma4 128k shipping per tier; qwen35 context variants kept — referenced by the ollama Modelfile, eagle3 smoke fallback, dflash target-meta provenance, and the HF release audit script). Records the hub repair: both live bundle manifests were schema-invalid for the current downloader (files.vision object-vs-array, 4b conflicting qwen35 mtp sha for the 128k text path, 5 manifest-pinned files missing from the bundle trees) plus globally stale SHA256SUMS. HF commits 1a7c0c7b / c6d9d5cb / 27ca1338. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…UF converter
Pulls elizaOS/llama.cpp feat/gemma4-assistant-mtp-converter (ae0a1eed2):
convert_hf_to_gguf.py Gemma4AssistantModel + gguf-py MODEL_ARCH.GEMMA4_ASSISTANT
/ NEXTN_PROJ_PRE/POST so google/gemma-4-E{2,4}B-it-assistant MTP drafter heads
convert to the fork's gemma4-assistant runtime arch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…1-2b The matched Gemma-4 E2B assistant MTP head (google/gemma-4-E2B-it-assistant) now converts via the fork's new Gemma4AssistantModel converter and is hosted at elizaos/eliza-1 bundles/2b/mtp/drafter-2b.gguf (sha256 0495d34e08d0…, HF commit 0ad707bf; manifest files.mtp + lineage.drafter + evals.mtp populated, SHA256SUMS updated). Flip ELIZA_1_HOSTED_MTP_TIER_IDS to ["eliza-1-2b"] so the catalog advertises components.mtp + runtime.mtp (draft-mtp, draftMax=1) and the load-args resolver wires the bundled drafter. Verified live on Apple M4 Max Metal against text/eliza-1-2b-128k.gguf (sha-identical to the hosted artifact): drafter loads with no tensor/KV errors via ctx_other shared-KV, and drafts get ACCEPTED during speculative decode — 21/25 (0.84) acceptance at --spec-draft-n-max 1, ~1.53x greedy speedup (167 vs 96 tok/s on the counting probe). Evidence: .github/issue-evidence/gemma4-assistant-mtp/ (convert log, drafter GGUF metadata, per-step accepted-draft server log, timings JSON, bench notes). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ant MTP drafter Rebuilt build-desktop-metal libelizainference.dylib (the staged one predated the streaming ABI) and drove the REAL desktop text path — bun:ffi loadElizaInferenceFfi -> eliza_inference_llm_stream_open with mtp_drafter_path -> the fused separate-drafter DRAFT_MTP engine (LLAMA_CONTEXT_TYPE_MTP ctx_other shared-KV) — against the eliza-1-2b gemma4 target with the hosted bundles/2b/mtp/drafter-2b.gguf drafter. Drafter loads with no tensor/KV mismatch and drafts get ACCEPTED: 32/38 = 0.842 acceptance at the shipped draftMax=1 window (matches the llama-server 0.84 measurement), ~1.2x wall speedup, coherent greedy output. One deterministic near-tie argmax flip on the counting probe (batched-verify Metal numerics, continuation matches baseline exactly — rollback correct) is documented in fused-ffi-bench.txt. Evidence: fused-ffi-harness.ts (runnable), fused-ffi-run.txt (full loader + per-step drafted/accepted log), fused-ffi-bench.txt (summary + honest flags). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
….72s (RTF 64x -> 0.15) + before/after evidence (#9033) Bump llama.cpp submodule to 114eee08e (fork branch perf/kokoro-accelerate-blas): kokoro-layers.h routes conv1d / conv_transpose1d / linear / LSTM-gate hot loops through Accelerate sgemm/sgemv on __APPLE__ (scalar fallback kept elsewhere). Root cause CORRECTED vs the baseline hypothesis: no ggml op was falling back from Metal — the iSTFTNet generator never enters ggml at all; it is a single-threaded scalar port, so the 261s (87% of synth) was pure scalar CPU. Evidence (same phrase/model/voice, M4 Max): generator 261,220 -> 595 ms; predictor 24,188 -> 113 ms; synth total 301,697 -> 723 ms. Audio unchanged: identical 112,800 samples, corr 0.99959, identical whisper transcript. Also force-adds the gitignored before/after-profile.log evidence files the README references. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
Contributor
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
23 tasks
…mory.test.ts Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Your trial has ended. Reactivate Greptile to resume code reviews.
This was referenced Jul 2, 2026
6 tasks
lalalune
added a commit
that referenced
this pull request
Jul 3, 2026
…liza-1-4b (#11782) bundles/4b/mtp/drafter-4b.gguf is now hosted at HF elizaos/eliza-1 (HF commit aecac19b97ec): gemma4-assistant arch, f16, 49 tensors, embedding_length_out=2560, nextn_predict_layers=1, converted from google/gemma-4-E4B-it-assistant with the fork converter at the pinned submodule commit 58c0391eb. Manifest files.mtp / lineage.drafter / evals.mtp and checksums/SHA256SUMS updated additively (validated with parseManifestOrThrow + validateManifest). - add eliza-1-4b to ELIZA_1_HOSTED_MTP_TIER_IDS (catalog auto-derives components.mtp + runtime.mtp draft window from the list) - update the shared + plugin catalog tests for the two hosted tiers - fused-eliza1-no-regression: the 4b tier now legitimately advertises runtime.mtp (draft-mtp, mtp/drafter-4b.gguf) - mmproj-routing: cover drafter resolution for both hosted tiers (2b, 4b); move the 'no hosted drafter' case to 9b - imagegen-backend-selector (__tests__ copy): align the stale linux-nvidia expectation with the shipped #10727 policy (pre-existing failure on develop; the src/services sibling suite already asserts the new policy) - evidence: .github/issue-evidence/gemma4-assistant-mtp/4b/ — M4 Max Metal capture through the fused libelizainference product path (acceptance 0.935, up to 1.87x wall greedy) and llama-server (3-prompt set 0.79 / ~1.33x mean, matching the 2b methodology; extended 8-prompt set 0.587 / 1.17x aggregate decode) Closes #11390 (2b shipped in #11517; 9b/27b drafters tracked by #11389) Co-authored-by: Shaw <shawgotbags@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
lalalune
pushed a commit
that referenced
this pull request
Jul 3, 2026
…l non-speculative fallback for a missing drafter GGUF (#11517 back-compat) The Gemma-4 MTP cutover (#11517, then #11782 for 4b) added eliza-1-2b/4b to ELIZA_1_HOSTED_MTP_TIER_IDS, so the catalog now declares separate-drafter runtime.mtp for those tiers. resolveLocalInferenceLoadArgs then HARD-THREW for any bundle missing mtp/drafter-<tier>.gguf — which is every bundle installed before the cutover — so a previously-working installed model could no longer load at all (2 red active-model.test.ts cases caught exactly this). The drafter is a perf-only speculative-decoding artifact (~1.3-1.6x greedy speedup). Missing drafter now means: load the text model without MTP, emit a console warning pointing at re-download to pick the drafter up — never throw. The fallback also covers no-bundleRoot installs (external-scan/single-file), which previously leaked a half-configured speculative state (draft window + mobileSpeculative with NO draft model) into the loader. Applied to both copies of the resolver (packages/ui client mirror and plugins/plugin-local-inference runtime) and their suites: - ui active-model.test.ts: split the stale pre-cutover tests into hosted-tier-enables-MTP / unhosted-tier-ignores-stray-drafter / pre-cutover-bundle-falls-back (warn asserted); 17/17 green. - plugin mmproj-routing.test.ts: + pre-cutover fallback case; 10/10 green. - plugin load-args-drafter.fuzz.test.ts: throw assertion -> fallback assertion; external-scan case now expects a fully-unset MTP block; 22/22. Signed: nubs-cloud [cloud-frontdoor] Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lalalune
added a commit
that referenced
this pull request
Jul 3, 2026
fix(local-inference): graceful non-speculative fallback for a missing MTP drafter GGUF — #11517 back-compat, never brick an installed model
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Focused local-inference slice cherry-picked clean onto develop (8 commits). Depends on fork PR elizaOS/llama.cpp#38 (submodule gitlink → 114eee08e).
What's in this PR
IMAGE_DESCRIPTIONproven E2E through the fusedlibelizainferencearbiter path; the long-standing "1536≠2048 mismatch" was a stale qwen35 local file, not a bug. Current HF gemma4 (embd 1536) matches mmproj-2b. Evidence:.github/issue-evidence/vision-gemma4-describe/..github/issue-evidence/kokoro-metal-perf/.Gemma4AssistantForCausalLM→GGUF converter (fork), drafter converted + uploaded to HF (bundles/2b/mtp/drafter-2b.gguf,embedding_length_out=1536), load+draft proven (0.842 acceptance) via the fused MTP path, wired for eliza-1-2b (ELIZA_1_HOSTED_MTP_TIER_IDS). Evidence:.github/issue-evidence/gemma4-assistant-mtp/..github/issue-evidence/gemma4-cutover-final/.vm_statreclaimable-pages fix for the memory-fit preflight.Evidence
Test
bun run verifylocally on the branch; plugin suite + shared catalog tests green (17/17 downloader, 10/10 catalog, 26/26 plugin routing/drafter-resolution).🤖 Generated with Claude Code
Post-merge follow-up: the non-Apple (Android/Windows) kokoro fast path (thread pool + NEON, 23/23 parity vs scalar, up to 107× conv microbench) landed after this squash-merged — fork gitlink bump
114eee08e→2bdcef890ships in #11545.