Fix memory explosion on Apple Silicon - #76
Merged
TheTom merged 1 commit intoApr 14, 2026
Merged
Conversation
|
pulling now for testing! |
Owner
|
tested on M2 Pro (16GB): build: clean, no errors smoke test: Qwen2.5-7B Q8_0 generation works correctly bench (p512/tg128, Qwen2.5-7B Q8_0):
PPL (turbo4 KV, 5 chunks wikitext): 6.3664 +/- 0.48 memory: 304 MiB compute on MTL0, no explosion fix looks correct. TurboFlash scratch only allocated when the turbo path is eligible, normal FA path stays lean. thanks for the find and fix! Co-Authored-By: tturney@psyguard.ai |
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 19, 2026
…026-07-13), ROADMAP AtomicBot-ai#56-TheTom#76, M3 abgeschlossen
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 20, 2026
Die 'Priorisierung nach Cost-Benefit' Sektion listete noch M1-M4 Items als 'Sofort umsetzbar' obwohl alle ✅/❌ sind. Aktualisiert auf M6 (Forschung): - Tier 2: AtomicBot-ai#36 Auto Parameter Fitting TP (einziger offener Tier-2-Item) - Tier 3: 11 Items nach Cost-Benefit sortiert (AtomicBot-ai#73, AtomicBot-ai#72, AtomicBot-ai#71, TheTom#76, ...) - Tier 4: 15 Forschungs-Items (3+ Monate Solo-Agent) - Verschobene Items (⏭️) mit Re-Eval-Hinweisen Header-Datum auf 2026-07-19 aktualisiert.
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 20, 2026
…hed ⏭️ + TheTom#76 CPU Fusion ⏭️ 3 parallele Tiefen-Evals für Tier-3 Items: AtomicBot-ai#74 Vulkan Descriptor Indexing (Bindless) ❌ VERWORFEN: - Redundant mit TheTom#85 Push Descriptors (✅ implementiert 2026-07-14) - Push Descriptors eliminieren dieselben CPU-Aufrufe - TheTom#85-Benchmark auf Mars RADV: ±0.1-0.3% (Rauschen) - Bindless würde über Push-Descriptors hinaus <0.5% bringen - Workload-Mismatch: Bindless für draw-heavy Rendering, nicht Compute - Mars/Venus bandwidth-bound, nicht descriptor-bound - Aufwand revidiert: 2-4 → 3-5 Wochen (Shader-Rewrite aller .comp-Files) AtomicBot-ai#75 Non-blocking Pipeline Scheduling ⏭️ SPÄTER: - PR ggml-org#19922 closed (2026-03-03, unmerged, 4+ Mo stale) - Fork hat bereits Upstream-Pipeline-Parallelismus - Konflikt mit TheTom#79 TP (✅+23-32% tg, split-mode-exklusiv) - NVIDIA ungetestet, PP-Regression auf 4x MI50 gemeldet - 2-GPU-Setup → geringer Bubble-Hebel - Aufwand revidiert: 3-4 → 4-6 Wochen TheTom#76 CPU Backend Operator Fusion ⏭️ SPÄTER: - RMS_NORM+MUL Fusion bereits im Fork (PR ggml-org#22423 upstream-merged) - MoE Gated FFN riskant: PR ggml-org#20596 zeigt Regressionen auf Consumer-CPUs (M2: 0.98-1.00x, qwen3moe 30B: 0.75-0.96x bei t=2-4) - Nur auf 96-Core-EPYC konsistente Gains (1.04-1.08x) - Styx/Uranus haben Consumer-CPUs → wahrscheinlich Regression - Re-Eval wenn PR ggml-org#20596 gemerged mit Regression-Freiheit
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 20, 2026
…efen-Evals — Tier-3 komplett Letzte 2 Tiefen-Evals für Tier-3 Items: AtomicBot-ai#43 SliderQuant (Sliding-layer PTQ) ⏭️ SPÄTER (conditional): - PTQ, Fork nutzt QAT Q4_K_XL (methodisch überlegen, wie AtomicBot-ai#22 GWQ) - Referenzcode vorhanden (github.com/deep-optimization/SliderQuant, PyTorch) aber hilft nicht beim C++/Vulkan-Port - Vulkan-Kernel-Neuentwicklung hochriskant für Mars/Venus - Hardware-Relevanz niedrig: QAT löst Low-Bit-Druck bereits - Aufwand revidiert: 4-6 → 6-10 Wochen - Reaktivierung nur bei QAT-Lücke für spezifisches Zielmodell AtomicBot-ai#44 Alloc-MoE (Budget-aware Expert Activation) ☐ MACHBAR GESTAFFELT: - Alloc-L (Layer-Level, DP + Sensitivity-Profiling) ist MVP mit 3-4 Wochen - Alloc-T (Token-Level) bringt bei Decode T=1 nichts — 1.34× decode-Speedup stammt aus Alloc-L (reduziertes K), nicht Alloc-T - Synergie mit AtomicBot-ai#40 (Freq-Tracking als Input) und AtomicBot-ai#62 (Profiler-Infra) - Kein Konflikt mit moe-cache.cu (synergistisch) - Styx profitiert klar (CPU-bound → halbe Aktivierungen = halbe CPU-Last) - RISIKO: 17% Quality-Drop bei K=2 (Gemma 4 A4B) — Quality-Benchmark vor Go - Aufwand revidiert: 6-8 → 3-4 Wo (Phase 1) / 7-9 Wo (voll+Vulkan) Tier-3 Tiefen-Evals KOMPLETT (11/11 Items): - ☐ Machbar: AtomicBot-ai#71, AtomicBot-ai#44, AtomicBot-ai#18 (gestaffelt, mit Solo-Plan) - ⏭️ Später: TheTom#76, AtomicBot-ai#75, AtomicBot-ai#23, AtomicBot-ai#43 (Abhängigkeiten/Risiko) - ❌ Verworfen: AtomicBot-ai#73, AtomicBot-ai#72, AtomicBot-ai#74, AtomicBot-ai#22 (Architektur/Redundanz/QAT-Konflikt) Priorisierung aktualisiert: 3 machbare + 4 spätere + 4 verworfene.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR offers a solution to issue #75
Additional information
Metal/TurboFlash change added extra temporary memory for a special fast attention path.
Requirements
Allocate it only when the TurboFlash path applies.
AI Disclosure: Found and implemented by Codex. Tested and Reviewed by me.