B1 mtp qwen rebase - #13
Merged
Merged
Conversation
… Qwen 3.6 NextN - Added detailed descriptions of AtomicChat `UDT` quantization process in NEXTN.md, including tensor-type file overrides and build entrypoints. - Updated README.md to include optional UDT quant information and links to relevant documentation. - Modified bench-matrix script to support combined GGUF benchmarking and added filtering options for benchmark modes. - Improved summary output in the benchmarking script to include optional markdown headings and better formatting.
- Introduced a new environment variable `QWEN_UDT_ABLATION_AUTO` to control filtering for benchmark modes based on model versions. - Refactored the `bench-qwen-udt-matrix-local.sh` script to improve clarity and structure, ensuring proper handling of model types and filtering. - Updated `bench-qwen-udt-quality.sh` to support an optional second pass on chat-style text files, with a default sample chat calibration file included. - Improved error handling in `get-wikitext-2.sh` for downloading and unzipping files. - Added a new sample chat calibration file to enhance benchmarking capabilities.
…Qwen 3.6 NextN enhancements - Revised NEXTN.md to highlight the new AtomicChat UDT collection, detailing the combined `_MTP.gguf` quants and their benefits for NextN processing. - Updated README.md to reflect changes in recommended sources for Qwen 3.6 models, emphasizing the AtomicChat UDT collection and its features. - Enhanced quantization scripts to support improved file handling and added compatibility for new tensor types. - Introduced a new script for running perplexity benchmarks on UDT quant models, generating detailed performance logs. - Improved error handling and user feedback in various scripts to streamline the quantization and benchmarking processes.
github-actions
Bot
added
documentation
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 5, 2026
* oai moe * compat with new checkpoint * add attn sink impl * add rope scaling yarn * logits match with latest transformers code * wip chat template * rm trailing space * use ggml_scale_bias * rm redundant is_swa_all * convert interleaved gate_up * graph : fix activation function to match reference (AtomicBot-ai#7) * vocab : handle o200k_harmony special tokens * ggml : add attention sinks support (AtomicBot-ai#1) * llama : add attn sinks * ggml : add attn sinks * cuda : add attn sinks * vulkan : add support for sinks in softmax remove unnecessary return * ggml : add fused swiglu_oai op (AtomicBot-ai#11) * ggml : add fused swiglu_oai op * Update ggml/src/ggml-cpu/ops.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * update CUDA impl * cont : metal impl * add vulkan impl * test-backend-ops : more test cases, clean up * llama : remove unfused impl * remove extra lines --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: slaren <slarengh@gmail.com> * repack mxfp4 upon conversion * clean up a bit * enable thinking * add quick hack to render only some special tokens * fix bf16 conversion * remove vocab hack * webui ok * support chat parsing for gpt-oss * fix webui * direct mapping mxfp4, FINALLY * force using mxfp4 * properly use lazy tensor * ggml : add mxfp4 ggml : use e8m0 conversion instead of powf Co-authored-by: Diego Devesa <slarengh@gmail.com> change kvalues_mxfp4 table to match e2m1 (AtomicBot-ai#6) metal : remove quantization for now (not used) cuda : fix disabled CUDA graphs due to ffn moe bias vulkan : add support for mxfp4 cont : add cm2 dequant * ggml : add ggml_add_id (AtomicBot-ai#13) * ggml : add ggml_add_id * add cuda impl * llama : add weight support check for add_id * perf opt * add vulkan impl * rename cuda files * add metal impl * allow in-place ggml_add_id * llama : keep biases on CPU with --cpu-moe * llama : fix compile error ggml-ci * cuda : add fallback for __nv_cvt_e8m0_to_bf16raw ggml-ci * cleanup ggml-ci * sycl : fix supports_op for MXFP4 ggml-ci * fix Unknown reasoning format * ggml-cpu : fix AVX build ggml-ci * fix hip build ggml-ci * cuda : add mxfp4 dequantization support for cuBLAS ggml-ci * ggml-cpu : fix mxfp4 fallback definitions for some architectures ggml-ci * cuda : fix version required for __nv_cvt_e8m0_to_bf16raw --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: slaren <slarengh@gmail.com>
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 5, 2026
* FlashAttention (AtomicBot-ai#13) * Add inplace softmax * Move rms_norm to split row approach * Update debug for supports_op * clean up debug statements * neg f16xf32xip builds and runs, havent actually ran a model that uses neg kernel yet though * neg passes backend test * unary operators pass ggml tests * rms_norm double declaration bug atoned * abides by editor-config * removed vestigial files * fixed autoconfig * All operators (inlcluding xielu) working * removed unnecesarry checking if node->src[1] exists for unary operators * responded and dealt with PR comments * implemented REPL_Template support and removed bug in unary operators kernel * formatted embed wgsl and ggml-webgpu.cpp * Faster tensors (AtomicBot-ai#8) Add fast matrix and matrix/vector multiplication. * Use map for shader replacements instead of pair of strings * Wasm (AtomicBot-ai#9) * webgpu : fix build on emscripten * more debugging stuff * test-backend-ops: force single thread on wasm * fix single-thread case for init_tensor_uniform * use jspi * add pthread * test: remember to set n_thread for cpu backend * Add buffer label and enable dawn-specific toggles to turn off some checks * Intermediate state * Fast working f16/f32 vec4 * Working float fast mul mat * Clean up naming of mul_mat to match logical model, start work on q mul_mat * Setup for subgroup matrix mat mul * Basic working subgroup matrix * Working subgroup matrix tiling * Handle weirder sg matrix sizes (but still % sg matrix size) * Working start to gemv * working f16 accumulation with shared memory staging * Print out available subgroup matrix configurations * Vectorize dst stores for sg matrix shader * Gemv working scalar * Minor set_rows optimization (AtomicBot-ai#4) * updated optimization, fixed errors * non vectorized version now dispatches one thread per element * Simplify * Change logic for set_rows pipelines --------- Co-authored-by: Neha Abbas <nehaabbas@macbookpro.lan> Co-authored-by: Neha Abbas <nehaabbas@ReeseLevines-MacBook-Pro.local> Co-authored-by: Reese Levine <reeselevine1@gmail.com> * Comment on dawn toggles * Working subgroup matrix code for (semi)generic sizes * Remove some comments * Cleanup code * Update dawn version and move to portable subgroup size * Try to fix new dawn release * Update subgroup size comment * Only check for subgroup matrix configs if they are supported * Add toggles for subgroup matrix/f16 support on nvidia+vulkan * Make row/col naming consistent * Refactor shared memory loading * Move sg matrix stores to correct file * Working q4_0 * Formatting * Work with emscripten builds * Fix test-backend-ops emscripten for f16/quantized types * Use emscripten memory64 to support get_memory * Add build flags and try ci --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> * Remove extra whitespace * Move wasm single-thread logic out of test-backend-ops for cpu backend * Disable multiple threads for emscripten single-thread builds in ggml_graph_plan * Refactored pipelines and workgroup calculations (AtomicBot-ai#10) * refactored pipelines * refactored workgroup calculation * removed commented out block of prior maps * Clean up ceiling division pattern --------- Co-authored-by: Neha Abbas <nehaabbas@eduroam-169-233-141-223.ucsc.edu> Co-authored-by: Reese Levine <reeselevine1@gmail.com> * Start work on flash attention * Shader structure set up (many bugs still) * debugging * Working first test * Working with head grouping, head sizes to 128, logit softcap, mask/sinks enabled, f32 * Generalize softmax to work with multiple subgroups, f16 accumulation, mask shared memory tiling * Start work on integrating pre-wgsl * Separate structs/initial shader compilation library into separate files * Work on compilation choices for flashattention * Work on subgroup matrix/tile size portability * subgroup size agnostic online softmax * Cleanups, quantization types * more cleanup * fix wasm build * Refactor flashattention to increase parallelism, use direct loads for KV in somce cases * Checkpoint * formatting * Update to account for default kv cache padding * formatting shader * Add workflow for ggml-ci webgpu * Try passing absolute path to dawn in ggml-ci * Avoid error on device destruction, add todos for proper cleanup * Fix unused warning * Forgot one parameter unused * Move some flashattn computation to f32 for correctness
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 5, 2026
AtomicBot-ai#13) Models with n_embd_head_k not divisible by QK_TURBO3_GROUP (128) — e.g. GLM-4.7 Flash / DeepSeek2 MLA with head_dim=576 — previously hard-crashed with GGML_ASSERT(ne00 % QK_TURBO3_GROUP == 0) in set-rows.cu. Fix: - supports_op for SET_ROWS: return false when turbo3 and ne00 % 128 != 0, so llama.cpp falls back to an unquantised KV path instead of asserting - get_best_fattn_kernel: return BEST_FATTN_KERNEL_NONE for turbo3 when K head dim is not 128-aligned (VEC kernel only instantiated for D∈{64,128,256}) Affected models can now load with -ctk turbo3 -ctv turbo3; the non-aligned heads silently use f16 KV while 128-aligned heads continue to use turbo3. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 5, 2026
…tomicBot-ai#13) Models with n_embd_head_k not divisible by 128 (e.g. GLM-4.7 Flash, DeepSeek2 MLA with head_dim=192/576) previously crashed with GGML_ASSERT in set-rows.cu or segfaulted in CPU flash attention. Changes: - llama-kv-cache.cpp: detect incompatible head_dim at KV cache init, log a warning, and auto-fall back to q8_0 instead of crashing - ggml-cuda.cu supports_op: return false for turbo3 SET_ROWS when ne00 % 128 != 0, and for TURBO_WHT when ne[0] % 128 != 0 - fattn.cu: return BEST_FATTN_KERNEL_NONE for turbo3 at non-128 D - ggml-cpu.c: add turbo3 entry to type_traits_cpu (vec_dot + from_float) so CPU flash attention can handle turbo3 K/V for models where CUDA FA returns NONE (e.g. D=192 which has no CUDA FA kernel for any type) - set-rows.cu: add tail kernel for non-128 remainder (future use) - turbo-wht.cu: head-dim-aware processing with tail pass-through - ops.cpp: CPU WHT head-dim-aware with tail identity copy Tested: DeepSeek-Coder-V2 (head_dim=192) now loads with -ctk turbo3 and auto-falls back to q8_0 with clear warning. Qwen3.5 (head_dim=128) continues to use turbo3 at full speed (223 t/s). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 5, 2026
Models with head_dim divisible by 64 but not 128 (e.g. DeepSeek2 MLA
head_dim=192, GLM-4.7 Flash head_dim=576) now use 64-element WHT
groups instead of crashing or falling back to q8_0.
Key changes:
**64-element WHT groups:**
- turbo-quant.cuh: 64-element sign arrays + FWHT-64 + rotate_64
- set-rows.cu: templated on GROUP_SIZE {128,64}, reads group_size
from SET_ROWS op_params (set by llama-kv-cache based on head_dim)
- turbo-wht.cu: templated on group_size, head-dim-aware dispatch
- ggml.h/ggml.c: ggml_turbo_wht() now takes explicit group_size
parameter (0=auto) to handle MLA where output dim differs from
K head dim
- ops.cpp: CPU WHT parameterized on group_size
**MLA fix — missing Q rotation in K-only build_attn:**
- build_attn(inp_attn_k, ...) had no turbo Q pre-rotation, while
SET_ROWS was applying WHT to K. Result: <unrotated_Q, rotated_K>
= garbage. Now all three build_attn variants apply Q rotation.
- Inverse WHT moved inside build_attn_mha (before v_mla projection)
and also added to the non-FA attention path.
**Guards + fallback:**
- supports_op, fattn.cu, llama-graph.cpp: relaxed from %128 to %64
- llama-kv-cache.cpp: falls back to q8_0 only when head_dim%64!=0
- CPU type_traits_cpu turbo3 entry for CPU FA vec_dot
Tested: GLM-4.7 Flash (head_dim=576) 208 t/s, DeepSeek-V2 (192) 143 t/s,
Qwen3.5 (128) 223 t/s — all producing correct output with turbo3.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 5, 2026
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 5, 2026
Status-Update nach Abschluss aller verfuegbaren Cherry-Picks: BLOCKIERT (erfordern Refactor/Infrastruktur): - B3: KV-Cache Fixes (erfordert KV-Cache-Refactor, 30-40h) - C2: gzip compression (andere WebUI-Architektur) - C3: DeepSeek V3.2 (erfordert KV-Cache-Refactor) - AtomicBot-ai#9: CUDA PDL MTP (PDL-Infrastruktur fehlt im Fork) BEREITS VORHANDEN / NICHT RELEVANT: - AtomicBot-ai#10: SVE FWHT Runtime Width (bereits in ops.cpp) - #17b: Gemma 4 Audio Embedding (Granite Speech, nicht unser Fork) - C1: Reasoning Interruption (WebUI nicht genutzt) IN MASTER GEMERGET: - AtomicBot-ai#13: Vulkan iq1 shared memory - AtomicBot-ai#14: Vulkan host memory lock contention - #17a: Gemma 4 Unified Conversion Fix ZURUECKGESTELLT: - C4: Self-Conditioning DiffusionGemma (nicht stabil genug) - D3: Video-Input (nice-to-have, nicht dringend)
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 8, 2026
…scal) Nach git filter-repo path-rename (styx→pascal) waren interne Referenzen inkonsistent: scripts/AGENTS.md, systemd-Service und Skript-Kommentare referenzierten Pascal-Host-server.sh statt pascal-server.sh. DOKU_BEOBACHTUNGEN.md: Sicherheitsverletzungen AtomicBot-ai#8,AtomicBot-ai#10,AtomicBot-ai#12,AtomicBot-ai#13,AtomicBot-ai#16 als ERLEDIGT markiert.
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 12, 2026
…t-ai#13 Two-Tier rejected AtomicBot-ai#3 Pascal CUDA MMVQ Benchmark (Styx, GTX 1070, E2B MoE): - Baseline: 65.1 t/s Generation, 908.9 t/s Prompt - Mit MMVQ: 70.9 t/s Generation (+8.9%), 915.3 t/s Prompt (+0.7%) - Besser als PR-Versprechen (+3-6%) AtomicBot-ai#13 Two-Tier Expert Cache: nicht implementieren - Alle 4 PRs closed ohne Merge - thecodacus Memory Pinning deckt Tier 2 bereits ab - Geringer ROI für Pascal (compute-bound)
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 12, 2026
…udit abgeschlossen AtomicBot-ai#7 Vulkan FA Refactor (PR ggml-org#19625): bereits im Fork (Commit 66e999e) - get_fa_tuning_params_scalar(), row_split, shmem_staging - Q caching in registers, vendor-specific Br selection - 28 Referenzen im Code ROADMAP-Audit Ergebnis: - M1: ✅ abgeschlossen - M2: ✅ evaluiert (AtomicBot-ai#6✅, AtomicBot-ai#7✅ bereits integriert, AtomicBot-ai#9❌, AtomicBot-ai#10❌) - M3: ⏳ teilweise (AtomicBot-ai#3✅, AtomicBot-ai#13❌, AtomicBot-ai#14⏭️, AtomicBot-ai#15 offen) - M4-M6: Noch nicht begonnen (Tier 2-4, 1+ Wochen Aufwand)
fukuro-kun
pushed a commit
to fukuro-kun/fukuro-llama-cpp-turboquant
that referenced
this pull request
Jul 12, 2026
Systematische Prüfung aller ROADMAP-Items gegen Fork-Code: - AtomicBot-ai#11 EAGLE-3 (PR ggml-org#18039): bereits integriert (Commit 5777425) - AtomicBot-ai#12 Coopmat2 (PR ggml-org#19075): bereits integriert (flash_attn_cm2.comp, SPV generiert) - AtomicBot-ai#20 Tensor Parallelism (PR ggml-org#19378): bereits integriert (Commit d850df3) - AtomicBot-ai#28 Adaptive MTP (PR ggml-org#22931): Fork hat eigene Implementierung (LLAMA_MTP_SKIP_STREAK_THRESHOLD), PR closed/inkompatibel Meilenstein-Status: - M1: ✅ abgeschlossen - M2: ✅ evaluiert (AtomicBot-ai#6✅, AtomicBot-ai#7✅, AtomicBot-ai#12✅ bereits integriert, AtomicBot-ai#9❌, AtomicBot-ai#10❌) - M3: ⏳ teilweise (AtomicBot-ai#3✅, AtomicBot-ai#13❌, AtomicBot-ai#14⏭️, AtomicBot-ai#15 offen) - M4: ✅ abgeschlossen (AtomicBot-ai#11✅, AtomicBot-ai#28✅ eigene Implementierung) - M5: ⏳ teilweise (AtomicBot-ai#12✅, AtomicBot-ai#20✅ bereits integriert, AtomicBot-ai#21 offen) - M6: ☐ offen (Tier 4 Forschung)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Additional information
Requirements