feat: add additional TurboQuant kernel templates for enhanced flash a… by Ooooze · Pull Request #2 · AtomicBot-ai/atomic-llama-cpp-turboquant

Ooooze · 2026-04-02T18:02:01Z

Overview

Additional information

https://github.com/TheTom/llama-cpp-turboquant/compare/experiment/turbo4-quality-investigation...AtomicBot-ai:atomic-llama-cpp-turboquant:feature/sdf?expand=1

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure:

…ttention support Added new kernel templates for 512x512 dimensions across TurboQuant configurations (turbo2, turbo3, turbo4) to improve flash attention capabilities. This enhancement allows for better performance and flexibility in handling larger input sizes.

Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed #3 (TURBO_D). #1 and #2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Complete experiment log: #1 4-mag LUT: 15.1 at 8K (BEST, +38%) #2 Batched extract: 13.7 (+25%) #3 Inline FA block: 13.5 (I-cache pressure) #4 Deferred norm: 12.9 (loses ILP) #5 2-pair half2: 12.0 (ternary overhead) #6 Select chain: 11.9 (branches kill) #7 Bit-arithmetic: 11.6 (ALU too heavy) #8 FMA branchless: 11.4 (ALU still too heavy) #9 Named-reg ternary: 10.3 (branches worst) #10 Main (8-LUT): 10.95 (baseline) #11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai

* vulkan: add TQ4_1S weight compression support Adds Vulkan shader support for TQ4_1S (4-bit WHT-rotated weight compression with 16 Lloyd-Max centroids, 32-element blocks). Shaders: - dequant_tq4_1s.comp: standalone dequant with WHT inverse via subgroupShuffleXor (32-thread workgroup, 5-stage butterfly) - mul_mat_vec_tq4_1s.comp: specialized MUL_MAT_VEC with inline activation pre-rotation (forward RHT on activation, centroid*scale dequant without inverse RHT) - copy_from_quant.comp: TQ4_1S dequant path with full WHT inverse - copy_to_quant.comp: TQ4_1S SET_ROWS quantization path with forward RHT, dual half-block RMS scales, 16-centroid quantization - types.glsl: block_tq4_1s struct (d0, d1, qs[16]) - dequant_funcs.glsl: TQ4_1S centroid*scale dequant (no RHT) Pipeline wiring (ggml-vulkan.cpp): - MUL_MAT, SET_ROWS, CPY supports_op - pipeline_dequant, pipeline_set_rows, pipeline_cpy_quant_f32 - Specialized MUL_MAT_VEC with forced subgroup workgroup size Tests: - test_set_rows_tq4_1s: SET_ROWS round-trip validation * vulkan: add fused mul_mat_vec kernel for TQ4_1S Adds a specialised MUL_MAT_VEC shader for TQ4_1S weights so the per-decode-step matrix-vector product no longer has to dequant the full weight tensor to f16 and then go through the generic matmul path. The kernel pre-rotates the activation via a forward Walsh-Hadamard Transform in shared memory and dot-products against the raw centroid*scale stored weights, folding the inverse-WHT on the weight side into the activation by the symmetry H = H^T. Math: w[k] = sign[k] * INV_SQRT32 * (H @ stored)[k] sum_k w[k] * a[k] = INV_SQRT32 * sum_j stored[j] * (H @ (sign * a))[j] Portability choices: - Workgroup size is pinned to 32 threads regardless of the DMMV_WG_SIZE bucket the rest of the mul_mat_vec family picks for the current architecture. The butterfly operates on 32-element blocks with one element per thread; that contract is fixed by the quantization format, not by the GPU. Earlier revisions used `gl_WorkGroupSize.x` as the stride unit, which silently skipped half the work on Intel drivers that force the subgroup to 16 (tests passed via NMSE tolerance while real inference output was garbage). - Butterfly implementation is shared memory only. A subgroup-shuffle variant (`subgroupShuffleXor`) was prototyped and measured on Intel Arc A380 with Mesa Xe HPG: it ran ~60-85 %% slower than the explicit shared-memory butterfly, because Mesa emulates subgroup shuffles via LDS and ends up doing the same LDS traffic with extra driver overhead. The shared-memory butterfly is correct on every device regardless of subgroup-op support, is the fastest path on every device we can actually measure, and leaves the `pipeline_dequant_mul_mat_vec_f32_f32[w][TQ4_1S]` slot uniform across all DMMV_WG_SIZE buckets. - Reduction is the shared-memory tree reduction (no subgroupAdd), for the same reason: on Intel Arc the subgroupAdd is also LDS-backed and the hybrid reduction path was measurably slower. Future vendor-specific heuristics can switch to the hybrid or pure-subgroup reduction variants on NVIDIA / AMD RDNA if hardware subgroup ops turn out to beat the LDS roundtrip there; the existing reduction modes in `mul_mat_vec_base.glsl` already provide the necessary variants. - NUM_ROWS is 8 so the butterfly cost amortises across 8 output rows per workgroup. Each thread holds one position of each of the 8 weight blocks and pairs them with the shared rotated activation. - `mul_mm` and `flash_attn_cm2` shader generation is skipped for TQ4_1S because it is a weight-only format that never reaches the coopmat2 matmul or the KV cache flash-attention paths. Tests: - `test-backend-ops` MUL_MAT tolerance tightened from 2.0 to 0.01 NMSE so real defects can't hide behind a loose check. - Added Gemma-4 E2B, Qwen, Phi and Llama dimensional coverage (k in {1536, 2048, 2304, 3072, 4096}, m in {256, 1152, 1536, 2048, 5120, 6144}, n in {1..8, 16, 64, 256}). 148 MUL_MAT test cases total. Verification (Intel Arc A380, 6 GB VRAM, Vulkan ANV / Mesa Xe HPG, `llama-bench -p 512 -n 128 -r 3` and `llama-perplexity -c 512 --chunks 20 wiki.test.raw`): | Model | Config | Size | Reduction | PPL Δ | pp512/Q8 | tg128/Q8 | |---------------|---------|----------:|----------:|-------:|---------:|---------:| | Qwen2.5-1.5B | I | 1570→1082 | -31.1% | +4.66% | 53.9% | 107.5% | | Phi-3.5-mini | I | 3873→2839 | -26.7% | +5.36% | 57.6% | 52.8% | | Llama-3.2-3B | hybrid | 3263→2147 | -34.2% | +2.03% | 82.4% | 84.2% | | Llama-3.2-3B | premium | 3263→2577 | -21.0% | +0.98% | 71.3% | 67.3% | Qwen2.5-1.5B is faster than its own Q8_0 baseline with Config I: the compressed model fits in less VRAM, and on a small model the TQ4_1S compute cost is offset by the reduced memory traffic. All four models produce coherent output end-to-end and the reductions line up with the TurboQuant paper's validation matrix (§5.8). The remaining gap to Q8_0 on the bigger models is compute-bound on the A380; it closes further on GPUs with more raw throughput. * vulkan: restructure TQ4_1S inner loop for cross-row smem reuse Splits the dequant+accumulate phase into two sub-loops: 1. Pre-compute w_vals[n] for all NUM_ROWS rows (centroid lookup + scale multiply, reads from weight buffer only). 2. Read the rotated activation from shared memory ONCE per column, then FMA across all rows in a tight register loop. This is the Vulkan analogue of the 'hot loop load dedup' from the CUDA kernel (PR #57 optimisation #2). It makes the shared memory read explicitly loop-invariant across rows, which helps compilers that don't auto-hoist LDS loads out of unrolled loops. Measured effect on Intel Arc A380 (Llama-3.2-3B premium, llama-bench tg128, r=5): 15.50 -> 15.78 t/s (+1.8%, within noise but not a regression). The structure is cleaner regardless and should benefit architectures with higher LDS latency.

…gml-org#16038) Initalizing RESERVED_NAME in is_reserved_name() is not thread safe and leads to corrupted memory when used from multiple threads as can be seen in the asan trace below. This fixes the initialization to make it thread-safe. #0 0x000100abd018 in std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, void*>*>, bool> std::__1::__hash_table<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>::__emplace_unique_key_args<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) __hash_table:1565 AtomicBot-ai#1 0x000100ab0320 in SchemaConverter::visit(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) json-schema-to-grammar.cpp:802 AtomicBot-ai#2 0x000100aafc48 in std::__1::__function::__func<build_grammar(std::__1::function<void (common_grammar_builder const&)> const&, common_grammar_options const&)::$_2, std::__1::allocator<build_grammar(std::__1::function<void (common_grammar_builder const&)> const&, common_grammar_options const&)::$_2>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&) function.h:319 AtomicBot-ai#3 0x000100a2c938 in std::__1::__function::__func<common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool)::$_0::operator()(common_grammar_builder const&) const::'lambda'(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&), std::__1::allocator<common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool)::$_0::operator()(common_grammar_builder const&) const::'lambda'(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&)>, void (nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&)>::operator()(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&) function.h:319 AtomicBot-ai#4 0x000100a139f8 in foreach_function(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&, std::__1::function<void (nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&)> const&) chat.cpp:762 AtomicBot-ai#5 0x000100a2a7f4 in std::__1::__function::__func<common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool)::$_0, std::__1::allocator<common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool)::$_0>, void (common_grammar_builder const&)>::operator()(common_grammar_builder const&) function.h:319 AtomicBot-ai#6 0x000100aa98f4 in build_grammar(std::__1::function<void (common_grammar_builder const&)> const&, common_grammar_options const&) json-schema-to-grammar.cpp:982 AtomicBot-ai#7 0x0001009c9314 in common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool) chat.cpp:1110 AtomicBot-ai#8 0x0001009b8afc in common_chat_templates_apply_jinja(common_chat_templates const*, common_chat_templates_inputs const&) chat.cpp:1992 AtomicBot-ai#9 0x0001009b533c in common_chat_templates_apply(common_chat_templates const*, common_chat_templates_inputs const&) chat.cpp:2074 AtomicBot-ai#10 0x000100810120 in llamacpp_apply_chat_template+0x724 (predict_oai-98384e17fb94e863:arm64+0x100090120) ... ==45482==Register values: x[0] = 0x00006020004147f8 x[1] = 0x00006080000013c8 x[2] = 0x0000000000000000 x[3] = 0x0000604006289738 x[4] = 0x0000000000000002 x[5] = 0x0000000000000001 x[6] = 0x04034000004b4000 x[7] = 0x0000000000000001 x[8] = 0xbebebebebebebebe x[9] = 0x17d7d7d7d7d7d7d7 x[10] = 0x00000c04000828ff x[11] = 0x0000000000000001 x[12] = 0x000000002018d383 x[13] = 0x0000000000000000 x[14] = 0xfa0000000000fafa x[15] = 0x000010700001ffff x[16] = 0x000000019dc012c0 x[17] = 0x00000001021284f8 x[18] = 0x0000000000000000 x[19] = 0x00000001700acdc0 x[20] = 0x0000000000000002 x[21] = 0x000000002018d384 x[22] = 0x16dd16fd2e731151 x[23] = 0x0000007000020000 x[24] = 0x0000000100c69c08 x[25] = 0x0000000100c69c20 x[26] = 0x00006080000013c7 x[27] = 0x0000000100c69c00 x[28] = 0x00000001700acd60 fp = 0x00000001700aceb0 lr = 0x0000000100abce30 sp = 0x00000001700acd60 AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV __hash_table:1565 in std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, void*>*>, bool> std::__1::__hash_table<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>::__emplace_unique_key_args<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) Thread T5 created by T0 here: #0 0x0001020b99d4 in pthread_create+0x5c (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x359d4) AtomicBot-ai#1 0x000100873910 in std::sys::pal::unix::thread::Thread::new::h77254fdd87a28e05+0x118 (predict_oai-98384e17fb94e863:arm64+0x1000f3910) AtomicBot-ai#2 0x0001007c7a1c in test::run_test::haeb3c2bcd5ed6cf6+0x76c (predict_oai-98384e17fb94e863:arm64+0x100047a1c) AtomicBot-ai#3 0x0001007aedb0 in test::console::run_tests_console::he9d142d704f3a986+0x149c (predict_oai-98384e17fb94e863:arm64+0x10002edb0) AtomicBot-ai#4 0x0001007c5758 in test::test_main::hf86a5e20735245b9+0x118 (predict_oai-98384e17fb94e863:arm64+0x100045758) AtomicBot-ai#5 0x0001007c5da0 in test::test_main_static::h61ee9c8fd30abca0+0x54 (predict_oai-98384e17fb94e863:arm64+0x100045da0) ... ==45482==ABORTING

…-ai#29 Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed AtomicBot-ai#3 (TURBO_D). AtomicBot-ai#1 and AtomicBot-ai#2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Complete experiment log: AtomicBot-ai#1 4-mag LUT: 15.1 at 8K (BEST, +38%) AtomicBot-ai#2 Batched extract: 13.7 (+25%) AtomicBot-ai#3 Inline FA block: 13.5 (I-cache pressure) AtomicBot-ai#4 Deferred norm: 12.9 (loses ILP) AtomicBot-ai#5 2-pair half2: 12.0 (ternary overhead) AtomicBot-ai#6 Select chain: 11.9 (branches kill) AtomicBot-ai#7 Bit-arithmetic: 11.6 (ALU too heavy) AtomicBot-ai#8 FMA branchless: 11.4 (ALU still too heavy) AtomicBot-ai#9 Named-reg ternary: 10.3 (branches worst) AtomicBot-ai#10 Main (8-LUT): 10.95 (baseline) AtomicBot-ai#11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai

* vulkan: add TQ4_1S weight compression support Adds Vulkan shader support for TQ4_1S (4-bit WHT-rotated weight compression with 16 Lloyd-Max centroids, 32-element blocks). Shaders: - dequant_tq4_1s.comp: standalone dequant with WHT inverse via subgroupShuffleXor (32-thread workgroup, 5-stage butterfly) - mul_mat_vec_tq4_1s.comp: specialized MUL_MAT_VEC with inline activation pre-rotation (forward RHT on activation, centroid*scale dequant without inverse RHT) - copy_from_quant.comp: TQ4_1S dequant path with full WHT inverse - copy_to_quant.comp: TQ4_1S SET_ROWS quantization path with forward RHT, dual half-block RMS scales, 16-centroid quantization - types.glsl: block_tq4_1s struct (d0, d1, qs[16]) - dequant_funcs.glsl: TQ4_1S centroid*scale dequant (no RHT) Pipeline wiring (ggml-vulkan.cpp): - MUL_MAT, SET_ROWS, CPY supports_op - pipeline_dequant, pipeline_set_rows, pipeline_cpy_quant_f32 - Specialized MUL_MAT_VEC with forced subgroup workgroup size Tests: - test_set_rows_tq4_1s: SET_ROWS round-trip validation * vulkan: add fused mul_mat_vec kernel for TQ4_1S Adds a specialised MUL_MAT_VEC shader for TQ4_1S weights so the per-decode-step matrix-vector product no longer has to dequant the full weight tensor to f16 and then go through the generic matmul path. The kernel pre-rotates the activation via a forward Walsh-Hadamard Transform in shared memory and dot-products against the raw centroid*scale stored weights, folding the inverse-WHT on the weight side into the activation by the symmetry H = H^T. Math: w[k] = sign[k] * INV_SQRT32 * (H @ stored)[k] sum_k w[k] * a[k] = INV_SQRT32 * sum_j stored[j] * (H @ (sign * a))[j] Portability choices: - Workgroup size is pinned to 32 threads regardless of the DMMV_WG_SIZE bucket the rest of the mul_mat_vec family picks for the current architecture. The butterfly operates on 32-element blocks with one element per thread; that contract is fixed by the quantization format, not by the GPU. Earlier revisions used `gl_WorkGroupSize.x` as the stride unit, which silently skipped half the work on Intel drivers that force the subgroup to 16 (tests passed via NMSE tolerance while real inference output was garbage). - Butterfly implementation is shared memory only. A subgroup-shuffle variant (`subgroupShuffleXor`) was prototyped and measured on Intel Arc A380 with Mesa Xe HPG: it ran ~60-85 %% slower than the explicit shared-memory butterfly, because Mesa emulates subgroup shuffles via LDS and ends up doing the same LDS traffic with extra driver overhead. The shared-memory butterfly is correct on every device regardless of subgroup-op support, is the fastest path on every device we can actually measure, and leaves the `pipeline_dequant_mul_mat_vec_f32_f32[w][TQ4_1S]` slot uniform across all DMMV_WG_SIZE buckets. - Reduction is the shared-memory tree reduction (no subgroupAdd), for the same reason: on Intel Arc the subgroupAdd is also LDS-backed and the hybrid reduction path was measurably slower. Future vendor-specific heuristics can switch to the hybrid or pure-subgroup reduction variants on NVIDIA / AMD RDNA if hardware subgroup ops turn out to beat the LDS roundtrip there; the existing reduction modes in `mul_mat_vec_base.glsl` already provide the necessary variants. - NUM_ROWS is 8 so the butterfly cost amortises across 8 output rows per workgroup. Each thread holds one position of each of the 8 weight blocks and pairs them with the shared rotated activation. - `mul_mm` and `flash_attn_cm2` shader generation is skipped for TQ4_1S because it is a weight-only format that never reaches the coopmat2 matmul or the KV cache flash-attention paths. Tests: - `test-backend-ops` MUL_MAT tolerance tightened from 2.0 to 0.01 NMSE so real defects can't hide behind a loose check. - Added Gemma-4 E2B, Qwen, Phi and Llama dimensional coverage (k in {1536, 2048, 2304, 3072, 4096}, m in {256, 1152, 1536, 2048, 5120, 6144}, n in {1..8, 16, 64, 256}). 148 MUL_MAT test cases total. Verification (Intel Arc A380, 6 GB VRAM, Vulkan ANV / Mesa Xe HPG, `llama-bench -p 512 -n 128 -r 3` and `llama-perplexity -c 512 --chunks 20 wiki.test.raw`): | Model | Config | Size | Reduction | PPL Δ | pp512/Q8 | tg128/Q8 | |---------------|---------|----------:|----------:|-------:|---------:|---------:| | Qwen2.5-1.5B | I | 1570→1082 | -31.1% | +4.66% | 53.9% | 107.5% | | Phi-3.5-mini | I | 3873→2839 | -26.7% | +5.36% | 57.6% | 52.8% | | Llama-3.2-3B | hybrid | 3263→2147 | -34.2% | +2.03% | 82.4% | 84.2% | | Llama-3.2-3B | premium | 3263→2577 | -21.0% | +0.98% | 71.3% | 67.3% | Qwen2.5-1.5B is faster than its own Q8_0 baseline with Config I: the compressed model fits in less VRAM, and on a small model the TQ4_1S compute cost is offset by the reduced memory traffic. All four models produce coherent output end-to-end and the reductions line up with the TurboQuant paper's validation matrix (§5.8). The remaining gap to Q8_0 on the bigger models is compute-bound on the A380; it closes further on GPUs with more raw throughput. * vulkan: restructure TQ4_1S inner loop for cross-row smem reuse Splits the dequant+accumulate phase into two sub-loops: 1. Pre-compute w_vals[n] for all NUM_ROWS rows (centroid lookup + scale multiply, reads from weight buffer only). 2. Read the rotated activation from shared memory ONCE per column, then FMA across all rows in a tight register loop. This is the Vulkan analogue of the 'hot loop load dedup' from the CUDA kernel (PR AtomicBot-ai#57 optimisation AtomicBot-ai#2). It makes the shared memory read explicitly loop-invariant across rows, which helps compilers that don't auto-hoist LDS loads out of unrolled loops. Measured effect on Intel Arc A380 (Llama-3.2-3B premium, llama-bench tg128, r=5): 15.50 -> 15.78 t/s (+1.8%, within noise but not a regression). The structure is cleaner regardless and should benefit architectures with higher LDS latency.

AtomicBot-ai#6 MUL_MAT_ID Subgroup (PR ggml-org#15524): - Subgroup ballot operations für MoE-Expert-Selection auf non-coopmat GPUs - 18 Merge-Konflikte in ggml-vulkan.cpp gelöst (HEAD-Features erhalten) - 2 PR-Änderungen übernommen: subgroup_min_size<=16 Check, fused RMS_NORM+MUL - mul_mm.comp: neuer subgroup load_row_ids() Code - vulkan-shaders-gen.cpp: MatMulIdType::SUBGROUP integration AtomicBot-ai#2 Tensor Split Regex (PR ggml-org#24710, open): - 29 std::regex Patterns von const auf static const geändert - Verhindert pro-Token Regex-Recompilation im Tensor-Split-Modus - Befreit ~40% des Decode-Threads auf Multi-GPU-Systemen Build: grün auf Hydra (CUDA)

…AtomicBot-ai#6 revert AtomicBot-ai#6 MUL_MAT_ID Subgroup (PR ggml-org#15524) wurde revertiert: - 23 Merge-Konflikte in 3 Vulkan-Dateien, davon 18 in ggml-vulkan.cpp - Subagent-Lösung beschädigte Shader-Generierung (undefined references) - Erfordert manuelle Portierung mit tiefem Vulkan-Shader-Verständnis - Für spätere Session mit fokussiertem Ansatz AtomicBot-ai#2 Tensor Split Regex (PR ggml-org#24710, open) erfolgreich: - 29 std::regex Patterns von const auf static const geändert - Verhindert pro-Token Regex-Recompilation im Tensor-Split-Modus

Nach AtomicBot-ai#6 Revert waren Reste der Subagent-Konfliktlösung in ggml-vulkan.cpp und vulkan-shaders-gen.cpp die Build-Fehler verursachten. Dateien vom letzten sauberen Stand (ef56b4c) wiederhergestellt. AtomicBot-ai#2 Tensor Split Regex bleibt aktiv (src/llama-model.cpp).

Status nach Solo-Session 05:00-14:00: - AtomicBot-ai#2 Tensor Split Regex ✅ angewendet - AtomicBot-ai#5 GTT Size Tuning ✅ bereits konfiguriert - AtomicBot-ai#4 n-gram Decoding ⏳ verfügbar, Benchmark ausstehend - AtomicBot-ai#1 MTP Logits Copy ❌ 19 Konflikte, skipped - AtomicBot-ai#6 MUL_MAT_ID Subgroup ❌ 23 Konflikte, revertiert - AtomicBot-ai#7 Vulkan FA Refactor ⏭️ verschoben (abhängig von AtomicBot-ai#6) - AtomicBot-ai#9 Vulkan Shmem-Staging ❌ PR closed, manuell portieren

AtomicBot-ai#4 n-gram Decoding Benchmark (E2B, Mars): - Baseline: 39.2 t/s, ngram-mod: 39.1 t/s — kein Speedup - Verfügbar für User, aber kein Default-Speedup auf kleinen Modellen M1 Status: ✅ abgeschlossen (AtomicBot-ai#2✅ AtomicBot-ai#4✅ AtomicBot-ai#5✅ AtomicBot-ai#1❌) M2 Status: ⏳ blockiert (AtomicBot-ai#6❌ AtomicBot-ai#7⏭️ AtomicBot-ai#9❌)

…026-07-12) 4 parallele Subagents (Vulkan/AMD, CUDA/MoE, arXiv, Multi-GPU/Batching) konsolidiert zu 25 neuen Items in ROADMAP: Tier 1 Quick Wins (3 neu): - AtomicBot-ai#31 K-Quant A-Matrix Transpose CM1 (PR ggml-org#22970, +5-15% PP auf Mars) - AtomicBot-ai#32 Pascal L1 Cache Tuning (-Xptxas -dlcm=ca, Styx) - AtomicBot-ai#33 Per-Quant MMVQ/MMQ Batch Threshold (AMD MFMA, Mars/Venus) Tier 2 (7 neu): - AtomicBot-ai#34 UBBoost, AtomicBot-ai#35 Row-Packing DMMV, AtomicBot-ai#36 Auto Param Fitting TP - AtomicBot-ai#37 LFRU Expert Caching, AtomicBot-ai#38 Conf-KV, AtomicBot-ai#39 Talon, AtomicBot-ai#40 MoE Load Balancing Tier 3 (5 neu): - AtomicBot-ai#41 GRKV, AtomicBot-ai#42 CapKV, AtomicBot-ai#43 SliderQuant, AtomicBot-ai#44 Alloc-MoE, AtomicBot-ai#45 CUDA Streams QKV Tier 4 (10 neu): - AtomicBot-ai#46-55: SpecMD, QUICK, FluxMoE, STAR-KV, VQKV, CompilerKV, SliceMoE, MoBiE, DASH-Q, GOOSE 7 PRs als bereits im Fork identifiziert (nicht erneut vorschlagen): ggml-org#21472, ggml-org#23764, ggml-org#22299, ggml-org#21611, ggml-org#22423, ggml-org#18749, Warp Shuffle, Constant Memory

P1 AtomicBot-ai#1: /cancel false-positive bei nicht-existierender task_id - METRICS-Check jetzt für ALLE task_ids (nicht nur bei leerem Body) - task_id wird gegen laufende Slots validiert vor Cancel-Post - Nicht-existierende task_id → {cancelled: false, error: 'task not found'} - Leerer Body ohne laufende Tasks → {cancelled: false, message: 'no running tasks'} - Zusätzlich: ältester Task (nach start_time) statt niedrigster Slot-Index P1 AtomicBot-ai#2: ggml_backend_cuda_device_reset thread-safety - device_mutex Lock hinzugefügt (wie ggml_backend_cuda_device_get_memory) - active_count > 0 → Reset verweigert (verhindert Context-Crash) - cudaGetLastError-Details in GGML_LOG_WARN P1 AtomicBot-ai#3: Test-Skript — Cancel-Wirkung verifiziert - Test 3: Stream muss abgebrochen sein (aborted=True oder wenige chunks) - Test 3b neu: nicht-existierende task_id → cancelled=false + error - Test akzeptiert nicht mehr normal beendeten Stream als Erfolg P1 AtomicBot-ai#4: dev_reset Rückgabewert nicht ignorieren - SRV_WRN bei fehlgeschlagenem Reset mit Device-Name P2 AtomicBot-ai#7: proxy_post try/catch bei leerem/ungültigem Body - Statt 500-Exception → 400 'Invalid JSON body' - res_err() statt nicht-existenter .error() Methode P2 AtomicBot-ai#8: Ältester Task statt niedrigster Slot-Index (in P1 AtomicBot-ai#1 fix enthalten)

…ging, cstdint P1 AtomicBot-ai#1: start_time Feld in server_slot::to_json() hinzugefügt - t_start_process_prompt (int64_t, Mikrosekunden) als 'start_time' exportiert - post_cancel liest jetzt int64_t statt int — kein Überlauf mehr - Ältester-Task-Auswahl funktioniert jetzt tatsächlich (vorher immer 0 → erster Slot) P1 AtomicBot-ai#2: dev_reset failure — Kommentar erklärt warum Sleep trotzdem betreten wird - Modell ist bereits zerstört wenn Reset fehlschlägt → Abbruch = unrecoverable - load_model() beim Aufwecken wird OOM-failen wenn VRAM noch belegt → geloggt P1 AtomicBot-ai#3: cudaSetDevice Fehlerlogging mit cudaGetLastError() korrigiert - Return-Wert von cudaSetDevice prüfen, cudaGetLastError() für Fehlermeldung - Vorher: cudaGetLastError() konnte veralteten/no-error Zustand loggen P2 AtomicBot-ai#4: <cstdint> explizit inkludiert für INT64_MAX (portabilität)

…ne, invalidate P1 AtomicBot-ai#1: Set-assoc lookup erkennt jetzt queued-Slots (key match bei queued=true) → kein Duplikat-Insert mehr im selben Set. Backfill entsprechend angepasst. P1 AtomicBot-ai#2: Workload window reset off-by-one — reset nach wsize Aufrufen, nicht am Start des wsize-ten Aufrufs. P1 AtomicBot-ai#4: invalidate() cleart jetzt set_lru_head/tail/n_used und window_count in allen Pools (wie trim() es schon tut). P1 AtomicBot-ai#5: Kommentar 'Aufrunden' → 'Abrunden' (korrigiert). Review: review-swe Subagent, 5 P1 + 7 P2 Issues gefunden. P1 AtomicBot-ai#3 (pool-init locking) ist bestehendes Problem, nicht neu.

… blocker AtomicBot-ai#2) E3 re-run discovery (2026-08-02, I2-FABRIC-E3-FIX re-run): the client scheduler hands the facade per-split graph VIEWS (ggml_graph_view: n_leafs=0, leafs=NULL, srcs reference tensors outside the view). The inner sched's gallocr sizes its hash table from n_nodes+n_leafs, so a view overflows on insertion -> GGML_ABORT at ggml-impl.h:319 on the first facade graph_compute (reproduced in llama-cli and llama-server). Fix: build a self-contained graph copy in the facade (same node pointers, every src-of-node listed as a leaf) before the inner sched runs (fabric_build_self_contained; cached ggml_context grown on demand). Also floor the inner sched's graph_size at 16384 nodes — the first facade graph is a tiny 1-node split view and would starve the sched's own hash table for the real decode graphs. Verified: the model now loads through the facade past the first compute (the gallocr crash is gone). This does NOT unblock E3 — see the second blocker: the facade presents ONE buffer type, so the model loader allocates ALL weights as one buffer on one server (server log: 'allocating 14138.10 MiB on device 0' for the -ngl 28 share; 21.5 GiB single alloc for -ngl 999). The 3060 Ti cannot receive weights; the designed 2-server layer split is unexecutable with the current wiring. E3 verdict: BLOCKED/ INCONCLUSIVE (structural) — see benchmarks/fabric-e3-fix-rerun.md.

github-actions Bot added Apple Metal ggml labels Apr 2, 2026

Vect0rM merged commit 7c01058 into feature/turboquant-kv-cache Apr 2, 2026
11 of 45 checks passed

Ooooze mentioned this pull request May 8, 2026

Repro: MTP path on CUDA aborts at fattn.cu:109 (DKQ=512) for Gemma 4 — Blackwell sm_120 + Ampere sm_86 #5

Draft

WillowOneVision mentioned this pull request May 21, 2026

Phase C.2 dispatch behavior: MTP+mmproj coexistence behind --allow-mtp-with-mmproj (5th first-in-world) #19

Open

9 tasks

Navigation Menu

Uh oh!

feat: add additional TurboQuant kernel templates for enhanced flash a… - #2

feat: add additional TurboQuant kernel templates for enhanced flash a…#2
Vect0rM merged 1 commit into
feature/turboquant-kv-cachefrom
feature/sdf

Ooooze commented Apr 2, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Ooooze commented Apr 2, 2026

Overview

Additional information

Requirements

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants