common : inhibit lazy grammar sampler while reasoning is active (#20970) by Ooooze · Pull Request #1 · AtomicBot-ai/atomic-llama-cpp-turboquant

Ooooze · 2026-03-30T19:53:22Z

common : inhibit grammar while reasoning budget is active
cont : update force_pos in accept
cont : fix tests
cont : tweak should apply logic
cont : return early not using grammar sampler
Add tests
cont : prevent backend sampling when reasoning budget enabled
cont : fix typo

Overview

Additional information

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure:

Automatically creates a prerelease with the macOS ARM64 binary on every push to feature/turboquant-kv-cache. Made-with: Cursor

Without target_commitish, softprops/action-gh-release creates tags on the default branch (master) instead of the triggering branch. Made-with: Cursor

Without -DLLAMA_BUILD_BORINGSSL=ON, cmake picks up Homebrew OpenSSL and links dynamically → Team ID mismatch on codesigned macOS apps. Changes: - Add -DLLAMA_BUILD_BORINGSSL=ON (static SSL, no dynamic dependency) - Add -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON (apply rpath at build time) - Switch to -DCMAKE_INSTALL_RPATH='@loader_path' (consistent with release.yml) - Add -DLLAMA_BUILD_TOOLS=ON - Add verification step: otool -L check fails CI if dynamic SSL found Made-with: Cursor

LLAMA_BUILD_BORINGSSL doesn't exist in this fork's CMakeLists.txt — the flag was silently ignored, binary still linked Homebrew OpenSSL. Correct approach: disable curl and OpenSSL entirely, build all libs statically. Produces a single self-contained binary with only system dylibs (libSystem, libc++, Metal frameworks). - BUILD_SHARED_LIBS=OFF — links libllama, libggml etc. statically - LLAMA_CURL=OFF — no curl dependency, no HF model download - LLAMA_OPENSSL=OFF — no OpenSSL/crypto dependency - hw.ncpu instead of hw.logicalcpu (correct macOS sysctl key) - Verification step: fail CI if any non-system dylib found Made-with: Cursor

…-org#20970) * common : inhibit grammar while reasoning budget is active * cont : update force_pos in accept * cont : fix tests * cont : tweak should apply logic * cont : return early not using grammar sampler * Add tests * cont : prevent backend sampling when reasoning budget enabled * cont : fix typo --------- Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>

Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed #3 (TURBO_D). #1 and #2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Complete experiment log: #1 4-mag LUT: 15.1 at 8K (BEST, +38%) #2 Batched extract: 13.7 (+25%) #3 Inline FA block: 13.5 (I-cache pressure) #4 Deferred norm: 12.9 (loses ILP) #5 2-pair half2: 12.0 (ternary overhead) #6 Select chain: 11.9 (branches kill) #7 Bit-arithmetic: 11.6 (ALU too heavy) #8 FMA branchless: 11.4 (ALU still too heavy) #9 Named-reg ternary: 10.3 (branches worst) #10 Main (8-LUT): 10.95 (baseline) #11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai

* oai moe * compat with new checkpoint * add attn sink impl * add rope scaling yarn * logits match with latest transformers code * wip chat template * rm trailing space * use ggml_scale_bias * rm redundant is_swa_all * convert interleaved gate_up * graph : fix activation function to match reference (AtomicBot-ai#7) * vocab : handle o200k_harmony special tokens * ggml : add attention sinks support (AtomicBot-ai#1) * llama : add attn sinks * ggml : add attn sinks * cuda : add attn sinks * vulkan : add support for sinks in softmax remove unnecessary return * ggml : add fused swiglu_oai op (AtomicBot-ai#11) * ggml : add fused swiglu_oai op * Update ggml/src/ggml-cpu/ops.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * update CUDA impl * cont : metal impl * add vulkan impl * test-backend-ops : more test cases, clean up * llama : remove unfused impl * remove extra lines --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: slaren <slarengh@gmail.com> * repack mxfp4 upon conversion * clean up a bit * enable thinking * add quick hack to render only some special tokens * fix bf16 conversion * remove vocab hack * webui ok * support chat parsing for gpt-oss * fix webui * direct mapping mxfp4, FINALLY * force using mxfp4 * properly use lazy tensor * ggml : add mxfp4 ggml : use e8m0 conversion instead of powf Co-authored-by: Diego Devesa <slarengh@gmail.com> change kvalues_mxfp4 table to match e2m1 (AtomicBot-ai#6) metal : remove quantization for now (not used) cuda : fix disabled CUDA graphs due to ffn moe bias vulkan : add support for mxfp4 cont : add cm2 dequant * ggml : add ggml_add_id (AtomicBot-ai#13) * ggml : add ggml_add_id * add cuda impl * llama : add weight support check for add_id * perf opt * add vulkan impl * rename cuda files * add metal impl * allow in-place ggml_add_id * llama : keep biases on CPU with --cpu-moe * llama : fix compile error ggml-ci * cuda : add fallback for __nv_cvt_e8m0_to_bf16raw ggml-ci * cleanup ggml-ci * sycl : fix supports_op for MXFP4 ggml-ci * fix Unknown reasoning format * ggml-cpu : fix AVX build ggml-ci * fix hip build ggml-ci * cuda : add mxfp4 dequantization support for cuBLAS ggml-ci * ggml-cpu : fix mxfp4 fallback definitions for some architectures ggml-ci * cuda : fix version required for __nv_cvt_e8m0_to_bf16raw --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: slaren <slarengh@gmail.com>

…gml-org#16038) Initalizing RESERVED_NAME in is_reserved_name() is not thread safe and leads to corrupted memory when used from multiple threads as can be seen in the asan trace below. This fixes the initialization to make it thread-safe. #0 0x000100abd018 in std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, void*>*>, bool> std::__1::__hash_table<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>::__emplace_unique_key_args<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) __hash_table:1565 AtomicBot-ai#1 0x000100ab0320 in SchemaConverter::visit(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) json-schema-to-grammar.cpp:802 AtomicBot-ai#2 0x000100aafc48 in std::__1::__function::__func<build_grammar(std::__1::function<void (common_grammar_builder const&)> const&, common_grammar_options const&)::$_2, std::__1::allocator<build_grammar(std::__1::function<void (common_grammar_builder const&)> const&, common_grammar_options const&)::$_2>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&)>::operator()(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&) function.h:319 AtomicBot-ai#3 0x000100a2c938 in std::__1::__function::__func<common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool)::$_0::operator()(common_grammar_builder const&) const::'lambda'(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&), std::__1::allocator<common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool)::$_0::operator()(common_grammar_builder const&) const::'lambda'(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&)>, void (nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&)>::operator()(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&) function.h:319 AtomicBot-ai#4 0x000100a139f8 in foreach_function(nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&, std::__1::function<void (nlohmann::json_abi_v3_12_0::basic_json<nlohmann::json_abi_v3_12_0::ordered_map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::json_abi_v3_12_0::adl_serializer, std::__1::vector<unsigned char, std::__1::allocator<unsigned char>>, void> const&)> const&) chat.cpp:762 AtomicBot-ai#5 0x000100a2a7f4 in std::__1::__function::__func<common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool)::$_0, std::__1::allocator<common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool)::$_0>, void (common_grammar_builder const&)>::operator()(common_grammar_builder const&) function.h:319 AtomicBot-ai#6 0x000100aa98f4 in build_grammar(std::__1::function<void (common_grammar_builder const&)> const&, common_grammar_options const&) json-schema-to-grammar.cpp:982 AtomicBot-ai#7 0x0001009c9314 in common_chat_params_init_llama_3_x(minja::chat_template const&, templates_params const&, bool) chat.cpp:1110 AtomicBot-ai#8 0x0001009b8afc in common_chat_templates_apply_jinja(common_chat_templates const*, common_chat_templates_inputs const&) chat.cpp:1992 AtomicBot-ai#9 0x0001009b533c in common_chat_templates_apply(common_chat_templates const*, common_chat_templates_inputs const&) chat.cpp:2074 AtomicBot-ai#10 0x000100810120 in llamacpp_apply_chat_template+0x724 (predict_oai-98384e17fb94e863:arm64+0x100090120) ... ==45482==Register values: x[0] = 0x00006020004147f8 x[1] = 0x00006080000013c8 x[2] = 0x0000000000000000 x[3] = 0x0000604006289738 x[4] = 0x0000000000000002 x[5] = 0x0000000000000001 x[6] = 0x04034000004b4000 x[7] = 0x0000000000000001 x[8] = 0xbebebebebebebebe x[9] = 0x17d7d7d7d7d7d7d7 x[10] = 0x00000c04000828ff x[11] = 0x0000000000000001 x[12] = 0x000000002018d383 x[13] = 0x0000000000000000 x[14] = 0xfa0000000000fafa x[15] = 0x000010700001ffff x[16] = 0x000000019dc012c0 x[17] = 0x00000001021284f8 x[18] = 0x0000000000000000 x[19] = 0x00000001700acdc0 x[20] = 0x0000000000000002 x[21] = 0x000000002018d384 x[22] = 0x16dd16fd2e731151 x[23] = 0x0000007000020000 x[24] = 0x0000000100c69c08 x[25] = 0x0000000100c69c20 x[26] = 0x00006080000013c7 x[27] = 0x0000000100c69c00 x[28] = 0x00000001700acd60 fp = 0x00000001700aceb0 lr = 0x0000000100abce30 sp = 0x00000001700acd60 AddressSanitizer can not provide additional info. SUMMARY: AddressSanitizer: SEGV __hash_table:1565 in std::__1::pair<std::__1::__hash_iterator<std::__1::__hash_node<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, void*>*>, bool> std::__1::__hash_table<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>::__emplace_unique_key_args<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) Thread T5 created by T0 here: #0 0x0001020b99d4 in pthread_create+0x5c (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x359d4) AtomicBot-ai#1 0x000100873910 in std::sys::pal::unix::thread::Thread::new::h77254fdd87a28e05+0x118 (predict_oai-98384e17fb94e863:arm64+0x1000f3910) AtomicBot-ai#2 0x0001007c7a1c in test::run_test::haeb3c2bcd5ed6cf6+0x76c (predict_oai-98384e17fb94e863:arm64+0x100047a1c) AtomicBot-ai#3 0x0001007aedb0 in test::console::run_tests_console::he9d142d704f3a986+0x149c (predict_oai-98384e17fb94e863:arm64+0x10002edb0) AtomicBot-ai#4 0x0001007c5758 in test::test_main::hf86a5e20735245b9+0x118 (predict_oai-98384e17fb94e863:arm64+0x100045758) AtomicBot-ai#5 0x0001007c5da0 in test::test_main_static::h61ee9c8fd30abca0+0x54 (predict_oai-98384e17fb94e863:arm64+0x100045da0) ... ==45482==ABORTING

…-ai#29 Codex post-commit review found: 1. TURBO_D was QK_TURBO3 (now 32) — broke turbo4 C array sizes 2. SET_ROWS kernel turbo3-specific but instantiated for turbo4 3. Tail block drop for non-128 head dims Fixed AtomicBot-ai#3 (TURBO_D). AtomicBot-ai#1 and AtomicBot-ai#2 don't affect turbo3+dk128 path. Co-Authored-By: tturney@psyguard.ai Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Complete experiment log: AtomicBot-ai#1 4-mag LUT: 15.1 at 8K (BEST, +38%) AtomicBot-ai#2 Batched extract: 13.7 (+25%) AtomicBot-ai#3 Inline FA block: 13.5 (I-cache pressure) AtomicBot-ai#4 Deferred norm: 12.9 (loses ILP) AtomicBot-ai#5 2-pair half2: 12.0 (ternary overhead) AtomicBot-ai#6 Select chain: 11.9 (branches kill) AtomicBot-ai#7 Bit-arithmetic: 11.6 (ALU too heavy) AtomicBot-ai#8 FMA branchless: 11.4 (ALU still too heavy) AtomicBot-ai#9 Named-reg ternary: 10.3 (branches worst) AtomicBot-ai#10 Main (8-LUT): 10.95 (baseline) AtomicBot-ai#11 Non-vec FA: 10.2 (wrong kernel) Ceiling: 24.5 (no dequant) Apple8 hardware truth: 1 divergent constant read < 7 ALU ops (even with fma) Branches cost MORE than divergent constant reads Array indexing ALWAYS spills on Metal 4 constant addresses is the sweet spot The 4-mag LUT is the dequant-level ceiling on Apple Silicon. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-Authored-By: tturney@psyguard.ai

Status nach Solo-Session 05:00-14:00: - AtomicBot-ai#2 Tensor Split Regex ✅ angewendet - AtomicBot-ai#5 GTT Size Tuning ✅ bereits konfiguriert - AtomicBot-ai#4 n-gram Decoding ⏳ verfügbar, Benchmark ausstehend - AtomicBot-ai#1 MTP Logits Copy ❌ 19 Konflikte, skipped - AtomicBot-ai#6 MUL_MAT_ID Subgroup ❌ 23 Konflikte, revertiert - AtomicBot-ai#7 Vulkan FA Refactor ⏭️ verschoben (abhängig von AtomicBot-ai#6) - AtomicBot-ai#9 Vulkan Shmem-Staging ❌ PR closed, manuell portieren

AtomicBot-ai#4 n-gram Decoding Benchmark (E2B, Mars): - Baseline: 39.2 t/s, ngram-mod: 39.1 t/s — kein Speedup - Verfügbar für User, aber kein Default-Speedup auf kleinen Modellen M1 Status: ✅ abgeschlossen (AtomicBot-ai#2✅ AtomicBot-ai#4✅ AtomicBot-ai#5✅ AtomicBot-ai#1❌) M2 Status: ⏳ blockiert (AtomicBot-ai#6❌ AtomicBot-ai#7⏭️ AtomicBot-ai#9❌)

rocprofv3 --kernel-trace works with GGML_CUDA_GRAPHS=0 (HIP tracing was the SIGABRT crash root cause). Per-kernel breakdown of the 6,843us SLOW step GPU compute window: MatMul 55.4% (q6_K 35.2%, iq4_xs 8.0%, q8_0 6.1%) Element-wise 8.6% Data Movement 8.4% Normalization 7.9% Quantization 7.9% MoE Routing 4.0% Flash Attn 3.4% SSM 2.4% RoPE 1.2% Key findings: q6_K matmul is the AtomicBot-ai#1 optimization target at 35.2%. SSM layers are only 2.4% (revised D7.4 skip-SSM upper bound: 10-15% TG, not +75%). MoE routing is surprisingly cheap at 4.0% despite 256 experts. Flash Attention already well-optimized at 3.4%. Assisted-by: Grok

… dense 4 models profiled with rocprofv3 --kernel-trace: Qwen3.6-35B-MTP-Q6_K (MatMul 55%, Attn 3.4%, SSM 2.4%, MoE 4%) Qwen3.6-35B-MTP-Q4_K (MatMul 61%, Attn 2.8%, SSM 2.2%, MoE 3.5%) Qwen3.5-35B-Q4_K_M (MatMul 60%, Attn 2.9%, SSM 2.1%, MoE 3.6%) Gemma4-12B-dense-Q4KM (MatMul 76%, Attn 9.5%, SSM 0%, MoE 0%) Key findings: - MatMul always dominates (55-76%), q6_K matmul is AtomicBot-ai#1 target - Q4_K matmul 22-29% faster per-call than Q6_K (16us vs 21us) - Dense models spend 3x more GPU on attention (9.5% vs 3%) - SSM layers 10x cheaper than attention (10us vs 38us per layer) - MoE routing is cheap (3.5-4%), expert FFN compute in matmul - MTP gives +74% TPS at similar per-step GPU cost - Gemma4-26B/31B exceed 8GB RPC GPU, can't profile on dual-GPU Assisted-by: Grok

… maps (UAF, state leaks) P0 AtomicBot-ai#1 (Use-After-Free): invalidate() cleared only role_base but left blk_down_base/blk_down_kb/blk_pair_pool/blk_down_pool stale. After a host buffer free, moe_cache_backfill_next() would build jobs reading from freed memory. Now clears ALL per-blk pointers that point into the freed range and resets pool indices to -1 for affected blocks. P1 AtomicBot-ai#3 (State leaks on model unload): glu_learn, learn_gate_dst, learn_up_dst, redirect, g_disc (seen/pending/stable_count/any_repeat) and backfill cursor survived invalidate() — a new model loaded at the same addresses would hit stale learned entries and pool decisions. All cleared now. P2 AtomicBot-ai#7: removed unused 'self' variable in moe_cache_begin(). Added comment documenting the owner-lock limitation (P1 AtomicBot-ai#4: not reset on model reload — acceptable for one-model-per-process, the common case).

The runtime maps rpc://...#0 to 3070 and AtomicBot-ai#1 to 3090, opposite of the intended assignment. All plans swapped so MoE layers (L3-6, L13-15) now go to 3070 and FA layers (L7, L16-19) to 3090. Assisted-by: Grok

P1 AtomicBot-ai#1: /cancel false-positive bei nicht-existierender task_id - METRICS-Check jetzt für ALLE task_ids (nicht nur bei leerem Body) - task_id wird gegen laufende Slots validiert vor Cancel-Post - Nicht-existierende task_id → {cancelled: false, error: 'task not found'} - Leerer Body ohne laufende Tasks → {cancelled: false, message: 'no running tasks'} - Zusätzlich: ältester Task (nach start_time) statt niedrigster Slot-Index P1 AtomicBot-ai#2: ggml_backend_cuda_device_reset thread-safety - device_mutex Lock hinzugefügt (wie ggml_backend_cuda_device_get_memory) - active_count > 0 → Reset verweigert (verhindert Context-Crash) - cudaGetLastError-Details in GGML_LOG_WARN P1 AtomicBot-ai#3: Test-Skript — Cancel-Wirkung verifiziert - Test 3: Stream muss abgebrochen sein (aborted=True oder wenige chunks) - Test 3b neu: nicht-existierende task_id → cancelled=false + error - Test akzeptiert nicht mehr normal beendeten Stream als Erfolg P1 AtomicBot-ai#4: dev_reset Rückgabewert nicht ignorieren - SRV_WRN bei fehlgeschlagenem Reset mit Device-Name P2 AtomicBot-ai#7: proxy_post try/catch bei leerem/ungültigem Body - Statt 500-Exception → 400 'Invalid JSON body' - res_err() statt nicht-existenter .error() Methode P2 AtomicBot-ai#8: Ältester Task statt niedrigster Slot-Index (in P1 AtomicBot-ai#1 fix enthalten)

…ging, cstdint P1 AtomicBot-ai#1: start_time Feld in server_slot::to_json() hinzugefügt - t_start_process_prompt (int64_t, Mikrosekunden) als 'start_time' exportiert - post_cancel liest jetzt int64_t statt int — kein Überlauf mehr - Ältester-Task-Auswahl funktioniert jetzt tatsächlich (vorher immer 0 → erster Slot) P1 AtomicBot-ai#2: dev_reset failure — Kommentar erklärt warum Sleep trotzdem betreten wird - Modell ist bereits zerstört wenn Reset fehlschlägt → Abbruch = unrecoverable - load_model() beim Aufwecken wird OOM-failen wenn VRAM noch belegt → geloggt P1 AtomicBot-ai#3: cudaSetDevice Fehlerlogging mit cudaGetLastError() korrigiert - Return-Wert von cudaSetDevice prüfen, cudaGetLastError() für Fehlermeldung - Vorher: cudaGetLastError() konnte veralteten/no-error Zustand loggen P2 AtomicBot-ai#4: <cstdint> explizit inkludiert für INT64_MAX (portabilität)

* Sycl tp stage1 (AtomicBot-ai#1) * SYCL: tensor parallelism (--split-mode tensor) for dual-GPU Adds the comm_init/comm_free/comm_allreduce_tensor trio that the meta-backend queries via get_proc_address to enable backend-specific all-reduce, mirroring the pattern used by ggml-cuda.cu. For N=2 (the common dual-GPU case) implements a degenerate ring all-reduce with two size-branched paths: * Small (nelem < 32768): FP32 direct memcpy + per-device ADD kernel chained via depends_on(memcpy_event). 4 SYCL submissions/call. * Large (nelem >= 32768): BF16-compressed. Each device compresses FP32 -> BF16 in a local outbox, cross-device memcpys to the peer's inbox (HALF the PCIe bytes), then decompresses + adds into the local FP32 partial. 6 SYCL submissions/call but PCIe bytes halved -- wins for any tensor where PCIe dominates kernel time. Threshold and BF16 path pattern mirror the CUDA NCCL allreduce. Storage: ONE persistent uint8_t buffer per device, 4 * nelem bytes (matches both path layouts: FP32 nelem floats; BF16 outbox+inbox = 2 * nelem uint16_t each). Single alloc+free per device keeps the SYCL pool's strict-LIFO invariant trivial. Initial impl handles N=2 FP32 contiguous tensors. Other cases return false, causing the meta-backend to use its generic butterfly fallback. Per-call sync is intentionally omitted. SYCL in-order queue semantics ensure that the meta-backend's next compute on the same per-device queue waits for our final ADD, and the next allreduce's first op on the same persistent buffer waits via the same queue. Only comm_free does an explicit final wait. OneCCL is NOT used: OneCCL 2021.17 hardcodes single-device-per-process in communicator_impl.hpp:47 (condition devices.size() == 1), which is incompatible with llama.cpp's single-process multi-GPU model. Measured on dual Intel Arc Pro B70 (NEO 26.05.x, oneAPI 2025.3 + DPC++ nightly): Llama-3.3-70B Q4_K_M, -sm tensor -fa 1 -ctk f16 -ctv f16: pp512 = 377.08 t/s (vs 313.65 layer mode = +20.2%) tg128 = 17.40 t/s (vs 9.74 layer mode = +78.6%) Qwen3-Coder-Next-80B-A3B Q3_K_M (MoE): pp512 = 216.56 t/s (vs 156.58 meta-backend butterfly = +38.3%) tg128 = 17.60 t/s (vs 14.31 meta-backend butterfly = +23.0%) Qwen3-4B Q4_K_M: pp64 = 984.51 t/s, tg16 = 49.29 t/s Llama-3.3-70B in SYCL TP now comfortably beats production layer mode on both prefill and decode. Coder-Next-80B-A3B (MoE) also wins on both — the BF16 path is what unlocks the many-medium-allreduces prefill pattern. Build/CMake: no changes. No new dependencies. ~210 lines added across ggml-sycl.h and ggml-sycl.cpp. * Fix comments * documentation update to address PR feedback * Bring over my device-to-device memcpy chagnes * move the dev2dev_memcpy calls to the upstream 7-parameter variety * Fix a typo and remove a trailing whitespace

Derive RPC-split uids from graph topology (ops + shapes + connectivity) instead of the global monotonic counter, so identical-topology rebuilds within the MTP cycle carry the same uid and the client reuses the server-cached graph via async GRAPH_RECOMPUTE instead of blocking GRAPH_COMPUTE every token. - T3a: ggml_graph_topology_hash() (FNV-1a 64-bit) in ggml.c; RPC-only stable-uid split assignment in ggml-backend.cpp gated by GGML_RPC_STABLE_UID=1 (default OFF, F8 fix AtomicBot-ai#1). Local splits keep the monotonic counter (F10 CUDA-graph guard). - T3b: document the CUDA-graph uid-replay invariant in ggml-cuda.cu. - T3c: bound seen_graph_uids to <=32 entries/device (clear-on-overflow, F9) and add per-device recompute hit/miss counters emitted via GGML_RPC_DEBUG for AC2 measurement. Co-Authored-By: Grok <<EMAIL>>

…ne, invalidate P1 AtomicBot-ai#1: Set-assoc lookup erkennt jetzt queued-Slots (key match bei queued=true) → kein Duplikat-Insert mehr im selben Set. Backfill entsprechend angepasst. P1 AtomicBot-ai#2: Workload window reset off-by-one — reset nach wsize Aufrufen, nicht am Start des wsize-ten Aufrufs. P1 AtomicBot-ai#4: invalidate() cleart jetzt set_lru_head/tail/n_used und window_count in allen Pools (wie trim() es schon tut). P1 AtomicBot-ai#5: Kommentar 'Aufrunden' → 'Abrunden' (korrigiert). Review: review-swe Subagent, 5 P1 + 7 P2 Issues gefunden. P1 AtomicBot-ai#3 (pool-init locking) ist bestehendes Problem, nicht neu.

Derive RPC-split uids from graph topology (ops + shapes + connectivity) instead of the global monotonic counter, so identical-topology rebuilds within the MTP cycle carry the same uid and the client reuses the server-cached graph via async GRAPH_RECOMPUTE instead of blocking GRAPH_COMPUTE every token. - T3a: ggml_graph_topology_hash() (FNV-1a 64-bit) in ggml.c; RPC-only stable-uid split assignment in ggml-backend.cpp gated by GGML_RPC_STABLE_UID=1 (default OFF, F8 fix AtomicBot-ai#1). Local splits keep the monotonic counter (F10 CUDA-graph guard). - T3b: document the CUDA-graph uid-replay invariant in ggml-cuda.cu. - T3c: bound seen_graph_uids to <=32 entries/device (clear-on-overflow, F9) and add per-device recompute hit/miss counters emitted via GGML_RPC_DEBUG for AC2 measurement. Co-Authored-By: Grok <<EMAIL>>

Vect0rM and others added 5 commits March 30, 2026 12:34

ci: add GitHub Release with downloadable tar.gz archive

a60b5bd

Automatically creates a prerelease with the macOS ARM64 binary on every push to feature/turboquant-kv-cache. Made-with: Cursor

fix: set target_commitish in release action to correct branch

c6b5330

Without target_commitish, softprops/action-gh-release creates tags on the default branch (master) instead of the triggering branch. Made-with: Cursor

github-actions Bot added testing examples server labels Mar 30, 2026

Vect0rM force-pushed the feature/turboquant-kv-cache branch from 6879672 to 7b8820c Compare March 31, 2026 08:21

Vect0rM merged commit d785414 into feature/turboquant-kv-cache Mar 31, 2026
8 of 44 checks passed

WillowOneVision mentioned this pull request May 21, 2026

Phase C.2 dispatch behavior: MTP+mmproj coexistence behind --allow-mtp-with-mmproj (5th first-in-world) #19

Open

9 tasks

Navigation Menu

Uh oh!

common : inhibit lazy grammar sampler while reasoning is active (#20970) - #1

common : inhibit lazy grammar sampler while reasoning is active (#20970)#1
Vect0rM merged 5 commits into
feature/turboquant-kv-cachefrom
fix/qwen

Ooooze commented Mar 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Ooooze commented Mar 30, 2026

Overview

Additional information

Requirements

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants