Rebase b10549 - #217
Merged
Merged
Rebase b10549#217
Conversation
The supports_op check for GGML_OP_UPSCALE rejected the BILINEAR|ANTIALIAS mode, forcing the Qwen3-VL / SigLIP vision graph's position-embedding interpolation onto the CPU and splitting the compute graph. Antialiasing only affects downsampling; when the destination grid is >= the source grid (the upsampling case the vision tower uses) the antialiased result is identical to plain bilinear, which the existing kernel_upscale_bilinear already computes exactly. Relax supports_op to accept BILINEAR + ANTIALIAS when upsampling so the whole vision graph stays on OpenCL (graph splits 3 -> 1). (cherry picked from commit fa2e13d) (cherry picked from commit f6640a9)
…ile, layernorm fusion Three quality-neutral ggml-vulkan optimizations for the Mali-G715 vision projector (mmproj/image encoder), benchmarked on Pixel 9 Pro (Qwen3.5-0.8B): 1. Disable flash attention on GPU projectors without efficient (coopmat) FA (tools/mtmd/clip.cpp). Uses runtime proc_address resolution to query the backend — no compile-time backend dependency. Mali FA_SCALAR ~2.6x less efficient than the matmul path; coopmat-capable GPUs keep FA enabled. 2. Mali/Valhall warptile tuning (ggml-vulkan.cpp, VK_VENDOR_ID_ARM) — large q8_0 MMQ tile to 32-warp/16-wide layout; ~90->~124 GFLOPS/s. Self-disables if shared memory is insufficient. Vendor-wide: also speeds the LLM prefill on GPU. 3. NORM+MUL+ADD (layernorm) Vulkan fusion (norm.comp, generic_binary_head.glsl, vulkan-shaders-gen.cpp, ggml-vulkan.cpp) — one dispatch replaces three; mirrors rms_norm+mul. -26 dispatches/encode. NOT Mali-specific. Result (4-run CPU-matched, profiler-off): within-run GPU/CPU mmproj-encode ratio ~1.46x (baseline) -> ~1.12x (optimized); near-parity 1.09x at high resolution. Quality 37.8%==37.8% (delta=0). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> (cherry picked from commit 3b25e74) --- b10297 rebase: - Squash 163b387: apply the 512-invocation Mali warptile only when the device reports maxComputeWorkGroupInvocations and maxComputeWorkGroupSize[0] >= 512. Squashed-with: 833d681, 163b387 (cherry picked from commit 897ec9c)
…fficient FA) The coopmat-quality FA-gate added in this PR was inert on Mali: Mali-G715 advertises VK_KHR_cooperative_matrix, so coopmat1_fa_support is true and ggml_backend_vk_supports_efficient_fa() returned true — yet Mali's flash attention still runs the slow path (~40 GFLOPS/s vs the ~100 GFLOPS/s matmul path). Per-op profiling on temp-9341/Pixel 9 showed the "optimized" build's clip encode was byte-for-byte identical to baseline (FLASH_ATTN_EXT present, zero SOFT_MAX) — the FA-disable never fired. Fix: - ggml-vulkan.cpp: ggml_backend_vk_supports_efficient_fa() returns false for VK_VENDOR_ID_ARM (Mali coopmat is not fast FA). Real-coopmat desktop GPUs (NVIDIA/AMD/Intel) still report efficient FA and keep it. - clip.cpp: apply the gate on flash_attn_type != DISABLED (the addon enables FA by default, not AUTO-only) and default to disabling FA when the backend can't confirm efficient FA (the safe original behaviour). Verified on temp-9341/Pixel 9 (Mali-G715): FLASH_ATTN_EXT -> SOFT_MAX, GPU clip-encode -30..35% at high resolution; 4-run CPU-matched within-run GPU/CPU mmproj-encode ratio 1.46x (baseline) -> 1.10x (optimized), quality neutral (37.8% == 37.8%) — recovering the original 1.46x->1.12x result. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit b1cab36) (cherry picked from commit d7b00e0)
…ry (Metal/CUDA) The FA-gate defaulted efficient_fa=false, so on non-Vulkan GPU backends (Metal, CUDA) — where ggml_backend_supports_efficient_fa is not implemented — flash attention was wrongly disabled. The resulting explicit-attention clip path overflows the pre-sized compute buffer at high n_pos (image_tile_mode=disabled + image_max_tokens=4096), hitting GGML_ASSERT in ggml-backend.cpp:2043 (SIGABRT) on darwin/iOS Metal integration tests. Fix: default efficient_fa=true and disable only when a backend affirmatively reports non-efficient FA. Only ggml-vulkan implements the query (returns false for VK_VENDOR_ID_ARM / non-coopmat), so the Mali FA-disable win is unchanged; Metal/CUDA/CPU keep their efficient FA. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 1ec3409) (cherry picked from commit b7035ae)
…pCount Review fix (PR tetherto#174): the fused-norm change dispatched GGML_OP_NORM as a direct {ne01, ne02, ne03} grid; on large row counts ne01 can exceed maxComputeWorkGroupCount[0] (spec minimum 65535) and trip the GGML_ASSERT in ggml_vk_dispatch_pipeline, where the previous flattened/tiled dispatch handled arbitrary ggml_nrows. Restore the flattened {512, 512, N} row tiling on the host (same group as SOFT_MAX/SUM_ROWS) and reconstruct {row, channel, sample} in norm.comp from the flat workgroup id (formula shared with soft_max.comp), with a workgroup-uniform bounds return for the tiling round-up. dst offset is unchanged: flat_row == (samp*nchannels + channel)*nrows + row by construction. No behavioural change for in-range shapes; the fusion's dispatch-count reduction is untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 68e6c52) (cherry picked from commit a49fd38)
…udget-aware) The warmup-time hard-disable for GPU projectors without efficient (coopmat) flash attention replaced AUTO/ENABLED with DISABLED, which short-circuited the budget-aware AUTO heuristic in clip_resolve_flash_attn_type(). At high n_pos (image_tile_mode=disabled with image_max_tokens=4096 -> 16384 ViT patches) the forced explicit attention path materializes an O(n^2 * n_head) score matrix, growing RSS to ~12 GB and getting the process lmkd-killed on Pixel 9 Pro (runQwen35ImageTileModeTokensTest). Downgrade to AUTO instead and record the inefficiency in clip_ctx::fa_backend_inefficient, which now also enables the AUTO cutoff default (previously Mali-detection only) so any non-coopmat backend gets the per-image budget decision: explicit attention below the cutoff (fast on scalar-FA GPUs), memory-frugal scalar FA at/above it or when the explicit scratch would not fit device memory. Explicit user DISABLED is still honored, and MTMD_CLIP_AUTO_FA_MIN_KV still overrides the cutoff. (cherry picked from commit cceee22) (cherry picked from commit 752a84b)
…q-chunking) On Galaxy S25 Ultra (Adreno 830, OpenCL) the monolithic 16384-patch ViT encode (image_tile_mode=disabled, image_max_tokens=4096) faults the GPU near the end of the encode (Adreno-GSL log_gpu_snapshot fires before any decode work reaches the device), after which the driver aborts the process from cl_a8x_cmdbuf_mgr_submit_ibs (os_exit) on the next submission. Two unbounded behaviours plausibly drive the fault and both are bounded here: - ggml_backend_opencl_graph_compute enqueued entire graphs (thousands of nodes, ~48 s of GPU work for the failing encode) with no intra-graph flush. Now clFlush every GGML_OPENCL_FLUSH_INTERVAL nodes (default 64, 0 disables) so the GSL command-buffer manager receives bounded batches. clFlush submits without stalling the host. - ggml_cl_flash_attn issued one dispatch covering all q rows; at n_q = n_kv = 16384 every workgroup loops the full KV, making a single very long kernel. Now chunked along q rows at GGML_OPENCL_FA_MAX_NQ rows per dispatch (default 4096, 0 disables) with a clFlush between chunks. The split is exact: the kernel resolves its q row relative to the Q/O/mask base offsets, is_causal is always 0 (masking is explicit) and alibi/sinks depend only on the head index, so shifting the row base via byte offsets while shrinking n_q is mathematically identical. No .cl kernel changes. The 512-token image-chunk decode is not implicated: the S25 VLM benchmark ran 304 full-512-row ubatch decodes cleanly. Only the giant monolithic encode (5-8x beyond anything previously run on this backend) triggers the fault. (cherry picked from commit 82a26df) (cherry picked from commit 74e2122)
…, tests Addresses the pre-merge review findings on the two QVAC-21914 crash-fix commits (P1/P2 performance, C1/C2 correctness, S1/S2 robustness, K nits): - ggml-opencl: gate the periodic graph flush on accumulated estimated WORK (GGML_OPENCL_FLUSH_WORK_MB, default 512 MB) instead of a bare node counter. Per-token LLM decode graphs never reach the budget by construction, so the decode hot path stays submission-free; the 16k-patch encode still flushes dozens of times. Single touch point in graph_compute (no more per-fusion-branch duplication). - ggml-opencl: both tunables move onto ggml_backend_opencl_context, resolved once at init with strtol-based parsing (clamp, warn on garbage instead of silently disabling the mitigation) and GGML_LOG_INFO'd like the file's other env knobs. FA chunking reads the context field. - ggml-opencl: GGML_ASSERT(is_causal == 0) before the FA chunk loop — the kernel's causal-boundary formula needs the TOTAL n_q, so chunks after the first would silently corrupt output if causal FA were ever enabled here; keep the invariant loud. Explicit n_q == 0 guard. - clip: rework the AUTO cutoff memory clamp. Total memory now provides the STABLE fast-path clamp (explicit scratch <= total/4); free memory (a volatile, load-dependent number) may only lower the cutoff further via the hard-fit requirement (scratch <= free), never the old free/2 heuristic that silently pushed normal-size Mali images onto the ~2.6x slower scalar-FA path under momentary memory pressure. No memory info at all now fails SAFE at a conservative 2048-patch cap instead of trusting the raw 4096 default (~3.2 GB scratch at n_head=16). The arithmetic is extracted into clip_fa_effective_min_kv() (pure, exposed via clip.h for tests). - tests: test-clip-fa-cutoff (pure CPU, locks in the fast path, the P2 regression guard, the fail-safe cap and edge cases; passing) and test-opencl-fa-chunking (chunked-vs-CPU numerical parity over unchunked / exact-chunk / partial-last-chunk / n_q==1, masked and unmasked, with GGML_OPENCL_FA_MAX_NQ=64 and a 1 MB flush budget; self-skips without a capable OpenCL device — PoCL lacks FP16, so it executes on Adreno-class hardware). - clip warmup comment: note ggml-opencl also lands in the "no efficient-FA query" bucket and its giant-encode fault is handled by the submission bounding inside that backend. GGML_OPENCL_FLUSH_INTERVAL (node-count knob) is replaced by GGML_OPENCL_FLUSH_WORK_MB; GGML_OPENCL_FA_MAX_NQ semantics unchanged. (cherry picked from commit fc09f36) --- b10297 rebase: - Squash a68b359: test-opencl-fa-chunking: call ggml_backend_load_all() and select the device by backend registry name ("OpenCL") instead of substring-matching the device name. Squashed-with: 649de77, a68b359 (cherry picked from commit c4ea9a4)
The pure AUTO-budget helper was defined out-of-line in clip.cpp and declared in the internal clip.h. On Windows mtmd builds as a shared library exporting only the MTMD_API-decorated public API; the internal clip_* symbols are absent from mtmd.lib, so test-clip-fa-cutoff (the first cross-DLL-boundary consumer of a clip_* symbol) failed to link (LNK2019). Linux/macOS export all default-visibility symbols, so it linked there. Move the function inline into clip.h (with its NO_MEMINFO_CAP constant); the test and clip.cpp both compile their own copy — no DLL export of an internal helper. CLIP_AUTO_FA_MIN_KV_MALI_DEFAULT stays in clip.cpp (its only user). Verified: mtmd + test-clip-fa-cutoff build and the test passes. (cherry picked from commit 99d6042) (cherry picked from commit c59af69)
…uard - parse_env_i64: GGML_LOG_WARN when an in-range-but-too-large GGML_OPENCL_FLUSH_WORK_MB / GGML_OPENCL_FA_MAX_NQ is clamped to max, matching the file's convention of logging every overridden value (previously the clamp was silent). - test-clip-fa-cutoff: the n_head=0 case passed total_mem==free_mem==0, which short-circuits to the NO_MEMINFO cap before any sqrt(.../n_head) branch runs — the div-by-zero guard was never exercised. Pass 16 GB total so the total-memory clamp runs with n_head=0; without the guard the (int)sqrt(x/0) path would now fail the assertion. Both from yingying0906's review; Windows/CPU-only surface, no Android behavior change. (cherry picked from commit 3da3e05) (cherry picked from commit b124606)
The graph_compute work-budget flush ran BEFORE the current node was dispatched: it accounted the node's work, and on crossing the budget flushed (submitting only the prior batch) then reset the counter to 0 — so the crossing node started a fresh batch. A large op, or the last large segment of the graph, could therefore begin an unflushed batch and be submitted unbounded at the implicit end-of-graph finish, defeating the bound. Move the budget check below the dispatch (convert the fused-op continue chain to if/else so every path reaches one touch point), so the node that crosses the budget is part of the flushed batch. Reported by @gianni-cor. (cherry picked from commit 7056f4c) (cherry picked from commit 60eea10)
Non-behavioral cleanups from the PR tetherto#181 review pass: - ggml-opencl graph_compute: correct the flush-cadence comment. The old "per-token decode hot path submission-free by construction" claim was false for multi-GB models — a decode step streams the whole model, so its estimated work crosses the default 512 MB budget a few times per token. Reworded to state that accurately (cost is negligible in practice since clFlush is non-blocking, and it is tunable/zeroable to make decode fully submission-free). Mechanism unchanged. - ggml_cl_flash_attn: GGML_ASSERT(q->ne[1] <= INT32_MAX) before the int n_q truncation, since the q-chunk loop accumulates into an int and derives cl_ulong offsets from it (defensive; not reachable with real shapes). - Consistency: normalize the ticket tag to bare `QVAC-21914` (drop the `qvac ` prefix) in clip.h and tests/CMakeLists.txt, matching the .cpp files and the fork's QVAC-21257 precedent. - tests/CMakeLists.txt: move the unconditional test-opencl-fa-chunking registration up beside test-copy-tbq-subgroups (its self-skipping sibling) instead of sitting right after the LLAMA_MTMD endif() where it read as MTMD-gated; add a comment noting it deliberately does not link mtmd. No functional change to the fix; local mtmd + both tests build, test-clip-fa-cutoff passes. (cherry picked from commit 2b927cf) (cherry picked from commit d983403)
The OpenCL kernels are compiled with -cl-finite-math-only and -cl-fast-relaxed-math, which let the compiler assume no Inf/NaN. The flash-attention online softmax initialises its running max to -INFINITY and masks padded scores with -INFINITY, so finite-math miscompiles the init/masking path. Compile the flash-attention programs with a relaxed option set that drops -cl-fast-relaxed-math, -cl-finite-math-only and -cl-unsafe-math-optimizations (keeping -cl-mad-enable for speed) so the -inf sentinels behave correctly. Also harden the strip: erase every occurrence of each flag (not just the first) and GGML_ASSERT that no finite-math/fast-math/unsafe-math flag survived, so a future compile_opts spelling/spacing change fails loudly at load time instead of silently reintroducing the -INFINITY miscompile. Re-ported onto b9840's rewritten OpenCL flash-attn (upstream PR ggml-org#14987 + follow-ups): the original per-dim kernel-compile loop is gone, so the strip is applied once in ggml_opencl_fa_compile_opts(), the single site every FA variant (F16/F32/F32_F16/Q8_0/Q4_0/PRE and _SPLIT) is compiled through. Squashed re-port of 0cbe362 + c1dace7 (finite-math part). (cherry picked from commit 348a910) (cherry picked from commit 91f4ccd)
The flash-attention dispatch inferred causal masking from shape with `is_causal = (mask == NULL && n_q > 1 && n_q == n_kv)`. A null mask means no masking, i.e. bidirectional attention (the SigLIP vision and embedding encoders), while causal attention always supplies an explicit causal mask in this codebase (llama-graph.cpp build_attn passes a kq_mask filled with -INFINITY). The heuristic therefore wrongly made the bidirectional Qwen3-VL vision tower attend causally, so each patch only saw earlier patches and the image embedding was corrupted. Set is_causal = 0 unconditionally; causality is always expressed via the explicit mask. This cannot regress the LLM, which already passes a real causal mask (is_causal was already 0 for it) and relies on that mask. Document the invariant in ggml_cl_flash_attn: a null mask is treated as bidirectional, so any caller needing causal masking must supply an explicit causal mask rather than relying on shape inference. Re-ported onto b9840's rewritten OpenCL flash-attn; b9840's own q-chunking path already GGML_ASSERTs is_causal == 0, so this is consistent with the existing code. Squashed re-port of 51dbb17 + c1dace7 (is_causal part). (cherry picked from commit 7ae4bc9) (cherry picked from commit e08f280)
…uard upscale zero dims The f32/f16 flash-attention kernels load K/V tiles into local memory, barrier, read them, then loop to overwrite the tiles for the next K/V block without a trailing barrier. Out-of-range lanes (the last partial BLOCK_M block) `continue` past the read and race ahead into the next tile load while active lanes are still reading l_k/l_v. With n_kv > BLOCK_N (e.g. the bidirectional vision tower, n_kv=247) this corrupts the shared tiles. Add a trailing barrier(CLK_LOCAL_MEM_FENCE) at the end of the K/V block loop and guard the score computation with `if (my_query_row < n_q)` instead of an early continue. flash_attn_f32_f16.cl already uses that guard + trailing barrier after b9840's redesign, so it is left unchanged. Also guard zero source dimensions in ggml_cl_upscale: the sf* scale factors divide by the source dims, so a zero source dim yields +inf; the existing early-exit only covered zero destination dims. Re-ported onto b9840's rewritten OpenCL flash-attn (b9840 widened the score unroll to j += 4; only the divergence guard + trailing barrier are re-applied, the body is unchanged). Squashed re-port of dc64397 + b7ad6d4. The barrier fix is a GPU-scheduling race whose only proof is on-device; the original b7ad6d4 validated on S25 Ultra / Adreno 830 (Qwen3-VL GPU vision projector matches CPU exactly, Delta 0.0 pp, ~26% faster on encode). Re-verify on-device before merge. (cherry picked from commit ce54dd5) --- b10297 rebase: - Squash 97a1ecd: flash_attn_f32.cl: apply the divergence guard unconditionally and drop the FA_SG<64-only trailing barrier; the tile race reproduces even on a single 64-wide Adreno subgroup (Adreno 830). Squashed-with: 278014e, 97a1ecd (cherry picked from commit ed29b04)
The upstream target of this former fixup! (50e0ad0, --clear-idle ggml-org#20993) already landed upstream, so this stays a standalone commit. Relying on exact log text is brittle, especially across rebases with upstream changes; use the timings fields instead and drain remaining logs for test cleanliness. (cherry picked from commit 1df9cb1)
A supports_op() regression never fails test-backend-ops: the case falls back to CPU and is reported 'not supported [backend]', i.e. skipped. e09ae0b removed Q4_1/Q4_K from OpenCL MUL_MAT supports_op with zero test failures --- b10297 rebase: - Squash e8fb282: add the UPSCALE f32 bilinear|antialias row to the opencl-pocl op-coverage manifest. Squashed-with: 5587e68, e8fb282 (cherry picked from commit c6e9ea8)
The FA sweep uses kv in {113, 512, 1024} x nb in {1, 3, 32, 75}, so nb
never equals kv and the mask==NULL && n_q==n_kv shape -- exactly what a
ViT self-attention layer produces -- is never exercised. A backend that
infers causality from that shape (OpenCL's is_causal heuristic,
ggml-opencl.cpp:15131) silently computes causal attention for the whole
vision tower and no CI test goes red.
Add explicit bidirectional cases at n_q == n_kv == 247 and 256 for head
sizes 64 and 80 (both in the OpenCL FA supported-dims table). 247 (odd)
additionally leaves partial tiles for any power-of-two tile size; 256 is
the aligned control separating causality bugs from tiling bugs.
Expected red on OpenCL until the is_causal heuristic is removed
(re-port of b9840 7ae4bc9); green on CPU/Vulkan/CUDA/Metal.
Note for landing: order this commit after the is_causal fix so the
series stays bisectable-green on OpenCL hardware.
Assisted-by: Claude (Anthropic AI)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DbaV79zWiPZJq1LrdWGDTj
(cherry picked from commit a0fe46a41d5c8e02f9923049c0a16cbaef8d7439)
(cherry picked from commit 121ceb0)
…riants n_q=33 with n_kv=513 leaves out-of-range query lanes for any pow2 query tile (Adreno OpenCL uses BLOCK_M=64 for dk 64/128) and a 1-valid-row final KV tile (513 = 16*32 + 1), with 17 tile-loop iterations worth of barrier crossings. Tiled kernels must keep out-of-range lanes inside the tile loop for the barriers while excluding them from the score loop; an early continue past the tile barrier (the b9840 ce54dd5 race class) or a missing trailing barrier corrupts the shared K/V tiles for in-range lanes. Cover each KV type separately: backends that specialize kernels per KV type (OpenCL picks flash_attn_f32.cl for f32 KV, f32_f16(+split) for f16 KV since n_kv=513 crosses the split threshold, and the q8_0/q4_0 tiled kernels for quant KV) would otherwise leave those variants untested at this shape. Note flash_attn_f16.cl itself is only reachable with an f16 Q tensor, which test-backend-ops never generates (Q is always f32) -- it stays covered only by code review. The race is scheduling-dependent: in-order devices (pocl) and drivers with cooperative-matrix FA paths will likely pass even with the bug; these shapes make the sweep able to catch it on the affected hardware class (original repro: Adreno 830). Validated: CPU supports all 5 cases; Vulkan RADV 5/5 PASS on 7900 XTX and Raphael iGPU. Assisted-by: Claude (Anthropic AI) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DbaV79zWiPZJq1LrdWGDTj (cherry picked from commit ab6ecf6b4ea23533640988f0557b11c48106bf01) (cherry picked from commit de1fd95)
Add fused-versus-fallback correctness and benchmark cases for HC post and the Lightning Indexer. Assisted-by: GPT-5.6 Sol (cherry picked from commit b5dcb61) --- b10297 rebase: - skip_backend for DSV4_HC_POST_BIT_EXACT matched only the "CUDA" reg name, but HIP builds register the same backend as "ROCm" (GGML_CUDA_NAME), so the FMA-contraction skip never fired there and the bit-exact gate failed deterministically on ROCm (6 cases, ERR=1.0). (cherry picked from commit ddfccd9)
Generate per-K-type generic pipelines and 32/64-head CM1 and CM2 variants, with quant-aware stride handling and guarded pipeline selection. CM1 stages decoded FP16 tiles under shared-memory limits; CM2 uses FP16 decode callbacks for quantized K tiles. Assisted-by: GPT-5.6 Sol (cherry picked from commit d2414ec) (cherry picked from commit a76ed47)
Three related fixes to the memory premises behind common_fit_params, from hardware evidence gathered on a 24 GiB Apple M4 Pro (QVAC-24112): - ggml-cpu: report available memory instead of free = total. On macOS this is physical minus wired and compressor pages; on Linux it is MemAvailable. Reporting total made every host-memory fit trivially pass: an 18 GiB model on a 24 GiB machine "fit" regardless of what was running. - ggml-metal: clamp unified-memory free by system-wide availability. currentAllocatedSize is per-process, so a fresh process (e.g. a fit preflight subprocess) saw an idle device no matter how much other processes had wired. Measured: an 11 GiB resident model in another process did not move the fit verdict at all. - common/fit: consult the host row. The nd == 0 arm budgeted against total host memory; it now uses the available figure. For nd >= 1, devices that share physical memory with the host (Apple silicon Metal, integrated GPUs) are folded together with the host row into one combined budget - the per-device rows cannot see that sum. A host-side deficit also forces context reduction and, for pinned parameters, a FAILURE instead of a silent pass. Availability is deliberately physical minus non-evictable (wired + compressor), not free + inactive: the kernel compresses and evicts anonymous and file-backed pages under pressure, and a free+inactive budget rejects loads that demonstrably run. Known limits, unchanged by this patch: the fitter's own projection runs a few hundred MiB high near the boundary, and macOS memory pressure is bistable - a partial offload can pass every budget and still die at decode once the compressor saturates. Projection cannot be denial-grade on this platform; consumers must verify at load (the llm-llamacpp probe decode does). (cherry picked from commit 2951370)
Addresses all five review findings on the previous commit (qvac-fabric-llm.cpp#214): - The fold was dead code: ggml_dev_shares_host_memory matched registry name "Metal" but the backend registers as "MTL". Replace the string match with a public ggml_backend_dev_props.memory_unified flag set by the backend itself (Metal from MTLDevice.hasUnifiedMemory; memset in ggml_backend_dev_get_props keeps it false elsewhere; Vulkan iGPUs are covered by the existing IGPU type arm). tests/test-unified-memory-props.cpp asserts the flag is alive on Apple silicon, so a dead fold can no longer validate. - Guard both step-2 divisions the host-forced descent made reachable: the context interpolation delta is exactly 0 when device rows are context-independent (n_gpu_layers == 0), and the per-layer estimate divided by n_gpu_layers directly. Faults on x86-64, silent zeros on AArch64. - Measure both sides of the context interpolation against the same budget: the deficit now also reduces sum_used_target, the result is clamped to the training context, and the reduction log is computed in int64 (it could interpolate 32768 -> 47104 and report a wrapped "reduction"). - Charge host demand in the same currency as host availability: the probe loads with LLAMA_LOAD_MODE_NONE, but under the caller's real LLAMA_LOAD_MODE_MMAP the weight pages are file-backed and evictable - exactly the pages the availability metric leaves in the pool. The mmap'd model portion is now excluded from the host budget (nd == 0 and the fold). - Contain the blast radius: step-3 targets for shared-memory devices are capped by the pool (host free - host margin - resident host demand) instead of the device row alone; a host-only deficit with pinned n_gpu_layers now throws a message naming the host shortfall instead of blaming the pin; and common_fit_params snapshots and restores mparams/cparams on FAILURE/ERROR, since common_init_from_params ignores the status and would otherwise load with a context silently clamped mid-reduction. A catch-all maps unexpected exceptions to ERROR. Known residual limits, documented for follow-up: for nd >= 1 the step-2 reduction sums remain device-side (a host-only deficit with an auto context fails conservatively instead of reducing the host-side KV), and the step-3 pool cap is static across the descent rather than re-derived per assignment. Re-validated on a 24 GiB M4 Pro with the fold provably engaged: the Gemma 4 31B ngl=48 false positive (loads, cannot decode) now reports does-not-fit; Gemma ngl=99 stays does-not-fit; gpt-oss-20B @32k and Qwen3.5 9B @131k stay fits; CPU-only mmap fits match observed behavior with no silent context clamping; the previous SIGFPE path returns a structured FAILURE with the host-shortfall message. (cherry picked from commit 3a8d167)
The step-1 shared-pool fold and the step-2 context interpolation are pure arithmetic, and every defect found in review lived in them. Extract them as common_fit_shared_pool_deficit and common_fit_reduced_n_ctx (declared in fit.h for tests), hoist the backend-registry query to a single touchpoint that fills a shares_host vector, and add a table-driven ctest (tests/test-fit-params.cpp) covering: discrete-GPU non-folding, the measured device-surplus/combined-deficit shape, mixed shared/discrete devices, the traced context-inflation case (asserting the clamp to the training context), the zero-delta division case, and the interior interpolation. The fixtures run on any machine with no live devices, which is what makes the previous dead-fold class of bug directly assertable. (cherry picked from commit 5c629bf)
test-unified-memory-props asserted that Apple silicon must enumerate a Metal device, but ci/run.sh configures -DGGML_METAL=OFF unless GG_BUILD_METAL is set. The gpu-vulkan-apple and gpu-webgpu-apple jobs run on Apple silicon with Metal off, so both failed on "no Metal device enumerated" while the build never had the backend. Define LLAMA_TEST_EXPECT_METAL from tests/CMakeLists.txt only when GGML_METAL is ON and require it in the #if. The assertion still runs where it carries its regression value (gpu-metal, macos-latest-arm64), so a dead fold still cannot validate. Verified on M-series: GGML_METAL=OFF enumerates BLAS+CPU and passes; GGML_METAL=ON enumerates MTL0 with memory_unified=1 and passes. Assisted-by: Claude Opus 5 (cherry picked from commit c3ce47f)
The per-device target for a shares_host device capped against the full pool budget, computed identically for every such device. With one shared device (Metal on Apple silicon, the case this PR validates) that is correct, but two or more could each fill the whole budget and together overrun the pool. Extract common_fit_shared_pool_target and divide the budget by the number of shares_host devices. A negative budget is returned whole, since dividing it would understate the shortfall the descent has to reduce against. At one shared device the arithmetic is unchanged, so the validated Apple path keeps its measured behaviour. Cover it in test-fit-params, which had no multi-shared-device case. Assisted-by: Claude Opus 5 (cherry picked from commit 5854ddd)
GCC 11 (Ubuntu 22.04 gpu-cuda CI, Release -O3 with LLAMA_FATAL_WARNINGS=ON)
falsely flags the std::vector<ggml_op> initializer-list inserts of the
topk-moe path in ggml_cuda_try_fuse as overflowing the reallocated buffer
("writing 20 bytes into a region of size 4" in stl_algobase.h). The code is
verbatim upstream b10549 and correct; the diagnostic only surfaced after the
b10549 rebase because new upstream fusion code (rms_norm+mul+rope) in the
same TU shifted GCC's inlining decisions. Upstream master still carries the
identical pattern and newer GCC does not warn.
Disable the warning group for this TU only, and only for GCC: the warning is
attributed to the STL headers, so it must be disabled before they are
included, and clang does not know -Wstringop-overflow so an unguarded pragma
would trip -Wunknown-warning-option under -Werror.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014g78kYgJihu5v1h2fhLcJR
Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
(cherry picked from commit a3ce387)
(cherry picked from commit 1e9bcd3)
for_each_token_in tested all LLAMA_MAX_SEQ sequences for every used cell, while a cell almost always belongs to one. The scan now stops once the cell's own sequences have been seen. Same visit order, same callback arguments, so behaviour is unchanged. get_prev_tokens is the only caller, so this affects the n-gram path. RTX PRO 6000, Qwen3.8-Flash-Next UD-Q4_K_XL, fa on, warm runs: 55k context generation 56.3 -> 74.3 t/s 132k context generation 33.6 -> 50.9 t/s Prompt processing is unchanged, the scan is amortised over the ubatch there. The gain follows the number of used cells, so it grows with context and is invisible on short prompts. (cherry picked from commit 787a93e)
…l-org#28032) * vulkan: add top-k radix sort shader for k >= 1024 * add Qwen 3.8 Flash Next top-k tests * add top-k qsa fusion * clean up code (cherry picked from commit 65c7c96)
On some vendors, filling an VkDeviceFaultCountsEXT instance sets `VkDeviceFaultCountsEXT::vendorBinarySize` to a non-zero value, re-using this instance as is for the second call to `vkGetDeviceFaultInfoEXT`` will crash on a null pointer deref on VkDeviceFaultInfoEXT::pVendorBinaryData, which must not be null when `vendorBinarySize > 0` as per vulkan spec.
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
The ALIGNED fast paths in load_a_to_shmem/load_b_to_shmem read a whole BM x BN tile unchecked, while the scalar fallbacks below them check idx_m/idx_n. ALIGNED only asserts that K is a whole number of tiles, though — it says nothing about M or N, and nothing pads them. So when M or N is not a multiple of the tile size, the last tile reads past the end of A/B. Seen as ErrorDeviceLost on Mali-G715 (bitnet TQ2_0): a 33-token prompt against 64-wide tiles reads B rows 33..63. Not specific to TQ2_0. Bound all seven ALIGNED blocks, zero-filling out-of-range tiles to match the scalar paths. The MUL_MAT_ID guard also covers row_ids[col], which is unwritten past _ne1. Those rows were already discarded at store time, so results are unchanged: test-backend-ops MUL_MAT 1104/1104, MUL_MAT_ID 952/952.
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
…y_loss_masked_back Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
Signed-off-by: makaveli10 <vineet.suryan@collabora.com>
Check only ragged last-tile loads so interior tiles remain branch-free. Assisted-By: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Korcan Hussein <korcan.hussein@collabora.com>
Use BK64 when k <= 1536 or m <= 1536. This keeps the small-shape gains without regressing larger shapes.
* phase two * First PoC on MoE cache, with promising results on decode Assisted-by: GPT-5.6 Sol * CUDA: use the fast mm_ids_helper path for any n_expert_used (ggml-org#27978) The optimized path grouped warp lanes by token and required warp_size % n_expert_used == 0, with a single hardcoded exception padding 6 up to 8. Every other count fell back to the generic path, which walks the tokens one at a time with a warp reduction per token, for each of the n_expert blocks. The lane group only has to divide the warp, and the loop body already guards the padded lanes with iex < n_expert_used, so the padding generalizes to the next power of two. The 6 -> 8 case and every count already dispatched keep the exact same padding as before. n_expert_used = 10 now reaches the fast path. Measured on Qwen3.8-Flash-Next (512 experts, 10 used) at 55k context on an RTX PRO 6000, warm runs with the first one discarded: prompt processing 2334 -> 2600 t/s Token generation is unaffected, since a single token leaves nothing to walk. Other expert counts reach the fast path by adding their case to the dispatch. Assisted-by: GPT-5.6 Sol * ggml: keep MoE IDs alive during cache remap * llama: pipeline batched MoE cache transfers * examples: add persistent MoE corpus runner * docs: condense MoE cache experiment notes * ggml: remove unused inactive expert IDs * examples: keep MoE debug artifacts local * examples: remove MoE benchmark results * docs: remove local benchmark summary * examples: remove remaining MoE debug tooling * common: keep debug instrumentation local * ggml: route MoE cache ops without env override * llama: remove experimental MoE prefill overlap * llama: warn about mixed MoE cache layouts * ggml: simplify persistent MoE cache path * ggml: address MoE cache review feedback * guard test-moe-cache * llama: log MoE cache stats at trace level * llama: reject MoE cache on OpenCL * ggml: stabilize MoE cache graph topology --------- Co-authored-by: Pascal <admin@serveurperso.com>
…er backward Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
…nd Vulkan Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
…rs so nvcc's launch stubs compile with a clang host Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
Signed-off-by: Marcus Edel <marcus.edel@collabora.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebase tag: https://github.com/ggml-org/llama.cpp/releases/tag/b10549
Includes PRs #226, #214, #225#, #218, #216, #211, and #210 .