Sync temp-10297 - #228
Sync temp-10297 #228
Conversation
…pCount Review fix (PR #174): the fused-norm change dispatched GGML_OP_NORM as a direct {ne01, ne02, ne03} grid; on large row counts ne01 can exceed maxComputeWorkGroupCount[0] (spec minimum 65535) and trip the GGML_ASSERT in ggml_vk_dispatch_pipeline, where the previous flattened/tiled dispatch handled arbitrary ggml_nrows. Restore the flattened {512, 512, N} row tiling on the host (same group as SOFT_MAX/SUM_ROWS) and reconstruct {row, channel, sample} in norm.comp from the flat workgroup id (formula shared with soft_max.comp), with a workgroup-uniform bounds return for the tiling round-up. dst offset is unchanged: flat_row == (samp*nchannels + channel)*nrows + row by construction. No behavioural change for in-range shapes; the fusion's dispatch-count reduction is untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit 68e6c52)
…udget-aware) The warmup-time hard-disable for GPU projectors without efficient (coopmat) flash attention replaced AUTO/ENABLED with DISABLED, which short-circuited the budget-aware AUTO heuristic in clip_resolve_flash_attn_type(). At high n_pos (image_tile_mode=disabled with image_max_tokens=4096 -> 16384 ViT patches) the forced explicit attention path materializes an O(n^2 * n_head) score matrix, growing RSS to ~12 GB and getting the process lmkd-killed on Pixel 9 Pro (runQwen35ImageTileModeTokensTest). Downgrade to AUTO instead and record the inefficiency in clip_ctx::fa_backend_inefficient, which now also enables the AUTO cutoff default (previously Mali-detection only) so any non-coopmat backend gets the per-image budget decision: explicit attention below the cutoff (fast on scalar-FA GPUs), memory-frugal scalar FA at/above it or when the explicit scratch would not fit device memory. Explicit user DISABLED is still honored, and MTMD_CLIP_AUTO_FA_MIN_KV still overrides the cutoff. (cherry picked from commit cceee22)
…q-chunking) On Galaxy S25 Ultra (Adreno 830, OpenCL) the monolithic 16384-patch ViT encode (image_tile_mode=disabled, image_max_tokens=4096) faults the GPU near the end of the encode (Adreno-GSL log_gpu_snapshot fires before any decode work reaches the device), after which the driver aborts the process from cl_a8x_cmdbuf_mgr_submit_ibs (os_exit) on the next submission. Two unbounded behaviours plausibly drive the fault and both are bounded here: - ggml_backend_opencl_graph_compute enqueued entire graphs (thousands of nodes, ~48 s of GPU work for the failing encode) with no intra-graph flush. Now clFlush every GGML_OPENCL_FLUSH_INTERVAL nodes (default 64, 0 disables) so the GSL command-buffer manager receives bounded batches. clFlush submits without stalling the host. - ggml_cl_flash_attn issued one dispatch covering all q rows; at n_q = n_kv = 16384 every workgroup loops the full KV, making a single very long kernel. Now chunked along q rows at GGML_OPENCL_FA_MAX_NQ rows per dispatch (default 4096, 0 disables) with a clFlush between chunks. The split is exact: the kernel resolves its q row relative to the Q/O/mask base offsets, is_causal is always 0 (masking is explicit) and alibi/sinks depend only on the head index, so shifting the row base via byte offsets while shrinking n_q is mathematically identical. No .cl kernel changes. The 512-token image-chunk decode is not implicated: the S25 VLM benchmark ran 304 full-512-row ubatch decodes cleanly. Only the giant monolithic encode (5-8x beyond anything previously run on this backend) triggers the fault. (cherry picked from commit 82a26df)
…, tests Addresses the pre-merge review findings on the two QVAC-21914 crash-fix commits (P1/P2 performance, C1/C2 correctness, S1/S2 robustness, K nits): - ggml-opencl: gate the periodic graph flush on accumulated estimated WORK (GGML_OPENCL_FLUSH_WORK_MB, default 512 MB) instead of a bare node counter. Per-token LLM decode graphs never reach the budget by construction, so the decode hot path stays submission-free; the 16k-patch encode still flushes dozens of times. Single touch point in graph_compute (no more per-fusion-branch duplication). - ggml-opencl: both tunables move onto ggml_backend_opencl_context, resolved once at init with strtol-based parsing (clamp, warn on garbage instead of silently disabling the mitigation) and GGML_LOG_INFO'd like the file's other env knobs. FA chunking reads the context field. - ggml-opencl: GGML_ASSERT(is_causal == 0) before the FA chunk loop — the kernel's causal-boundary formula needs the TOTAL n_q, so chunks after the first would silently corrupt output if causal FA were ever enabled here; keep the invariant loud. Explicit n_q == 0 guard. - clip: rework the AUTO cutoff memory clamp. Total memory now provides the STABLE fast-path clamp (explicit scratch <= total/4); free memory (a volatile, load-dependent number) may only lower the cutoff further via the hard-fit requirement (scratch <= free), never the old free/2 heuristic that silently pushed normal-size Mali images onto the ~2.6x slower scalar-FA path under momentary memory pressure. No memory info at all now fails SAFE at a conservative 2048-patch cap instead of trusting the raw 4096 default (~3.2 GB scratch at n_head=16). The arithmetic is extracted into clip_fa_effective_min_kv() (pure, exposed via clip.h for tests). - tests: test-clip-fa-cutoff (pure CPU, locks in the fast path, the P2 regression guard, the fail-safe cap and edge cases; passing) and test-opencl-fa-chunking (chunked-vs-CPU numerical parity over unchunked / exact-chunk / partial-last-chunk / n_q==1, masked and unmasked, with GGML_OPENCL_FA_MAX_NQ=64 and a 1 MB flush budget; self-skips without a capable OpenCL device — PoCL lacks FP16, so it executes on Adreno-class hardware). - clip warmup comment: note ggml-opencl also lands in the "no efficient-FA query" bucket and its giant-encode fault is handled by the submission bounding inside that backend. GGML_OPENCL_FLUSH_INTERVAL (node-count knob) is replaced by GGML_OPENCL_FLUSH_WORK_MB; GGML_OPENCL_FA_MAX_NQ semantics unchanged. (cherry picked from commit fc09f36) --- b10297 rebase: - Squash a68b359: test-opencl-fa-chunking: call ggml_backend_load_all() and select the device by backend registry name ("OpenCL") instead of substring-matching the device name. Squashed-with: 649de77, a68b359
The pure AUTO-budget helper was defined out-of-line in clip.cpp and declared in the internal clip.h. On Windows mtmd builds as a shared library exporting only the MTMD_API-decorated public API; the internal clip_* symbols are absent from mtmd.lib, so test-clip-fa-cutoff (the first cross-DLL-boundary consumer of a clip_* symbol) failed to link (LNK2019). Linux/macOS export all default-visibility symbols, so it linked there. Move the function inline into clip.h (with its NO_MEMINFO_CAP constant); the test and clip.cpp both compile their own copy — no DLL export of an internal helper. CLIP_AUTO_FA_MIN_KV_MALI_DEFAULT stays in clip.cpp (its only user). Verified: mtmd + test-clip-fa-cutoff build and the test passes. (cherry picked from commit 99d6042)
…uard - parse_env_i64: GGML_LOG_WARN when an in-range-but-too-large GGML_OPENCL_FLUSH_WORK_MB / GGML_OPENCL_FA_MAX_NQ is clamped to max, matching the file's convention of logging every overridden value (previously the clamp was silent). - test-clip-fa-cutoff: the n_head=0 case passed total_mem==free_mem==0, which short-circuits to the NO_MEMINFO cap before any sqrt(.../n_head) branch runs — the div-by-zero guard was never exercised. Pass 16 GB total so the total-memory clamp runs with n_head=0; without the guard the (int)sqrt(x/0) path would now fail the assertion. Both from yingying0906's review; Windows/CPU-only surface, no Android behavior change. (cherry picked from commit 3da3e05)
The graph_compute work-budget flush ran BEFORE the current node was dispatched: it accounted the node's work, and on crossing the budget flushed (submitting only the prior batch) then reset the counter to 0 — so the crossing node started a fresh batch. A large op, or the last large segment of the graph, could therefore begin an unflushed batch and be submitted unbounded at the implicit end-of-graph finish, defeating the bound. Move the budget check below the dispatch (convert the fused-op continue chain to if/else so every path reaches one touch point), so the node that crosses the budget is part of the flushed batch. Reported by @gianni-cor. (cherry picked from commit 7056f4c)
Non-behavioral cleanups from the PR #181 review pass: - ggml-opencl graph_compute: correct the flush-cadence comment. The old "per-token decode hot path submission-free by construction" claim was false for multi-GB models — a decode step streams the whole model, so its estimated work crosses the default 512 MB budget a few times per token. Reworded to state that accurately (cost is negligible in practice since clFlush is non-blocking, and it is tunable/zeroable to make decode fully submission-free). Mechanism unchanged. - ggml_cl_flash_attn: GGML_ASSERT(q->ne[1] <= INT32_MAX) before the int n_q truncation, since the q-chunk loop accumulates into an int and derives cl_ulong offsets from it (defensive; not reachable with real shapes). - Consistency: normalize the ticket tag to bare `QVAC-21914` (drop the `qvac ` prefix) in clip.h and tests/CMakeLists.txt, matching the .cpp files and the fork's QVAC-21257 precedent. - tests/CMakeLists.txt: move the unconditional test-opencl-fa-chunking registration up beside test-copy-tbq-subgroups (its self-skipping sibling) instead of sitting right after the LLAMA_MTMD endif() where it read as MTMD-gated; add a comment noting it deliberately does not link mtmd. No functional change to the fix; local mtmd + both tests build, test-clip-fa-cutoff passes. (cherry picked from commit 2b927cf)
…on-contiguous (permuted) inputs read the right columns Signed-off-by: Marcus Edel <marcus.edel@collabora.com> (cherry picked from commit 6053e42)
Only claim OUT_PROD support for src types ggml_get_to_fp32_cuda can requantize. ggml_cuda_out_prod converts non-F32 srcs to F32 before the f32-only cuBLAS GEMM and aborts on e.g. TQ2_0 which has no CUDA dequantizer. (cherry picked from commit d529485)
The OpenCL kernels are compiled with -cl-finite-math-only and -cl-fast-relaxed-math, which let the compiler assume no Inf/NaN. The flash-attention online softmax initialises its running max to -INFINITY and masks padded scores with -INFINITY, so finite-math miscompiles the init/masking path. Compile the flash-attention programs with a relaxed option set that drops -cl-fast-relaxed-math, -cl-finite-math-only and -cl-unsafe-math-optimizations (keeping -cl-mad-enable for speed) so the -inf sentinels behave correctly. Also harden the strip: erase every occurrence of each flag (not just the first) and GGML_ASSERT that no finite-math/fast-math/unsafe-math flag survived, so a future compile_opts spelling/spacing change fails loudly at load time instead of silently reintroducing the -INFINITY miscompile. Re-ported onto b9840's rewritten OpenCL flash-attn (upstream PR ggml-org#14987 + follow-ups): the original per-dim kernel-compile loop is gone, so the strip is applied once in ggml_opencl_fa_compile_opts(), the single site every FA variant (F16/F32/F32_F16/Q8_0/Q4_0/PRE and _SPLIT) is compiled through. Squashed re-port of 0cbe362 + c1dace7 (finite-math part). (cherry picked from commit 348a910)
The flash-attention dispatch inferred causal masking from shape with `is_causal = (mask == NULL && n_q > 1 && n_q == n_kv)`. A null mask means no masking, i.e. bidirectional attention (the SigLIP vision and embedding encoders), while causal attention always supplies an explicit causal mask in this codebase (llama-graph.cpp build_attn passes a kq_mask filled with -INFINITY). The heuristic therefore wrongly made the bidirectional Qwen3-VL vision tower attend causally, so each patch only saw earlier patches and the image embedding was corrupted. Set is_causal = 0 unconditionally; causality is always expressed via the explicit mask. This cannot regress the LLM, which already passes a real causal mask (is_causal was already 0 for it) and relies on that mask. Document the invariant in ggml_cl_flash_attn: a null mask is treated as bidirectional, so any caller needing causal masking must supply an explicit causal mask rather than relying on shape inference. Re-ported onto b9840's rewritten OpenCL flash-attn; b9840's own q-chunking path already GGML_ASSERTs is_causal == 0, so this is consistent with the existing code. Squashed re-port of 51dbb17 + c1dace7 (is_causal part). (cherry picked from commit 7ae4bc9)
…uard upscale zero dims The f32/f16 flash-attention kernels load K/V tiles into local memory, barrier, read them, then loop to overwrite the tiles for the next K/V block without a trailing barrier. Out-of-range lanes (the last partial BLOCK_M block) `continue` past the read and race ahead into the next tile load while active lanes are still reading l_k/l_v. With n_kv > BLOCK_N (e.g. the bidirectional vision tower, n_kv=247) this corrupts the shared tiles. Add a trailing barrier(CLK_LOCAL_MEM_FENCE) at the end of the K/V block loop and guard the score computation with `if (my_query_row < n_q)` instead of an early continue. flash_attn_f32_f16.cl already uses that guard + trailing barrier after b9840's redesign, so it is left unchanged. Also guard zero source dimensions in ggml_cl_upscale: the sf* scale factors divide by the source dims, so a zero source dim yields +inf; the existing early-exit only covered zero destination dims. Re-ported onto b9840's rewritten OpenCL flash-attn (b9840 widened the score unroll to j += 4; only the divergence guard + trailing barrier are re-applied, the body is unchanged). Squashed re-port of dc64397 + b7ad6d4. The barrier fix is a GPU-scheduling race whose only proof is on-device; the original b7ad6d4 validated on S25 Ultra / Adreno 830 (Qwen3-VL GPU vision projector matches CPU exactly, Delta 0.0 pp, ~26% faster on encode). Re-verify on-device before merge. (cherry picked from commit ce54dd5) --- b10297 rebase: - Squash 97a1ecd: flash_attn_f32.cl: apply the divergence guard unconditionally and drop the FA_SG<64-only trailing barrier; the tile race reproduces even on a single 64-wide Adreno subgroup (Adreno 830). Squashed-with: 278014e, 97a1ecd
The upstream target of this former fixup! (50e0ad0, --clear-idle ggml-org#20993) already landed upstream, so this stays a standalone commit. Relying on exact log text is brittle, especially across rebases with upstream changes; use the timings fields instead and drain remaining logs for test cleanliness.
…pend on the torch pin surviving pip install Signed-off-by: Marcus Edel <marcus.edel@collabora.com> (cherry picked from commit 2d3f403)
A supports_op() regression never fails test-backend-ops: the case falls back to CPU and is reported 'not supported [backend]', i.e. skipped. e09ae0b removed Q4_1/Q4_K from OpenCL MUL_MAT supports_op with zero test failures --- b10297 rebase: - Squash e8fb282: add the UPSCALE f32 bilinear|antialias row to the opencl-pocl op-coverage manifest. Squashed-with: 5587e68, e8fb282
The FA sweep uses kv in {113, 512, 1024} x nb in {1, 3, 32, 75}, so nb
never equals kv and the mask==NULL && n_q==n_kv shape -- exactly what a
ViT self-attention layer produces -- is never exercised. A backend that
infers causality from that shape (OpenCL's is_causal heuristic,
ggml-opencl.cpp:15131) silently computes causal attention for the whole
vision tower and no CI test goes red.
Add explicit bidirectional cases at n_q == n_kv == 247 and 256 for head
sizes 64 and 80 (both in the OpenCL FA supported-dims table). 247 (odd)
additionally leaves partial tiles for any power-of-two tile size; 256 is
the aligned control separating causality bugs from tiling bugs.
Expected red on OpenCL until the is_causal heuristic is removed
(re-port of b9840 7ae4bc9); green on CPU/Vulkan/CUDA/Metal.
Note for landing: order this commit after the is_causal fix so the
series stays bisectable-green on OpenCL hardware.
Assisted-by: Claude (Anthropic AI)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DbaV79zWiPZJq1LrdWGDTj
(cherry picked from commit a0fe46a41d5c8e02f9923049c0a16cbaef8d7439)
…riants n_q=33 with n_kv=513 leaves out-of-range query lanes for any pow2 query tile (Adreno OpenCL uses BLOCK_M=64 for dk 64/128) and a 1-valid-row final KV tile (513 = 16*32 + 1), with 17 tile-loop iterations worth of barrier crossings. Tiled kernels must keep out-of-range lanes inside the tile loop for the barriers while excluding them from the score loop; an early continue past the tile barrier (the b9840 ce54dd5 race class) or a missing trailing barrier corrupts the shared K/V tiles for in-range lanes. Cover each KV type separately: backends that specialize kernels per KV type (OpenCL picks flash_attn_f32.cl for f32 KV, f32_f16(+split) for f16 KV since n_kv=513 crosses the split threshold, and the q8_0/q4_0 tiled kernels for quant KV) would otherwise leave those variants untested at this shape. Note flash_attn_f16.cl itself is only reachable with an f16 Q tensor, which test-backend-ops never generates (Q is always f32) -- it stays covered only by code review. The race is scheduling-dependent: in-order devices (pocl) and drivers with cooperative-matrix FA paths will likely pass even with the bug; these shapes make the sweep able to catch it on the affected hardware class (original repro: Adreno 830). Validated: CPU supports all 5 cases; Vulkan RADV 5/5 PASS on 7900 XTX and Raphael iGPU. Assisted-by: Claude (Anthropic AI) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DbaV79zWiPZJq1LrdWGDTj (cherry picked from commit ab6ecf6b4ea23533640988f0557b11c48106bf01)
Add the tuned cooperative-matrix path without experimental runtime controls. (cherry picked from commit 9675568) Co-authored-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Vectorize the fused HC post path for ARM and x86 while preserving scalar fallback behavior. (cherry picked from commit 7c21b4a)
Assisted-by: GPT-5.6 Sol (cherry picked from commit 5f41c2e) Build-fix-hoisted-from: d19b9ddbadc9 (drop stray closing brace after ggml_vk_lightning_indexer)
Assisted-by: GPT-5.6 Sol (cherry picked from commit 9afa822)
Assisted-by: GPT-5.6 Sol (cherry picked from commit e5d7104)
Assisted-by: GPT-5.6 Sol (cherry picked from commit 16d647b)
Add fused-versus-fallback correctness and benchmark cases for HC post and the Lightning Indexer. Assisted-by: GPT-5.6 Sol (cherry picked from commit b5dcb61) --- b10297 rebase: - skip_backend for DSV4_HC_POST_BIT_EXACT matched only the "CUDA" reg name, but HIP builds register the same backend as "ROCm" (GGML_CUDA_NAME), so the FMA-contraction skip never fired there and the bit-exact gate failed deterministically on ROCm (6 cases, ERR=1.0).
Generate per-K-type generic pipelines and 32/64-head CM1 and CM2 variants, with quant-aware stride handling and guarded pipeline selection. CM1 stages decoded FP16 tiles under shared-memory limits; CM2 uses FP16 decode callbacks for quantized K tiles. Assisted-by: GPT-5.6 Sol (cherry picked from commit d2414ec)
Exercise matrix paths for 32- and 64-head layouts across supported K-cache types, including dispatch-tail boundaries and a strided-Q scalar fallback. (cherry picked from commit 1a8d1a6)
Keep quantized cache concatenation on the GPU to avoid per-layer CPU fallbacks and excessive graph splits. (cherry picked from commit 7e53028)
(cherry picked from commit 581bbda)
Keep DeepSeek V4 mask construction on the GPU with a vectorized path for strided inputs. (cherry picked from commit 807cc6a)
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
Assisted-by: GPT-5.5
…ntain permissions' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Keep the Socket-patched dependency on a converter-compatible version. Assisted-by: GPT-5.5
Drop decorators for a ModelBase API that is not present on this branch. Assisted-by: GPT-5.5
Allow the observed OpenVINO precision variance for the new skinny F32 matmul case. Assisted-by: GPT-5.5
Clean up trailing whitespace and missing final newlines reported by CI. Assisted-by: GPT-5.5
Escape backslashes before rendering function names in LaTeX output. Assisted-by: GPT-5.5
Set read-only contents permission for the self-hosted workflow token. Assisted-by: GPT-5.5
QVAC-24112 fit: budget against real memory availability
Use one replacement path for both backslashes and underscores so CodeQL recognizes the escaping. Assisted-by: GPT-5.5
Avoid compiling the helper when CUDA FP4 is available but Blackwell code is disabled. Assisted-by: GPT-5.5
Copy the test binaries into the model work directory before the long gguf-split script runs so later invocations do not depend on the build tree still being present. Assisted-by: GPT-5.5
CUDA: fix build warning for non blackwell builds
for_each_token_in tested all LLAMA_MAX_SEQ sequences for every used cell, while a cell almost always belongs to one. The scan now stops once the cell's own sequences have been seen. Same visit order, same callback arguments, so behaviour is unchanged. get_prev_tokens is the only caller, so this affects the n-gram path. RTX PRO 6000, Qwen3.8-Flash-Next UD-Q4_K_XL, fa on, warm runs: 55k context generation 56.3 -> 74.3 t/s 132k context generation 33.6 -> 50.9 t/s Prompt processing is unchanged, the scan is amortised over the ubatch there. The gain follows the number of used cells, so it grows with context and is invisible on short prompts.
…l-org#28032) * vulkan: add top-k radix sort shader for k >= 1024 * add Qwen 3.8 Flash Next top-k tests * add top-k qsa fusion * clean up code
qwen4-next-fixes: upstream perf fixes for qwen4exp (Vulkan TOP_K, kv-cells, GDN/LID, graph splits)
Import the small constants directly so the MCP default-value unit test does not load the full constants barrel while checking settings defaults. Assisted-by: GPT-5.5
20 participants
Sync temp-10297