Skip to content

opencl: transpose the FA prefill K tile in local memory for perf optimization - #26428

Merged
lhez merged 1 commit into
ggml-org:masterfrom
qualcomm:hq/fa-k-lds-transpose-pr-r0730
Aug 10, 2026
Merged

opencl: transpose the FA prefill K tile in local memory for perf optimization#26428
lhez merged 1 commit into
ggml-org:masterfrom
qualcomm:hq/fa-k-lds-transpose-pr-r0730

Conversation

@wanghqc

@wanghqc wanghqc commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Overview

This PR is to optimize the current FA OpenCL C kernels:

  • The flash-attention prefill kernels (flash_attn_f32_f16 / _q8_0 / _q4_0) stage the K tile in local memory and the QK loop reads.
  • For each step, several KV rows that are DK_VEC apart — i.e. one narrow 64-bit local read per row.
  • Transpose that tile in local memory so the KV rows the loop walks together become adjacent: a pair (f16) is then one 128-bit read, halving the LDS issues for the exact same bytes.
  • Arithmetic and FMA order untouched → bit-identical output.

Default-on at DK ≤ 128 (the geometry where the paired read is valid);

  • kill switch GGML_OPENCL_FA_K_LDS_T=0.
  • DK=256 stays on the original path (measured negative there).
  • Layout-only. PPL/greedy bit-identical everywhere measured.
  • TBO FLASH_ATTN_EXT counts are identical to base on every device tested.

Additional information

Earlier fleet data (pre-rebase content, unchanged kernels):

  • Win on X1/X2/840, X1-85 pp4096 +8.6% / pp8192 +11.5%, 740 neutral.
  • Intel NEO neutral +0.1% (2565/2565).
  • No Adreno gate needed.
  • Only cfg->dk <= 128.
device correctness (TBO FLASH_ATTN_EXT) perf vs base
Adreno X2-90 (asus-x2, native ARM64) identical counts, 0 FAIL pp4096 485.4 → 528.3 = +8.8%; tg32 neutral (−0.5%)
Adreno 840 (NX809J, NDK arm64) 2653/2653, identical to base pp4096 +5.4..+8.6% (128–132 → 139.2); tg32 neutral

Requirements

@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning OpenCL Issues specific to the OpenCL backend labels Aug 2, 2026
@wanghqc
wanghqc marked this pull request as ready for review August 3, 2026 17:59
@wanghqc
wanghqc requested a review from a team as a code owner August 3, 2026 17:59
@lhez
lhez force-pushed the hq/fa-k-lds-transpose-pr-r0730 branch 2 times, most recently from 22f61fe to de775bc Compare August 6, 2026 01:11
@lhez
lhez requested a review from max-krasnyansky August 6, 2026 23:56
@lhez lhez added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Aug 7, 2026
@wanghqc
wanghqc force-pushed the hq/fa-k-lds-transpose-pr-r0730 branch from de775bc to 59b58ee Compare August 8, 2026 19:04
@lhez
lhez merged commit 689e227 into ggml-org:master Aug 10, 2026
23 of 28 checks passed
wanghqc added a commit to qualcomm/llama.cpp that referenced this pull request Aug 10, 2026
28 upstream commits since 687e778. One touches ggml-opencl: 689e227
(ggml-org#26428, FA prefill K-tile transpose) -- our own PR, merged upstream this
window, so its four files conflicted against the x2ue originals they were
carved from.

Resolved to ours in all four: x2ue is a strict superset (FA_Q_HALF /
FA_O_HALF, the FA*_PROBE_NO_LDS diagnostics and FA_V_LDS_T all landed on
top). Verified line-by-line that no ggml-org#26428 content was lost -- the seven
upstream lines absent from the merged tree are the same code reshaped by
our later commits (#elif rather than #if because our probe branch precedes,
mad() rather than *, Q_PRIV_TO_ACC4() around q_priv), all still reading
through FA_LK/FA_LK_PAIR, plus one reworded comment.

Also verified the automerge kept upstream intact: of the 659 files upstream
touched, the merged tree is byte-identical to upstream/master on all but
the eight we also modified, and all 247 upstream-added lines in the four
auto-merged files (clip.cpp, clip-model.h, llama-context.cpp,
test-backend-ops.cpp -- Muse Glimmer ggml-org#26841, multi-output sampling ggml-org#25532,
Granite-Switch ggml-org#25107) are present.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. OpenCL Issues specific to the OpenCL backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants