Skip to content

opencl: add int8 dp4 dense and MoE prefill optimization for Adreno GPUs - #25537

Merged
max-krasnyansky merged 2 commits into
ggml-org:masterfrom
qualcomm:hq/dp4a-prefill-moe
Jul 11, 2026
Merged

opencl: add int8 dp4 dense and MoE prefill optimization for Adreno GPUs#25537
max-krasnyansky merged 2 commits into
ggml-org:masterfrom
qualcomm:hq/dp4a-prefill-moe

Conversation

@wanghqc

@wanghqc wanghqc commented Jul 10, 2026 •
edited by lhez
Loading

Copy link
Copy Markdown
Contributor

Overview

  • This is to add int8 (dp4a) prefill MatMul path alongside the f16 GEMMs: activations quantized to q8_1 once and do int8 dot-product with weights, which may be re-quantized to int8 if needed.
  • This applies to many flavors of dense GEMMs, including q4_K/q6_K/q5_K/q8_0/q5_0/IQ4_NL (+ weight-as-texture).
  • MoE: add added generic per-quant MoE expert GEMM + dedicated q4_0/mxfp4 MoE kernels.
  • Ragged MoE with DP4A: added ragged tiling (skip fully padded per-expert token tiles), and MoE-combine fusion.

Additional information

Here is a quick perf on X2-90, pp512, ON vs OFF same binary, without QPM bin kernels:

  • Qwen3-30B Q4_K +66.7%
  • gpt-oss MXFP4 +54.5%
  • Qwen3-30B Q4_0 +70.2%
  • Token generation neutral (prefill only)

For X1-85: q4_K/q6_K MoE X1 gate wins +20–23%, dense tile +56%.

Right now, it is default-on for Adreno X2E, and there are some regresses on X1's narrower ALU / lower occupancy. X1 keeps the f16 path mostly.

Requirements

@wanghqc
wanghqc requested a review from a team as a code owner July 10, 2026 19:03
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning OpenCL Issues specific to the OpenCL backend labels Jul 10, 2026
@lhez
lhez force-pushed the hq/dp4a-prefill-moe branch 2 times, most recently from 1a5011f to 7fd1258 Compare July 10, 2026 21:07
@lhez lhez changed the title opencl: add DP4A (int8) and MoE ragged prefill optimization for Adreno GPUs opencl: add int8 dp4 dense and MoE prefill optimization for Adreno GPUs Jul 10, 2026
@lhez
lhez force-pushed the hq/dp4a-prefill-moe branch from 7fd1258 to c9d5ec1 Compare July 10, 2026 23:57
@lhez
lhez force-pushed the hq/dp4a-prefill-moe branch from c9d5ec1 to 5149b40 Compare July 11, 2026 00:19
@lhez
lhez requested a review from max-krasnyansky July 11, 2026 03:32
@max-krasnyansky
max-krasnyansky merged commit 1d1d9a9 into ggml-org:master Jul 11, 2026
26 checks passed
Geminihaha added a commit to Geminihaha/llama.cpp that referenced this pull request Jul 13, 2026
@Geminihaha
Merge ggml-org/master into fix_opencl_v2x_adreno
5671eaf 
Keep this branch's OpenCL backend; take upstream everywhere else.

The only upstream OpenCL change in this merge is
1d1d9a9 "opencl: add int8 dp4 dense and MoE prefill optimization for
Adreno GPUs (ggml-org#25537)". Its new dp4a kernels call dot_acc_sat_4x8packed_ss_int
(a cl_khr_integer_dot_product intrinsic) unconditionally. Adreno 6xx
(Adreno 660) does not expose cl_khr_integer_dot_product — only the older
cl_qcom_dot_product8 — so those kernels fail to compile (clBuildProgram
err=-11, "implicit declaration of dot_acc_sat_4x8packed_ss_int"). They are
loaded eagerly with a fatal build, so this aborts EVERY model load on 6xx.

The optimization is also X2E-gated at dispatch (q4k_dense_dp4a_on defaults to
adreno_gen == X2E), so it can never run on 6xx even if it compiled — it is
pure regression there. So the OpenCL backend files (ggml-opencl.cpp,
CMakeLists.txt, cvt.cl, gemv_moe_*_ns.cl) are kept at this branch's version
and the 16 new dp4a/quant_a/moe_* kernel files are dropped; all other
upstream changes are merged.

Verified on gemma-4-E2B-it-qat: loads and generates coherently (-ngl 99
-fa on, f16 KV, no err=-11/crash); test-backend-ops FLASH_ATTN_EXT
2551/2571 (0 crashes; gemma GQA+quant-KV still routed to CPU; 20 failures
are the pre-existing hsk=40 dim-3-broadcast cases). MUL_MAT_ID has 305
failures, but those are pre-existing on this branch (OpenCL backend is now
byte-identical to the pre-merge HEAD, and upstream did not touch the MoE
CPU reference or the MUL_MAT_ID tests) and do not affect dense gemma.
wanghqc added a commit to qualcomm/llama.cpp that referenced this pull request Jul 15, 2026
@wanghqc
Merge upstream master (e920c52) into x2-unified-everything
c139562 
Brings the branch to current upstream tip (24 commits). The only upstream
commit touching ggml-opencl is 1d1d9a9 -- our own dp4a PR (ggml-org#25537), which
came back squash-merged and review-modified.

Reconciled with the carve we sent upstream (hq/dp4a-prefill-moe) as the
three-way base, since both sides descend from it:
  - the 16 dp4a kernel files came back BYTE-IDENTICAL to what we sent, so this
    branch keeps its own supersets of them (ragged variants, wimg, quarter
    tile-skip granularity) unchanged;
  - ggml-opencl.cpp: kept this branch's versions -- they are strictly newer
    (per-gen fa_c8_cluster / fa_c8_default_on from the capability work, the
    program-cache build_program_from_source_ex signature, the K-image and
    g8_c32 decode dispatches). Upstream's side is the older carve.
  - CMakeLists.txt: dropped a duplicate gemm_moe_q8_0_f32_ns entry that this
    branch had listed twice; upstream's review had already removed it.
@wanghqc wanghqc mentioned this pull request Jul 17, 2026
Merged
CowboyTim pushed a commit to aardbeiplantje/llama.cpp that referenced this pull request Jul 21, 2026
@wanghqc @lhez ROCm FP4 Contributor
opencl: add int8 dp4 dense and MoE prefill optimization for Adreno GP…
4d99056 
…Us (ggml-org#25537)

* opencl: add int8 dp4 dense and moe GEMM

* opencl: refactor

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
satindergrewal pushed a commit to satindergrewal/llama.cpp that referenced this pull request Aug 12, 2026
@wanghqc @lhez
opencl: add int8 dp4 dense and MoE prefill optimization for Adreno GP…
10bd83c 
…Us (ggml-org#25537)

* opencl: add int8 dp4 dense and moe GEMM

* opencl: refactor

---------

Co-authored-by: Li He <lih@qti.qualcomm.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@max-krasnyansky max-krasnyansky max-krasnyansky approved these changes
@lhez lhez lhez approved these changes

Assignees

No one assigned

Labels

ggml changes relating to the ggml tensor library for machine learning OpenCL Issues specific to the OpenCL backend

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wanghqc @max-krasnyansky @lhez