ggml-zendnn : group matmul direct API for mul_mat_id - #25918
Merged
ggerganov merged 2 commits intoJul 31, 2026
Conversation
github-actions
Bot
added
ggml
Member
|
cc: @z-vishal |
z-sachin
added a commit
to z-sachin/llama.cpp
that referenced
this pull request
Jul 24, 2026
…nly) Rebased on top of ggml-org#25918's grouped-GEMM mul_mat_id rework. Q4_0 support is restricted to plain MUL_MAT: ggml-cpu's CPU_REPACK mechanism claims Q4_0 weights into the 8-row-interleaved block_q4_0x8 layout, and ZenDNN detects that layout (is_superblock) to compute over it directly instead of forcing plain blocks. mul_mat_id (MoE) has no Q4_0 path yet, so Q4_0 experts continue to fall back to CPU there. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
z-sachin
added a commit
to z-sachin/llama.cpp
that referenced
this pull request
Jul 24, 2026
With grouped-GEMM (ggml-org#25918) the per-call overhead dropped, but ZenDNN's mul_mat_id still loses to CPU below a real per-batch token count of ~384 on gpt-oss-20B (32 experts, top-4) - measured empirically. Fall back to CPU under that threshold instead of always taking the ZenDNN path once the expert-count gate passes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
avinashcpandey
approved these changes
Jul 30, 2026
taronaeo
approved these changes
Jul 30, 2026
taronaeo
added
the
merge ready
kashif
pushed a commit
to kashif/llama.cpp
that referenced
this pull request
Aug 2, 2026
* ggml-zendnn : group matmul API for mul_mat_id * ggml-zendnn : scale MUL_MAT_ID fallback threshold by expert count
huaxel
pushed a commit
to huaxel/CachyLLama
that referenced
this pull request
Aug 2, 2026
* ggml-zendnn : group matmul API for mul_mat_id * ggml-zendnn : scale MUL_MAT_ID fallback threshold by expert count
smalinin
pushed a commit
to smalinin/llama.cpp
that referenced
this pull request
Aug 4, 2026
* ggml-zendnn : group matmul API for mul_mat_id * ggml-zendnn : scale MUL_MAT_ID fallback threshold by expert count
satindergrewal
pushed a commit
to satindergrewal/llama.cpp
that referenced
this pull request
Aug 11, 2026
* ggml-zendnn : group matmul API for mul_mat_id * ggml-zendnn : scale MUL_MAT_ID fallback threshold by expert count
satindergrewal
pushed a commit
to satindergrewal/llama.cpp
that referenced
this pull request
Aug 12, 2026
* ggml-zendnn : group matmul API for mul_mat_id * ggml-zendnn : scale MUL_MAT_ID fallback threshold by expert count
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
This PR updates the vendored ZenDNN dependency to the latest WW28 release and switches the ZenDNN mul_mat_id implementation to use grouped GEMM, allowing all active experts to be processed in a single backend call.
Changes
Add
ggml_zendnn_group_matmul()andggml_zendnn_group_gemm()wrappers that batch all active experts' GEMMs into a single ZenDNN grouped GEMM call instead of issuing one call per expert.Rework
ggml_zendnn_compute_forward_mul_mat_id()to support grouped execution:total_rowsinstead ofmax_rows, since data for all active experts must coexist for a single grouped GEMM.src1→vec_dot_typeconversion into the gather step, eliminating the separate conversion pass.Restructure the execution flow from
for each expert: gather → GEMM → scattertogather all experts → one grouped GEMM → scatter all experts.Remove the outdated comment in
device_supports_op()that described grouped GEMM as future work. The existing 32-expert limit remains unchanged.Requirements