vulkan: Support Q2_0 - #25430
Merged
Merged
Conversation
github-actions
Bot
added
testing
Contributor
Author
|
Metal crash would presumably be fixed by #25419, if that gets merged soon I'm fine to wait for it. |
jeffbolznv
force-pushed
the
q2_0
branch
2 times, most recently
from
July 13, 2026 14:54
255f067 to
a16d750
Compare
Member
It's merged now. |
ggerganov
approved these changes
Jul 14, 2026
0cc4m
approved these changes
Jul 14, 2026
The backend perf tests for mat-vec-mul weren't very good at first (worse than q2_k), doubling the rows per workgroup made a big difference.
Contributor
Author
|
Had to resolve a nontrivial conflict, will need another +1 to merge. |
ggerganov
approved these changes
Jul 14, 2026
MarkShark2
added a commit
to MarkShark2/llama.cpp
that referenced
this pull request
Jul 15, 2026
Q2_0 (ternary) GPU kernels for CUDA and Vulkan, cherry-picked from the upstream-targeted PrismML PR branches (ggml-org#25603 cuda, ggml-org#25430 vulkan). Upstream already carries the Q2_0 type + CPU backend (ggml-org#24448) and the qwen35 arch, so these kernels are the only gap for running Ternary-Bonsai-27B (Q2_g64).
MarkShark2
added a commit
to MarkShark2/llama.cpp
that referenced
this pull request
Jul 17, 2026
Upstream landed native Q2_0 (type 42) with full Vulkan support (ggml-org#25430) and refactored the CUDA MMQ kernel configuration (ggml-org#24127). Reconciled feat/q2_0-gpu against both: - Vulkan Q2_0: took upstream's implementation (identical QK2_0=64 block format; our branch had cherry-picked the same PR). Dropped our redundant shader/pipeline copies. - CUDA Q2_0: dropped entirely (MMQ port collided with the ggml-org#24127 refactor and the half-kept dequant path would have routed small-batch matmuls to a mmvq kernel that no longer exists). Q2_0 now runs on Vulkan (BC-250) only, matching upstream; the 3090 has no Q2_0 path, same as mainline. Re-porting the CUDA kernel is a deliberate follow-up if a House config ever needs Q2_0 on CUDA. Preserved fork features through the merge: mtp_dev pin, hy_v3 (.bias suffix + per-layer n_ff_exp_impl), nemotron-h MTP filter, ssm d_state 96 (CUDA+Vulkan), vk-uma-mem clamp, disk-cache, rpc-cache(-preflight), decode-timing instrumentation. Assisted-by: Claude Fable 5 <noreply@anthropic.com>
3 tasks
ggerganov
pushed a commit
to am17an/llama.cpp
that referenced
this pull request
Jul 28, 2026
* vulkan: Support Q2_0 The backend perf tests for mat-vec-mul weren't very good at first (worse than q2_k), doubling the rows per workgroup made a big difference. * reorder * resolve merge conflict, adjust err threshold for f16->q2_0 set_rows
smalinin
pushed a commit
to smalinin/llama.cpp
that referenced
this pull request
Aug 4, 2026
* vulkan: Support Q2_0 The backend perf tests for mat-vec-mul weren't very good at first (worse than q2_k), doubling the rows per workgroup made a big difference. * reorder * resolve merge conflict, adjust err threshold for f16->q2_0 set_rows
satindergrewal
pushed a commit
to satindergrewal/llama.cpp
that referenced
this pull request
Aug 12, 2026
* vulkan: Support Q2_0 The backend perf tests for mat-vec-mul weren't very good at first (worse than q2_k), doubling the rows per workgroup made a big difference. * reorder * resolve merge conflict, adjust err threshold for f16->q2_0 set_rows
mndodd
added a commit
to mndodd/llama.cpp
that referenced
this pull request
Aug 12, 2026
…g every type and aborting
ggml_backend_sycl_device_supports_op()'s MUL_MAT case ended in an unconditional
return true, so the backend claimed every src0 type including ones with no kernel
and no dequantizer. The scheduler took it at its word and the dispatch then hit
GGML_ABORT("unsupport data type"). A backend that aborts rather than declining is
not "unsupported", it is a crash: when upstream added Q2_0 (ggml-org#25430, a Vulkan PR)
it killed test-backend-ops -o MUL_MAT at case 72 of 1769.
That mattered because two thirds of our own added MUL_MAT coverage sits past case
72, so it had silently stopped running (findings/206).
The set is derived, not invented. MMVQ/MMQ/DMMV are optimisations that each claim
only some shapes; when none does, mul_mat falls back to dequantize -> oneDNN GEMM.
So the real predicate is "does convert.cpp have a dequantizer", and the list
mirrors ggml_get_to_fp16_sycl/ggml_get_to_fp32_sycl plus the native float types.
ggml_get_to_fp16_sycl cannot be used as the predicate directly because it ABORTs
on an unknown type rather than returning nullptr -- probing it would trigger the
crash being prevented.
Measured, -o MUL_MAT default env: 72 cases -> 658. q2_0 declined cleanly in 9
cases instead of aborting. Previously-unreachable coverage now runs: MMVQ
ncols_dst 13/14/15 0 -> 18, non-reorder multi-column 0 -> 219, MMQ n=9 probe
9 -> 62. No deployed type is declined -- q4_0/q8_0/q4_K/q5_K/q6_K all still
supported and passing; the f32/f16 "not supported" entries are the pre-existing
a->ne[3] != b->ne[3] gate, which sits after this check.
Same defect shape as the CONCAT supports_op fall-through. The remaining -o MUL_MAT
abort is upstream's CPU reference (vec.cpp:369 !isnan(sumf)) on degenerate f16
data, not ours.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Support Q2_0.
The backend perf tests for mat-vec-mul weren't very good at first (worse than q2_k), doubling the rows per workgroup (using rm_kq_int) made a big difference.
Requirements