HIP : route Q6_K through MMQ up to ne11 = 1024 on RDNA 3.5 - #38
Open
Nyovelt wants to merge 1 commit into
Open
Conversation
On gfx1151 the Q6_K MMQ kernel runs at a flat 16-17 TFLOPS while the dequantize + hipBLASLt path drops to 11-12 TFLOPS once K >= 8192 (the ffn_down shapes of 8B models) and pays a full f16 dequantize round trip per prefill pass. At ne11 = 2048 the two paths trade places per shape (K = 4096 favours hipBLASLt, K >= 12288 favours MMQ), so the gate stops at 1024. Keep RDNA 3.0 and other targets on the existing threshold. Assisted-by: Claude
There was a problem hiding this comment.
🟢 Approval recommended
The functional change is narrowly scoped to RDNA 3.5 Q6_K routing and appears consistent with existing architecture-gated heuristics; only a minor inline-comment maintainability tweak was noted.
Pull request overview
This PR adjusts the HIP/CUDA MMQ routing heuristic for AMD RDNA 3.5 (gfx1151) so Q6_K matmuls use the MMQ kernel for larger prompt-side column counts (up to ne11 <= 1024), matching the Strix Halo performance measurements described in the PR.
Changes:
- Extend the
GGML_TYPE_Q6_KMMQ eligibility threshold tone11 <= 1024specifically onGGML_CUDA_CC_IS_RDNA3_5(cc). - Keep existing thresholds unchanged for RDNA 3.0 and all other architectures.
File summaries
| File | Description |
|---|---|
| ggml/src/ggml-cuda/mmq.cu | Updates ggml_cuda_should_use_mmq() to route RDNA 3.5 Q6_K matmuls through MMQ up to ne11 = 1024. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+556
to
+557
| // RDNA 3.5 (gfx1151): dequantize + hipBLASLt loses to MMQ up to ne11 = 1024; at 2048 the two | ||
| // paths trade places per shape (K = 4096 favours hipBLASLt, K >= 12288 favours MMQ), see PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
On gfx1151
ggml_cuda_should_use_mmqsends Q6_K matmuls with ne11 > 256 to dequantize + hipBLASLt. That path isslower than MMQ up to ne11 = 1024 on this GPU: the Q6_K MMQ kernel runs at a flat 16-17 TFLOPS on the fork's RDNA 3.5
tile rows, while the Tensile GEMM drops from 20-22 TFLOPS at K = 5376 to 11-12 TFLOPS for K >= 8192 at ne11 = 512
(the ffn_down shapes of 8B models; PMC: L2 hit rate 68% -> 38%, DRAM fetch 1.2x -> 2.8x the unique bytes once the
per-workgroup f16 weight panel outgrows the 2 MB L2) and the dequantize pass adds a full f16 write + read of every
weight per prefill pass. At ne11 = 2048 the two paths trade places per shape (K = 4096: hipBLASLt 0.85-1.0x of MMQ
time, K >= 12288: MMQ 1.17-1.35x faster), so the gate stops at 1024 rather than going to
return true. This PRraises the Q6_K threshold to
ne11 <= 1024underGGML_CUDA_CC_IS_RDNA3_5; RDNA 3.0 and everything else keep theexisting threshold.
Effect: prefill +8-10% on Q4_K_M / Q5_K_M files (their ffn_down and attn_v tensors are Q6_K), +30% on pure Q6_K files,
up to +50% at -ub 384 where the Tensile path has a hole, no change at -ub 2048 (gate off), no change on files without
Q6_K tensors in the prompt path (Q2_K, Q3_K_M, IQ4_XS, Q4_0, Q8_0, tested), no change to decode (MMVQ path). Measured across 12 GGUFs and three more models (Gemma-4-E4B, Qwen3.6-35B-A3B MoE, Gemma-4-31B:
no change / no change / inconclusive) before this PR; the tables below are the merge-base re-run required here.
Interaction with upstream ggml-org#25940: its Q6_K vec_dot change makes the MMQ side another ~1.25x faster
on gfx1151 (19-20 TFLOPS), which only widens the margin this rule relies on.
Measurements
Baseline:
pass 1: Qwen3-8B Q4_K_M -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf -p 512 -n 128 -r 5pass 2: Qwen3-8B Q4_K_M -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf -p 512 -n 128 -r 5Qwen3-8B Q4_K_M ubatch axis -p 2048 -n 0 -ub 128..2048 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf -p 2048 -n 0 -ub 128,256,384,512,1024,2048 -r 5Qwen3-8B Q6_K ubatch axis
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q6_K.gguf -p 2048 -n 0 -ub 128,256,384,512,1024,2048 -r 5Qwen3-8B Q6_K -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q6_K.gguf -p 512 -n 128 -r 5Llama-3.1-8B Q4_K_M -p 512 -n 128 -r 5
llama-bench -m Llama-3.1-8B-Instruct-GGUF/Llama-3.1-8B-Instruct-Q4_K_M.gguf -p 512 -n 128 -r 5Qwen3-8B Q8_0 (no Q6_K tensors: control) -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q8_0.gguf -p 512 -n 128 -r 5After:
pass 1: Qwen3-8B Q4_K_M -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf -p 512 -n 128 -r 5pass 2: Qwen3-8B Q4_K_M -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf -p 512 -n 128 -r 5Qwen3-8B Q4_K_M ubatch axis -p 2048 -n 0 -ub 128..2048 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q4_K_M.gguf -p 2048 -n 0 -ub 128,256,384,512,1024,2048 -r 5Qwen3-8B Q6_K ubatch axis
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q6_K.gguf -p 2048 -n 0 -ub 128,256,384,512,1024,2048 -r 5Qwen3-8B Q6_K -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q6_K.gguf -p 512 -n 128 -r 5Llama-3.1-8B Q4_K_M -p 512 -n 128 -r 5
llama-bench -m Llama-3.1-8B-Instruct-GGUF/Llama-3.1-8B-Instruct-Q4_K_M.gguf -p 512 -n 128 -r 5Qwen3-8B Q8_0 (no Q6_K tensors: control) -p 512 -n 128 -r 5
llama-bench -m Qwen3-8B-GGUF/Qwen3-8B-Q8_0.gguf -p 512 -n 128 -r 5Correctness:
test-backend-ops test -b ROCm0 -o MUL_MAT,MUL_MAT_IDon the change build:llama-perplexity -f wiki.test.raw -c 512 -b 512 -ub 512 --chunks 32 -ngl 99(this change only moves prompt-sized Q6_K matmuls from hipBLASLt f16 to MMQ int8, so PPL is the oracle):Additional information
an RDNA 3.5 Q6_K MMQ regression. A K-aware gate (MMQ for K >= 12288 at any ne11) would add another +17-35% on the
ffn_down shapes at -ub 2048, but
ggml_cuda_should_use_mmqdoes not receive K today; left as a follow-up.return true; its -ub 2048 row for the pure Q6_K file was 2.7% belowbaseline (1063.6 vs 1093.6 t/s), which is why the gate was lowered to 1024 and everything re-run. That log is
kept as
logs/strixpr-20260910-run1-returntrue.login the research repo. The decision data (kernel traces, PMC, K-sweeps) is in the research notes linkedfrom the branch description; happy to paste more here.
Requirements
belong in halo-box/llama.cpp instead
5-line change, ran the interleaved A/B harness and this PR's benchmark protocol, and wrote this description. The
owner reviewed the change and the numbers.
HIP_LAUNCH_BLOCKING=1; CDNA and RDNA 4 are notaffected by the gate but were not built or run; the Vulkan backend is untouched.
🤖 Generated with Claude Code
https://claude.ai/code/session_01CzgPbiT4PgFRYgLXRob86e