Skip to content

metal : dequantize q8_0 using packed types - #27370

Merged
ggerganov merged 1 commit into
masterfrom
gg/metal-dequantize-q8_0-packed
Aug 19, 2026
Merged

metal : dequantize q8_0 using packed types#27370
ggerganov merged 1 commit into
masterfrom
gg/metal-dequantize-q8_0-packed

Conversation

@ggerganov

Copy link
Copy Markdown
Member

Overview

Extract the packed_char4 pattern from @tarruda's #25556. This is a free win for -ctk q8_0 -ctv q8_0 on Metal with large contexts and it's a simple change.

Additional information

./scripts/compare-commits.sh master gg/metal-dequantize-q8_0-packed llama-bench -hf ggml-org/Qwen3.6-35B-A3B-GGUF:Q8_0 -ub 2048 -d 0,8192,16384,32768,65536 -n 32 -r 5 -fa 1 -ctk q8_0 -ctv q8_0

| CPU           | Model                  | Test         |   t/s master |   t/s PR  |   Speedup |
|:--------------|:-----------------------|:-------------|-------------:|----------:|----------:|
|Apple M2 Ultra | qwen35moe 35B.A3B Q8_0 | pp512        |      1976.83 |   1989.83 |      1.01 |
|Apple M2 Ultra | qwen35moe 35B.A3B Q8_0 | pp512@d8192  |      1337.86 |   1338.22 |      1.00 |
|Apple M2 Ultra | qwen35moe 35B.A3B Q8_0 | pp512@d16384 |      1004.65 |   1004.76 |      1.00 |
|Apple M2 Ultra | qwen35moe 35B.A3B Q8_0 | pp512@d32768 |       672.73 |    671.86 |      1.00 |
|Apple M2 Ultra | qwen35moe 35B.A3B Q8_0 | pp512@d65536 |       399.99 |    399.37 |      1.00 |
|Apple M2 Ultra | qwen35moe 35B.A3B Q8_0 | tg32         |        74.07 |     75.17 |      1.01 |
|Apple M2 Ultra | qwen35moe 35B.A3B Q8_0 | tg32@d8192   |        58.59 |     61.40 |      1.05 |
|Apple M2 Ultra | qwen35moe 35B.A3B Q8_0 | tg32@d16384  |        46.21 |     50.30 |      1.09 |
|Apple M2 Ultra | qwen35moe 35B.A3B Q8_0 | tg32@d32768  |        32.57 |     36.96 |      1.13 |
|Apple M2 Ultra | qwen35moe 35B.A3B Q8_0 | tg32@d65536  |        20.42 |     23.98 |      1.17 |

Requirements

@ggerganov
ggerganov requested a review from a team as a code owner August 19, 2026 08:48
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning Apple Metal https://en.wikipedia.org/wiki/Metal_(API) labels Aug 19, 2026
@ggerganov
ggerganov merged commit 8ef78e6 into master Aug 19, 2026
24 of 25 checks passed
@ggerganov
ggerganov deleted the gg/metal-dequantize-q8_0-packed branch August 19, 2026 11:10
ravel7524 pushed a commit to ravel7524/llama.cpp that referenced this pull request Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Apple Metal https://en.wikipedia.org/wiki/Metal_(API) ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant