powerpc: add FP16 MMA path for Q4/Q8 matmul by shalinib-ibm · Pull Request #19709 · ggml-org/llama.cpp

shalinib-ibm · 2026-02-18T10:51:09Z

Avoid xvi8ger4pp signed→unsigned bias correction by dequantizing Q4/Q8 inputs to FP16 and using FP16×FP16→FP32 MMA. This removes post-processing overhead and improves performance.

Performance Impact:
1.5 ~ 2x improvement in PP_Speed for Q4 and Q8 Models, measured with llama-bench and llama-batched-bench. Q8 Model: granite-4.0-h-micro-Q8_0.gguf (from huggingface) Q4 Model: Meta-Llama3-8b Q4 model (generated with llama-quantize from f32 model)

llama-bench Q8 Model Results:
model size params backend threads test Base t/s Patch t/s
granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp8 64.48 ± 4.72 73.99 ± 0.27
granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp16 80.11 ± 0.32 112.53 ± 0.40
granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp32 89.10 ± 0.27 152.95 ± 0.68
granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp64 93.65 ± 0.25 187.83 ± 0.83
granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp128 99.93 ± 0.02 201.32 ± 0.11
granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp256 102.32 ± 0.40 208.32 ± 0.41
granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp512 103.42 ± 0.40 209.98 ± 0.14
granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 tg128 20.35 ± 0.01 19.57 ± 0.01

llama-bench Q4 Model Results:
model size params backend threads test Base t/s Patch t/s
llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp8 34.77 ± 0.10 41.23 ± 0.08
llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp16 40.81 ± 0.04 64.55 ± 0.15
llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp32 44.65 ± 0.05 90.84 ± 0.22
llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp64 47.49 ± 0.03 114.39 ± 0.11
llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp128 49.29 ± 0.24 120.13 ± 0.19
llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp256 49.77 ± 0.23 121.51 ± 0.11
llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp512 49.89 ± 0.23 117.52 ± 0.10
llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 tg128 13.40 ± 0.01 13.37 ± 0.00

Results with Q8 Model llama-cli
.//bin/llama-cli -m granite-4.0-h-micro-Q8_0.gguf -p "The journey into the heart of the Chamber of Whispering Archives became a descent into a psychological labyrinth. The low hum, once merely an ambient sound, now resonated within my very bones, a thousand overlapping frequencies that threatened to dismantle my sanity. The violet light, a constant companion, began to pulse and flicker with a manic rhythm, painting the crystalline rods in a dazzling, disorienting array of shifting colors. Each rod, each frozen memory, seemed to awaken as I passed. I no longer just felt the latent whispers; I saw them. Faint, ethereal apparitions of long-dead beings flickered around me, their faces etched with a silent, profound sorrow. They were not malicious, but their collective grief was a physical weight, pressing in on me from all sides. I was an intruder in their eternal wake, a dissonant chord in their symphony of loss" -t 20 -no-cnv

  Base : 95.2 t/s PP; 26.5 t/s TG
Patch : 164.9 t/s PP; 26.8 t/s TG

Llama Perplexity Results:

granite-4.0-h-micro-Q8_0
Base Final PPL Estimate: 1.3862 +/- 0.04424
Patch Final PPL Estimate: 1.3868 +/- 0.04432

Meta-Llama3-8b Q4
Base Final PPL Estimate: 1.3801 +/- 0.04116
Patch Final PPL Estimate: 1.3803 +/- 0.04116

Make sure to read the contributing guidelines before submitting a PR

shalinib-ibm · 2026-02-18T11:32:32Z

@taronaeo Can you please help review this change ?

taronaeo · 2026-02-18T12:03:55Z

I'll review in a while. Quite curious, why was ggml/src/ggml-cpu/llamafile/sgemm-ppc.h moved into sgemm.h again? I remember there was a PR separating the code.

shalinib-ibm · 2026-02-18T12:22:23Z

I'll review in a while. Quite curious, why was ggml/src/ggml-cpu/llamafile/sgemm-ppc.h moved into sgemm.h again? I remember there was a PR separating the code.
Hi @taronaeo ,
The original split (moving the PPC SGEMM code into llamafile/sgemm-ppc.h) was done when we introduced the Q4/Q8 tiled GEMM path. At that point, separating the files helped the compiler optimize the tiled GEMM and the mnpack paths independently and kept the PPC-specific Q4/Q8 changes scoped.
With the current patch, that separation is no longer required for optimisation benefit. The code paths are now aligned with how other data types FP32 , BF16 , FP16 handle this ( everything kept under single file sgemm.cpp in saperate classes).

So the move back into sgemm.cpp is mainly for consistency and simpler maintenance, while preserving the same generated code and performance characteristics.

taronaeo

Minor styling changes but otherwise LGTM. I didn't highlight all the styling changes but there are more :)

taronaeo · 2026-02-18T14:09:44Z

+        for (int i = 0; i < 8; ++i)
+            vec_xst(s[i], 0, (vec_t*)(vecOffset + i * 16));


Suggested change

for (int i = 0; i < 8; ++i)

vec_xst(s[i], 0, (vec_t*)(vecOffset + i * 16));

for (int i = 0; i < 8; ++i) {

vec_xst(s[i], 0, (vec_t*)(vecOffset + i * 16));

}

Looks like this is still missing

taronaeo · 2026-02-18T14:10:28Z

-   template<int size>
-   void tinyBLAS_Q0_PPC<TA>::packNormalInt4(const TA* a, int64_t lda, int rows, int cols, int8_t* vec, std::array<int, size>& comparray) {
+  private:
+    inline void save_res(int ii, int jj, int idx, vector float* fin_res, int RM=4, int RN=4) {


Suggested change

inline void save_res(int ii, int jj, int idx, vector float* fin_res, int RM=4, int RN=4) {

inline void save_res(int ii, int jj, int idx, vector float * fin_res, int RM = 4, int RN = 4) {

taronaeo · 2026-02-18T14:10:50Z

+    }
+
+    template<typename ArrayType>
+    inline void compute(acc_t* ACC, int c_idx, int s_idx, ArrayType& comparray, vector float* vs, vector float* fin_res) {


Suggested change

inline void compute(acc_t* ACC, int c_idx, int s_idx, ArrayType& comparray, vector float* vs, vector float* fin_res) {

inline void compute(acc_t* ACC, int c_idx, int s_idx, ArrayType & comparray, vector float * vs, vector float * fin_res) {

taronaeo · 2026-02-18T14:11:01Z

+        }
+    }
+
+    inline void process_q4_elements(vector signed char (&c)[2], int* ca) {


Suggested change

inline void process_q4_elements(vector signed char (&c)[2], int* ca) {

inline void process_q4_elements(vector signed char (&c)[2], int * ca) {

taronaeo · 2026-02-18T14:11:19Z

+    }
+
+    template <typename V1, typename V2>
+    inline void vector_permute_store(V2 &s1, V2 &s2, V2 &s3, V2 &s4, V1 *vecOffset, bool flip) {


Suggested change

inline void vector_permute_store(V2 &s1, V2 &s2, V2 &s3, V2 &s4, V1 *vecOffset, bool flip) {

inline void vector_permute_store(V2 & s1, V2 & s2, V2 & s3, V2 & s4, V1 * vecOffset, bool flip) {

taronaeo · 2026-02-18T14:11:37Z

+        vec_xst(t8, 0, vecOffset+48);
+    }
+
+    inline void unpack_q4_to_q8( vector signed char packed, vector signed char &lo, vector signed char &hi) {


Suggested change

inline void unpack_q4_to_q8( vector signed char packed, vector signed char &lo, vector signed char &hi) {

inline void unpack_q4_to_q8(vector signed char packed, vector signed char & lo, vector signed char & hi) {

taronaeo · 2026-02-18T14:11:49Z

+        hi = vec_sub(hi, v8);
+    }
+
+    inline void vector_permute_store_fp16(vec_t *c, unsigned char *vecOffset) {


Suggested change

inline void vector_permute_store_fp16(vec_t *c, unsigned char *vecOffset) {

inline void vector_permute_store_fp16(vec_t * c, unsigned char * vecOffset) {

taronaeo · 2026-02-18T14:12:09Z

+            vec_xst(s[i], 0, (vec_t*)(vecOffset + i * 16));
+    }
+
+    static inline void convert_and_scale_q8( vector signed char raw, vector float v_scale,  vector unsigned short& out_hi, vector unsigned short& out_lo) {


Suggested change

static inline void convert_and_scale_q8( vector signed char raw, vector float v_scale, vector unsigned short& out_hi, vector unsigned short& out_lo) {

static inline void convert_and_scale_q8(vector signed char raw, vector float v_scale, vector unsigned short & out_hi, vector unsigned short & out_lo) {

taronaeo · 2026-02-18T14:12:21Z

+        out_lo = vec_pack_to_short_fp32(vec_mul(f_lo_h, v_scale), vec_mul(f_lo_l, v_scale));
+    }
+
+    void packNormal_q4_fp16(const block_q4_0* a, int64_t lda, int rows, int blocks, unsigned char* vec) {


Suggested change

void packNormal_q4_fp16(const block_q4_0* a, int64_t lda, int rows, int blocks, unsigned char* vec) {

void packNormal_q4_fp16(const block_q4_0 * a, int64_t lda, int rows, int blocks, unsigned char * vec) {

@taronaeo Thanks for pointing it out. I have followed the style you mentioned for this class ( tinyBLAS_Q0_PPC) and made appropriate changes. Can you please take a look when you get time ?

taronaeo · 2026-02-19T02:52:36Z

+        vec_xst(t8, 0, vecOffset+48);
+    }
+
+    inline void unpack_q4_to_q8( vector signed char packed, vector signed char & lo, vector signed char & hi) {


Suggested change

inline void unpack_q4_to_q8( vector signed char packed, vector signed char & lo, vector signed char & hi) {

inline void unpack_q4_to_q8(vector signed char packed, vector signed char & lo, vector signed char & hi) {

taronaeo · 2026-02-19T02:53:19Z

+        for (int i = 0; i < 8; ++i)
+            vec_xst(s[i], 0, (vec_t*)(vecOffset + i * 16));


Looks like this is still missing

taronaeo · 2026-02-19T02:53:40Z

+            vec_xst(s[i], 0, (vec_t*)(vecOffset + i * 16));
+    }
+
+    static inline void convert_and_scale_q8( vector signed char raw, vector float v_scale,  vector unsigned short & out_hi, vector unsigned short & out_lo) {


Suggested change

static inline void convert_and_scale_q8( vector signed char raw, vector float v_scale, vector unsigned short & out_hi, vector unsigned short & out_lo) {

static inline void convert_and_scale_q8(vector signed char raw, vector float v_scale, vector unsigned short & out_hi, vector unsigned short & out_lo) {

taronaeo · 2026-02-19T02:54:06Z

+            for (int blk = 0; blk < blocks; blk++) {
+                vector unsigned short hp_res[8][4];
+                for (int r = 0; r < 8; r++) {
+                    const block_q4_0* current_blk = rows_base[r] + blk;


Suggested change

const block_q4_0* current_blk = rows_base[r] + blk;

const block_q4_0 * current_blk = rows_base[r] + blk;

taronaeo · 2026-02-19T02:54:39Z

+		rows_base[r] = a + (i + r) * lda;
+	    }


taronaeo · 2026-02-19T04:11:11Z

+	}
+    }
+
+    static inline void convert_and_scale_q8(vector signed char raw, vector float v_scale,  vector unsigned short & out_hi, vector unsigned short & out_lo) {


Suggested change

static inline void convert_and_scale_q8(vector signed char raw, vector float v_scale, vector unsigned short & out_hi, vector unsigned short & out_lo) {

static inline void convert_and_scale_q8(vector signed char raw, vector float v_scale, vector unsigned short & out_hi, vector unsigned short & out_lo) {

taronaeo · 2026-02-19T04:16:35Z

Okay merge on CI green + the suggested changes :)

shalinib-ibm · 2026-02-19T04:49:22Z

Okay merge on CI green + the suggested changes :)

Thanks @taronaeo for your time reviewing this. I have added the perplexity results also to the commit.

Avoid xvi8ger4pp signed→unsigned bias correction by dequantizing Q4/Q8 inputs to FP16 and using FP16×FP16→FP32 MMA. This removes post-processing overhead and improves performance. Performance Impact: 1.5 ~ 2x improvement in PP_Speed for Q4 and Q8 Models, measured with llama-bench and llama-batched-bench. Q8 Model: granite-4.0-h-micro-Q8_0.gguf (from huggingface) Q4 Model: Meta-Llama3-8b Q4 model (generated with llama-quantize from f32 model) llama-bench Q8 Model Results: model size params backend threads test Base t/s Patch t/s granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp8 64.48 ± 4.72 73.99 ± 0.27 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp16 80.11 ± 0.32 112.53 ± 0.40 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp32 89.10 ± 0.27 152.95 ± 0.68 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp64 93.65 ± 0.25 187.83 ± 0.83 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp128 99.93 ± 0.02 201.32 ± 0.11 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp256 102.32 ± 0.40 208.32 ± 0.41 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp512 103.42 ± 0.40 209.98 ± 0.14 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 tg128 20.35 ± 0.01 19.57 ± 0.01 llama-bench Q4 Model Results: model size params backend threads test Base t/s Patch t/s llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp8 34.77 ± 0.10 41.23 ± 0.08 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp16 40.81 ± 0.04 64.55 ± 0.15 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp32 44.65 ± 0.05 90.84 ± 0.22 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp64 47.49 ± 0.03 114.39 ± 0.11 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp128 49.29 ± 0.24 120.13 ± 0.19 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp256 49.77 ± 0.23 121.51 ± 0.11 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp512 49.89 ± 0.23 117.52 ± 0.10 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 tg128 13.40 ± 0.01 13.37 ± 0.00 Llama perplexity Results: Model Base Final PPL Estimate Patch Final PPL Estimate granite-4.0-h-micro-Q8_0 1.3862 +/- 0.04424 1.3868 +/- 0.04432 Meta-Llama3-8b Q4 1.3801 +/- 0.04116 1.3803 +/- 0.04116 Signed-off-by: Shalini.Salomi.Bodapati <Shalini.Salomi.Bodapati@ibm.com>

shalinib-ibm · 2026-02-19T06:11:13Z

@taronaeo I have run the test that failed in CI (ubuntu-24-cmake-vulkan) in my local environment with this change.
Red Hat Linux ppc64le Power10 Box .The test passed. Below is the log.

 ./test-tokenizer-0 ../../models/ggml-vocabs/PLaMo2/ggml-vocab-plamo2.gguf
main : reading vocab from: '../../models/ggml-vocabs/PLaMo2/ggml-vocab-plamo2.gguf'
llama_model_loader: loaded meta data with 30 key-value pairs and 0 tensors from ../../models/ggml-vocabs/PLaMo2/ggml-vocab-plamo2.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = plamo2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Plamo2
llama_model_loader: - kv   3:                          plamo2.vocab_size u32              = 100032
llama_model_loader: - kv   4:             plamo2.attention.head_count_kv arr[i32,32]      = [0, 4, 0, 4, 0, 4, 0, 4, 0, 4, 0, 4, ...
llama_model_loader: - kv   5:                      plamo2.context_length u32              = 10485760
llama_model_loader: - kv   6:                    plamo2.embedding_length u32              = 4096
llama_model_loader: - kv   7:                         plamo2.block_count u32              = 32
llama_model_loader: - kv   8:                plamo2.attention.head_count u32              = 32
llama_model_loader: - kv   9:    plamo2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  10:                      plamo2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  11:                      plamo2.ssm.state_size u32              = 64
llama_model_loader: - kv  12:                     plamo2.ssm.conv_kernel u32              = 4
llama_model_loader: - kv  13:                  plamo2.ssm.time_step_rank u32              = 64
llama_model_loader: - kv  14:                      plamo2.ssm.inner_size u32              = 8192
llama_model_loader: - kv  15:                     plamo2.ssm.group_count u32              = 0
llama_model_loader: - kv  16:                 plamo2.feed_forward_length u32              = 16384
llama_model_loader: - kv  17:                          general.file_type u32              = 1
llama_model_loader: - kv  18:               general.quantization_version u32              = 2
llama_model_loader: - kv  19:                       tokenizer.ggml.model str              = plamo2
llama_model_loader: - kv  20:                         tokenizer.ggml.pre str              = default
llama_model_loader: - kv  21:                      tokenizer.ggml.tokens arr[str,100032]  = ["<|plamo:unk|>", "<|plamo:bos|>", "<...
llama_model_loader: - kv  22:                      tokenizer.ggml.scores arr[f32,100032]  = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  23:                  tokenizer.ggml.token_type arr[i32,100032]  = [2, 3, 3, 3, 3, 3, 3, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  24:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  25:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  26:            tokenizer.ggml.padding_token_id u32              = 3
llama_model_loader: - kv  27:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  28:                tokenizer.ggml.eot_token_id u32              = 4
llama_model_loader: - kv  29:            tokenizer.ggml.add_space_prefix bool             = false
print_info: file format = GGUF V3 (latest)
print_info: file type   = F16
print_info: file size   = 0.00 MiB (nan BPW) 
init_tokenizer: initializing tokenizer for type 6
load: 0 unused tokens
load: control token:      3 '<|plamo:pad|>' is not marked as EOG
load: control token:    127 '<|plamo:reserved:0x7F|>' is not marked as EOG
load: control token:     18 '<|plamo:reserved:0x12|>' is not marked as EOG
load: control token:    286 '<|plamo:reserved:0x11E|>' is not marked as EOG
load: control token:     26 '<|plamo:reserved:0x1A|>' is not marked as EOG
load: control token:     15 '<|plamo:reserved:0x0F|>' is not marked as EOG
load: control token:     31 '<|plamo:reserved:0x1F|>' is not marked as EOG
load: control token:     29 '<|plamo:reserved:0x1D|>' is not marked as EOG
load: control token:     20 '<|plamo:reserved:0x14|>' is not marked as EOG
load: control token:     22 '<|plamo:reserved:0x16|>' is not marked as EOG
load: control token:    287 '<|plamo:reserved:0x11F|>' is not marked as EOG
load: control token:      4 '<|plamo:op|>' is not marked as EOG
load: control token:     16 '<|plamo:reserved:0x10|>' is not marked as EOG
load: control token:      5 '<|plamo:file_separator|>' is not marked as EOG
load: control token:      2 '<|plamo:eos|>' is not marked as EOG
load: control token:     21 '<|plamo:reserved:0x15|>' is not marked as EOG
load: control token:      1 '<|plamo:bos|>' is not marked as EOG
load: control token:      6 '<file_sep>' is not marked as EOG
load: control token:     30 '<|plamo:reserved:0x1E|>' is not marked as EOG
load: control token:     17 '<|plamo:reserved:0x11|>' is not marked as EOG
load: control token:     25 '<|plamo:reserved:0x19|>' is not marked as EOG
load: control token:     23 '<|plamo:reserved:0x17|>' is not marked as EOG
load: control token:     14 '<|plamo:reserved:0x0E|>' is not marked as EOG
load: control token:     27 '<|plamo:reserved:0x1B|>' is not marked as EOG
load: control token:     19 '<|plamo:reserved:0x13|>' is not marked as EOG
load: control token:     24 '<|plamo:reserved:0x18|>' is not marked as EOG
load: control token:     28 '<|plamo:reserved:0x1C|>' is not marked as EOG
load: control token:    383 '<|plamo:reserved:0x17F|>' is not marked as EOG
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 1 ('<|plamo:bos|>')
load:   - 4 ('<|plamo:op|>')
load: special tokens cache size = 29
load: token to piece cache size = 0.7989 MB
print_info: arch                  = plamo2
print_info: vocab_only            = 1
print_info: no_alloc              = 0
print_info: ssm_d_conv            = 0
print_info: ssm_d_inner           = 0
print_info: ssm_d_state           = 0
print_info: ssm_dt_rank           = 0
print_info: ssm_n_group           = 0
print_info: ssm_dt_b_c_rms        = 0
print_info: model type            = ?B
print_info: model params          = 0.00 K
print_info: general.name          = Plamo2
print_info: vocab type            = PLaMo2
print_info: n_vocab               = 100032
print_info: n_merges              = 0
print_info: BOS token             = 1 '<|plamo:bos|>'
print_info: EOS token             = 1 '<|plamo:bos|>'
print_info: EOT token             = 4 '<|plamo:op|>'
print_info: UNK token             = 0 '<|plamo:unk|>'
print_info: PAD token             = 3 '<|plamo:pad|>'
print_info: LF token              = 10 '
'
print_info: EOG token             = 1 '<|plamo:bos|>'
print_info: EOG token             = 4 '<|plamo:op|>'
print_info: max token length      = 48
llama_model_load: vocab only - skipping tensors
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 512
llama_context: n_ctx_seq     = 512
llama_context: n_batch       = 512
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = false
llama_context: freq_base     = 0.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (512) > n_ctx_train (0) -- possible training context overflow

src: ''
res: ''
tok: 

src: '	'
res: '	'
tok: 9 

src: '	
'
res: '	
'
tok: 9 10 

src: '
'
res: '
'
tok: 10 

src: '

'
res: '

'
tok: 515 

src: '


'
res: '


'
tok: 515 10 

src: '
 

 


 	 		 	
  
   
    
     
🚀 (normal) 😶‍🌫️ (multiple emojis concatenated) ✅ 🦙🦙 3 33 333 3333 33333 333333 3333333 33333333 3.3 3..3 3...3 កាន់តែពិសេសអាច😁 ?我想在apple工作1314151天～ ------======= нещо на Български ''''''```````""""......!!!!!!?????? I've been 'told he's there, 'RE you sure? 'M not sure I'll make it, 'D you like some tea? We'Ve a'lL'
res: '
 

 


 	 		 	
  
   
    
     
🚀 (normal) 😶‍🌫️ (multiple emojis concatenated) ✅ 🦙🦙 3 33 333 3333 33333 333333 3333333 33333333 3.3 3..3 3...3 កាន់តែពិសេសអាច😁 ?我想在apple工作1314151天～ ------======= нещо на Български ''''''```````""""......!!!!!!?????? I've been 'told he's there, 'RE you sure? 'M not sure I'll make it, 'D you like some tea? We'Ve a'lL'
tok: 10 32 515 32 515 10 32 9 32 512 32 9 45058 45058 32 45059 45059 32 10 240 159 154 128 296 4276 45173 240 159 152 182 226 128 141 240 159 140 171 8935 296 14313 3007 19543 115 32 45718 100 45173 826 32 240 159 166 153 240 159 166 153 307 307 51 307 51 51 307 51 51 51 307 51 51 51 51 307 51 51 51 51 51 307 51 51 51 51 51 51 307 51 51 51 51 51 51 51 307 279 307 46 279 307 696 279 32 225 158 128 225 158 182 225 158 147 225 159 139 225 158 143 225 159 130 225 158 150 225 158 183 225 158 159 225 159 129 225 158 159 225 158 162 225 158 182 225 158 133 536 319 50637 52576 48477 15644 54263 49 51 49 52 49 53 49 47787 47112 675 45 699 45293 97150 97142 9622 99366 99391 99382 15363 21839 99377 99388 295 39 39 39 39 39 45311 45311 96 34 34 34 34 697 697 14022 33 11248 11248 12207 295 40368 5948 1214 44 295 3215 1099 1901 63 295 77 12216 6832 4736 44 295 68 1099 1132 1131 6492 11862 39 21382 353 39 108 76 

src: '
 ='
res: '
 ='
tok: 10 317 

src: ' '
res: ' '
tok: 32 

src: '  '
res: '  '
tok: 288 

src: '   '
res: '   '
tok: 288 32 

src: '    Hello'
res: '    Hello'
tok: 516 6721 

src: '    Hello
    Hello'
res: '    Hello
    Hello'
tok: 516 6721 45059 6721 

src: '   Hello'
res: '   Hello'
tok: 288 19531 

src: '  Hello'
res: '  Hello'
tok: 288 6721 

src: ' ('
res: ' ('
tok: 296 

src: ' Hello'
res: ' Hello'
tok: 19531 

src: ' Hello World'
res: ' Hello World'
tok: 19531 2375 

src: ' Hello World!'
res: ' Hello World!'
tok: 19531 2375 33 

src: ' Hello world'
res: ' Hello world'
tok: 19531 1462 

src: ' Hello, world!'
res: ' Hello, world!'
tok: 19531 44 1462 33 

src: ' discards'
res: ' discards'
tok: 24189 115 

src: ' this is 🦙.cpp'
res: ' this is 🦙.cpp'
tok: 1087 3337 240 159 166 153 46 45776 

src: '!!!!!!'
res: '!!!!!!'
tok: 14022 33 

src: '' era'
res: '' era'
tok: 39 6462 

src: '3'
res: '3'
tok: 51 

src: '33'
res: '33'
tok: 51 51 

src: '333'
res: '333'
tok: 51 51 51 

src: '3333'
res: '3333'
tok: 51 51 51 51 

src: '33333'
res: '33333'
tok: 51 51 51 51 51 

src: '333333'
res: '333333'
tok: 51 51 51 51 51 51 

src: '3333333'
res: '3333333'
tok: 51 51 51 51 51 51 51 

src: '33333333'
res: '33333333'
tok: 51 51 51 51 51 51 51 51 

src: '333333333'
res: '333333333'
tok: 51 51 51 51 51 51 51 51 51 

src: 'Cửa Việt'
res: 'Cửa Việt'
tok: 67 98263 97 6110 98241 116 

src: 'Hello'
res: 'Hello'
tok: 6721 

src: 'Hello World'
res: 'Hello World'
tok: 6721 2375 

src: 'Hello world'
res: 'Hello world'
tok: 6721 1462 

src: 'Hello, world!'
res: 'Hello, world!'
tok: 6721 44 1462 33 

src: 'Hello, y'all! How are you 😁 ?我想在apple工作1314151天～'
res: 'Hello, y'all! How are you 😁 ?我想在apple工作1314151天～'
tok: 6721 44 377 39 1499 33 3476 1089 28316 536 319 50637 52576 48477 15644 54263 49 51 49 52 49 53 49 47787 47112 

src: 'ied 4 ½ months'
res: 'ied 4 ½ months'
tok: 6466 308 32 789 2343 

src: 'w048 7tuijk dsdfhu'
res: 'w048 7tuijk dsdfhu'
tok: 119 48 52 56 311 6107 44843 32 3605 4099 5190 

src: 'Äpfel'
res: 'Äpfel'
tok: 195 132 112 32384 

src: 'нещо на Български'
res: 'нещо на Български'
tok: 99373 97142 9622 99366 99391 99382 15363 21839 99377 99388 

src: 'កាន់តែពិសេសអាចខលចេញ'
res: 'កាន់តែពិសេសអាចខលចេញ'
tok: 225 158 128 225 158 182 225 158 147 225 159 139 225 158 143 225 159 130 225 158 150 225 158 183 225 158 159 225 159 129 225 158 159 225 158 162 225 158 182 225 158 133 225 158 129 225 158 155 225 158 133 225 159 129 225 158 137 

src: '🚀 (normal) 😶‍🌫️ (multiple emojis concatenated) ✅ (only emoji that has its own token)'
res: '🚀 (normal) 😶‍🌫️ (multiple emojis concatenated) ✅ (only emoji that has its own token)'
tok: 240 159 154 128 296 4276 45173 240 159 152 182 226 128 141 240 159 140 171 8935 296 14313 3007 19543 115 32 45718 100 45173 826 296 4193 3007 19543 1081 1107 5276 4282 41 

Tests passed

taronaeo · 2026-02-19T06:27:38Z

Yup, failure unrelated to this PR. Merging.

Avoid xvi8ger4pp signed→unsigned bias correction by dequantizing Q4/Q8 inputs to FP16 and using FP16×FP16→FP32 MMA. This removes post-processing overhead and improves performance. Performance Impact: 1.5 ~ 2x improvement in PP_Speed for Q4 and Q8 Models, measured with llama-bench and llama-batched-bench. Q8 Model: granite-4.0-h-micro-Q8_0.gguf (from huggingface) Q4 Model: Meta-Llama3-8b Q4 model (generated with llama-quantize from f32 model) llama-bench Q8 Model Results: model size params backend threads test Base t/s Patch t/s granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp8 64.48 ± 4.72 73.99 ± 0.27 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp16 80.11 ± 0.32 112.53 ± 0.40 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp32 89.10 ± 0.27 152.95 ± 0.68 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp64 93.65 ± 0.25 187.83 ± 0.83 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp128 99.93 ± 0.02 201.32 ± 0.11 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp256 102.32 ± 0.40 208.32 ± 0.41 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 pp512 103.42 ± 0.40 209.98 ± 0.14 granitehybrid 3B Q8_0 3.16 GiB 3.19 B CPU 10 tg128 20.35 ± 0.01 19.57 ± 0.01 llama-bench Q4 Model Results: model size params backend threads test Base t/s Patch t/s llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp8 34.77 ± 0.10 41.23 ± 0.08 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp16 40.81 ± 0.04 64.55 ± 0.15 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp32 44.65 ± 0.05 90.84 ± 0.22 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp64 47.49 ± 0.03 114.39 ± 0.11 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp128 49.29 ± 0.24 120.13 ± 0.19 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp256 49.77 ± 0.23 121.51 ± 0.11 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 pp512 49.89 ± 0.23 117.52 ± 0.10 llama 8B Q4_0 4.33 GiB 8.03 B CPU 10 tg128 13.40 ± 0.01 13.37 ± 0.00 Llama perplexity Results: Model Base Final PPL Estimate Patch Final PPL Estimate granite-4.0-h-micro-Q8_0 1.3862 +/- 0.04424 1.3868 +/- 0.04432 Meta-Llama3-8b Q4 1.3801 +/- 0.04116 1.3803 +/- 0.04116 Signed-off-by: Shalini.Salomi.Bodapati <Shalini.Salomi.Bodapati@ibm.com>

shalinib-ibm requested a review from ggerganov as a code owner February 18, 2026 10:51

github-actions Bot added the ggml changes relating to the ggml tensor library for machine learning label Feb 18, 2026

shalinib-ibm force-pushed the ppc_q0_matmul_with_fp16_mma branch 2 times, most recently from 5e66199 to d1e15ea Compare February 18, 2026 11:13

shalinib-ibm force-pushed the ppc_q0_matmul_with_fp16_mma branch from d1e15ea to bd3507c Compare February 18, 2026 11:50

taronaeo approved these changes Feb 18, 2026

View reviewed changes

shalinib-ibm force-pushed the ppc_q0_matmul_with_fp16_mma branch 3 times, most recently from 908a4c6 to 6a026ff Compare February 18, 2026 17:10

taronaeo reviewed Feb 19, 2026

View reviewed changes

shalinib-ibm force-pushed the ppc_q0_matmul_with_fp16_mma branch from 6a026ff to 37bdf1c Compare February 19, 2026 03:13

taronaeo reviewed Feb 19, 2026

View reviewed changes

shalinib-ibm force-pushed the ppc_q0_matmul_with_fp16_mma branch from 37bdf1c to 7a8ede2 Compare February 19, 2026 04:43

shalinib-ibm force-pushed the ppc_q0_matmul_with_fp16_mma branch from 7a8ede2 to 7013d4a Compare February 19, 2026 04:57

taronaeo merged commit 3bb2fcc into ggml-org:master Feb 19, 2026
77 of 78 checks passed

shalinib-ibm mentioned this pull request Mar 20, 2026

ggml-cpu: add always_inline to tinyBLAS_PPC accumulator saves #20791

Merged

		for (int i = 0; i < 8; ++i)
		vec_xst(s[i], 0, (vec_t)(vecOffset + i 16));

	inline void save_res(int ii, int jj, int idx, vector float* fin_res, int RM=4, int RN=4) {
	inline void save_res(int ii, int jj, int idx, vector float * fin_res, int RM = 4, int RN = 4) {

	inline void compute(acc_t* ACC, int c_idx, int s_idx, ArrayType& comparray, vector float* vs, vector float* fin_res) {
	inline void compute(acc_t* ACC, int c_idx, int s_idx, ArrayType & comparray, vector float * vs, vector float * fin_res) {

	inline void process_q4_elements(vector signed char (&c)[2], int* ca) {
	inline void process_q4_elements(vector signed char (&c)[2], int * ca) {

	inline void vector_permute_store(V2 &s1, V2 &s2, V2 &s3, V2 &s4, V1 *vecOffset, bool flip) {
	inline void vector_permute_store(V2 & s1, V2 & s2, V2 & s3, V2 & s4, V1 * vecOffset, bool flip) {

	inline void unpack_q4_to_q8( vector signed char packed, vector signed char &lo, vector signed char &hi) {
	inline void unpack_q4_to_q8(vector signed char packed, vector signed char & lo, vector signed char & hi) {

	inline void vector_permute_store_fp16(vec_t c, unsigned char vecOffset) {
	inline void vector_permute_store_fp16(vec_t * c, unsigned char * vecOffset) {

	static inline void convert_and_scale_q8( vector signed char raw, vector float v_scale, vector unsigned short& out_hi, vector unsigned short& out_lo) {
	static inline void convert_and_scale_q8(vector signed char raw, vector float v_scale, vector unsigned short & out_hi, vector unsigned short & out_lo) {

	void packNormal_q4_fp16(const block_q4_0* a, int64_t lda, int rows, int blocks, unsigned char* vec) {
	void packNormal_q4_fp16(const block_q4_0 * a, int64_t lda, int rows, int blocks, unsigned char * vec) {

	const block_q4_0* current_blk = rows_base[r] + blk;
	const block_q4_0 * current_blk = rows_base[r] + blk;

Uh oh!

Conversation

shalinib-ibm commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shalinib-ibm commented Feb 18, 2026

Uh oh!

taronaeo commented Feb 18, 2026

Uh oh!

shalinib-ibm commented Feb 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

taronaeo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taronaeo commented Feb 19, 2026

Uh oh!

shalinib-ibm commented Feb 19, 2026

Uh oh!

shalinib-ibm commented Feb 19, 2026

Uh oh!

taronaeo commented Feb 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shalinib-ibm commented Feb 18, 2026 •

edited

Loading

shalinib-ibm commented Feb 18, 2026 •

edited

Loading