ggml-cpu: add RVV vec dot kernels for quantization types

[rehan-10xengineer]

Summary

This PR adds RVV vector dot kernels for a number of quantization types.

Key Changes

• Added the following RVV kernels:

Kernel	VLEN
quantize_row_q8_K	128 and above
ggml_vec_dot_mxfp4_q8_0	128, 256
ggml_vec_dot_iq2_xs_q8_K	256
ggml_vec_dot_iq2_xxs_q8_K	128, 256
ggml_vec_dot_iq3_xxs_q8_K	256
ggml_vec_dot_iq4_nl_q8_0	128, 256
ggml_vec_dot_iq4_xs_q8_K	256

Testing

Kernels were functionally tested through test-quantize-fns for 128-bit and 256-bit VLENs.

Benchmarking Results

End-to-end benchmarking on BananaPI-BPI F3 (VLEN=256) with llama-bench.

IQ4_NL

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama IQ4 NL 1.1B	32	2.81	9.72
Tinyllama IQ4 NL 1.1B	64	2.78	9.76
Tinyllama IQ4 NL 1.1B	128	2.77	9.02
Tinyllama IQ4 NL 1.1B	256	2.77	8.99
Tinyllama IQ4 NL 1.1B	512	2.72	8.86

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama IQ4 NL 1.1B	10	2.53	7.94
Tinyllama IQ4 NL 1.1B	16	2.39	7.68
Tinyllama IQ4 NL 1.1B	32	2.40	7.45
Tinyllama IQ4 NL 1.1B	64	2.32	7.36
Tinyllama IQ4 NL 1.1B	100	2.30	7.24

MXFP4

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama MXFP4 1.1B	32	2.73	8.67
Tinyllama MXFP4 1.1B	64	2.88	8.91
Tinyllama MXFP4 1.1B	128	2.73	8.77
Tinyllama MXFP4 1.1B	256	2.66	8.60
Tinyllama MXFP4 1.1B	512	2.69	8.33

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama MXFP4 1.1B	10	2.24	7.58
Tinyllama MXFP4 1.1B	16	2.30	7.66
Tinyllama MXFP4 1.1B	32	2.31	7.57
Tinyllama MXFP4 1.1B	64	2.22	7.06
Tinyllama MXFP4 1.1B	100	2.17	6.53

IQ2_XXS

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama IQ2 XXS 1.1B	32	1.15	8.24
Tinyllama IQ2 XXS 1.1B	64	1.13	8.32
Tinyllama IQ2 XXS 1.1B	128	1.13	8.36
Tinyllama IQ2 XXS 1.1B	256	1.13	8.35
Tinyllama IQ2 XXS 1.1B	512	1.13	7.96

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama IQ2 XXS 1.1B	10	1.13	6.06
Tinyllama IQ2 XXS 1.1B	16	1.00	6.18
Tinyllama IQ2 XXS 1.1B	32	1.00	5.47
Tinyllama IQ2 XXS 1.1B	64	0.99	5.71
Tinyllama IQ2 XXS 1.1B	100	0.98	5.81

IQ4_XS

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama IQ4 XS 1.1B	32	2.55	5.93
Tinyllama IQ4 XS 1.1B	64	2.44	5.99
Tinyllama IQ4 XS 1.1B	128	2.30	5.71
Tinyllama IQ4 XS 1.1B	256	2.32	5.91
Tinyllama IQ4 XS 1.1B	512	2.27	5.82

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama IQ4 XS 1.1B	10	1.74	3.66
Tinyllama IQ4 XS 1.1B	16	1.73	4.08
Tinyllama IQ4 XS 1.1B	32	1.77	4.11
Tinyllama IQ4 XS 1.1B	64	1.74	3.99
Tinyllama IQ4 XS 1.1B	100	1.73	3.91

IQ2_XS

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama IQ2_XS 1.1B	32	1.14	9.51
Tinyllama IQ2_XS 1.1B	64	1.12	9.37
Tinyllama IQ2_XS 1.1B	128	1.13	9.16
Tinyllama IQ2_XS 1.1B	256	1.12	8.89
Tinyllama IQ2_XS 1.1B	512	1.10	8.51

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama IQ2_XS 1.1B	10	0.87	5.59
Tinyllama IQ2_XS 1.1B	16	0.92	5.44
Tinyllama IQ2_XS 1.1B	32	0.92	5.12
Tinyllama IQ2_XS 1.1B	64	0.91	5.25
Tinyllama IQ2_XS 1.1B	100	0.90	5.3

IQ3_XXS

Tokens / Second

Prefill

Model	Prompt Size	Scalar	Vectorized
Tinyllama IQ3_XXS 1.1B	32	1.05	3.21
Tinyllama IQ3_XXS 1.1B	64	1.07	3.27
Tinyllama IQ3_XXS 1.1B	128	1.07	2.98
Tinyllama IQ3_XXS 1.1B	256	1.06	3.00
Tinyllama IQ3_XXS 1.1B	512	1.07	2.73

Decode

Model	Prompt Size = 32	Scalar	Vectorized
Tinyllama IQ3_XXS 1.1B	10	0.88	2.25
Tinyllama IQ3_XXS 1.1B	16	0.88	2.29
Tinyllama IQ3_XXS 1.1B	32	0.87	2.31
Tinyllama IQ3_XXS 1.1B	64	0.87	2.31
Tinyllama IQ3_XXS 1.1B	100	0.87	2.33

Additional Notes

Kernel selection for the right VLEN is handled through a parent function.

void ggml_vec_dot_iq4_nl_q8_0(int n, float * GGML_RESTRICT s, size_t bs, const void * GGML_RESTRICT vx, size_t bx, const void * GGML_RESTRICT vy, size_t by, int nrc) {
#if defined __riscv_v_intrinsic
    switch (__riscv_vlenb() * 8) {
        case 128:
            ggml_vec_dot_iq4_nl_q8_0_vl128(n, s, bs, vx, bx, vy, by, nrc);
            break;
        default:
            ggml_vec_dot_iq4_nl_q8_0_vl256(n, s, bs, vx, bx, vy, by, nrc);
            break;
    }
#else
    ggml_vec_dot_iq4_nl_q8_0_generic(n, s, bs, vx, bx, vy, by, nrc);
#endif
}

Future Work

Subsequent PRs plan to add RVV kernels for several other quantization types, as well as extend existing support to other VLENs.

[taimur-10x]

@ggerganov, @xctan, could this be reviewed please? Thank you.

[rehan-10xengineer]

@ggerganov @xctan can you please review this PR? Thank you

[ggerganov]

Rebase on latest master and lets run the CI

[rehan-10xengineer]

@ggerganov I have rebased on latest master, please approve the CICD workflow to run.