Will the new K quantizations be added anytime soon for StarCoder?

[richardr1126]

I just quantized to q4_0 with WizardCoder-15b, using the Starcoder examples folder in the repo.

I really want to be able to use 3-bit k quant (Q3_K_M) with my model. Will this ever be possible? Or is it not possible with the starcoder models?

[saharNooby]

+1 for the question! k_quants.h and k_quants.c are missing in the main ggml repo. Is there a reason why this repo does not support k-quants yet?

(BTW, I've tried to copy-paste the k-quants files in rwkv.cpp, but the inference did not work because of assertion failure: [ggml_is_transposed](ggml.c:6161: !ggml_is_transposed(a)); I did not dig deeper)

[Green-Sky]

FYI, starcoder can now be used with llama.cpp ggml-org/llama.cpp#3187
i did not check if it supports k-quants

[richardr1126]

FYI, starcoder can now be used with llama.cpp ggerganov/llama.cpp#3187 i did not check if it supports k-quants

confirmed Starcoder k-quants do work on llama.cpp