Add python script that converts GGUF imatrix files to the format supported here

[saood06]

As mentioned here the new GGUF imatrix files can now be found on a lot of recent model releases on huggingface. This mostly vibe coded script allows for conversion without needing to have a compiled version of mainline.

By default swaps the .gguf extension for .dat but you can use --outfile to specify an output file.

Tested by converting and using the resulting imatrix to quantize but didn't do any objective tests (like perplexity).

[ubergarm]

Thanks @saood06 its good to have this ability here in addition to the mainline llama-imatrix --output-format dat --in-file imatrix.gguf --output-file imatrix.dat conversion method.

I haven't tried it yet, but am curious in general if imatrix files computed against the new mainline pre-merged ffn_(gate|up)_exps bf16s will apply to non pre-merged bf16 ggufs...

I'm guessing not unfortunately...

[ikawrakow]

I'm guessing not unfortunately...

The imatrix for ffn_up and ffn_gate tensors is exactly the same as these two tensors "see" the exact same activations. Hence, it would be a matter of a quick hack to make it work.

Having said that, I don't think such nonsense should be supported.

Oh, is somebody going to tell AesSedai that his new GGUFs do not work here?

[saood06]

Thanks @saood06 its good to have this ability here

Well like I said the script was mostly vibe coded. The reason the PR was delayed was the original working script was too ugly. The current script was made with newer models.

I haven't tried it yet, but am curious in general if imatrix files computed against the new mainline pre-merged ffn_(gate|up)_exps bf16s will apply to non pre-merged bf16 ggufs...

I'm guessing not unfortunately...

Your guess is correct. The script does only basic conversion, nothing fancy.