Bug: `llama-export-lora` fails merging a T5 model with its LoRA adapter

[cyanic-selkie]

What happened?

I ran:

./llama-export-lora -m flan-t5-small-q8_0.gguf --lora adapter_model-q8_0.gguf -o merged.gguf

Which failed with the following error:

file_input: loaded gguf from adapter_model-q8_0.gguf
merge_tensor : dec.blk.0.attn_k.weight [512, 384, 1, 1]
merge_tensor :   + dequantize base tensor from q8_0 to F32
merge_tensor :   + merging from adapter[0] type=q8_0
merge_tensor :     input_scale=1.000000 calculated_scale=1.000000 rank=32
merge_tensor :   + output type is f16
ggml/src/ggml.c:9109: fatal error
ggml/src/ggml.c:9109: fatal errorggml/src/ggml.c:9109: fatal error
ggml/src/ggml.c:9109: fatal error
ggml/src/ggml.c:9109: fatal error

ggml/src/ggml.c:9109: fatal error
./llama-export-lora(+0x4fcdb)[0x63917799ccdb]
./llama-export-lora(+0x517b7)[0x63917799e7b7]
./llama-export-lora(+0x4fcdb)[0x63917799ccdb]
./llama-export-lora(+0x61ea6)[0x6391779aeea6]
./llama-export-lora(+0x517b7)[0x63917799e7b7]
./llama-export-lora(+0x82fab)[0x6391779cffab]
./llama-export-lora(+0x61ea6)[0x6391779aeea6]
./llama-export-lora(+0x83199)[0x6391779d0199]
./llama-export-lora(+0x82fab)[0x6391779cffab]
/lib/x86_64-linux-gnu/libgomp.so.1(+0x1dc0e)[0x71844dc7cc0e]
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x71844b094ac3]
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x71844b126850]
./llama-export-lora(+0x83199)[0x6391779d0199]
/lib/x86_64-linux-gnu/libgomp.so.1(+0x1dc0e)[0x71844dc7cc0e]
Aborted (core dumped)

The base model can be found here: https://huggingface.co/repetitio/flan-t5-small.
The adapter can be found here: https://huggingface.co/repetitio/distilled-simplifier.

Name and Version

version: 3560 (6afd1a9)
built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

What operating system are you seeing the problem on?

Linux

Relevant log output

No response

[fairydreaming]

@cyanic-selkie I think it may work if you run the llama-export-lora command on the unquantized model and lora adapter files and then quantize the resulting merged model file.

[fairydreaming]

From the debugging the exact cause of failure is GGML_OP_CPY failing during ggml_compute_forward_dup() where dst->src[0] tensor is GGML_TYPE_Q8_0 while dst tensor is GGML_TYPE_F32. This GGML_OP_CPY is a result of ggml_cast(ctx0, inp_a[i], GGML_TYPE_F32).

[ngxson]

quantized lora is not supported for now (only quantized model is supported, but not the lora adapter). we may need to add a check to prevent such case.

[fairydreaming]

@ngxson or at least mention this in llama-export-lora README.md, as it currently contains example with quantized lora adapter that from what you say is not supported:

./bin/llama-export-lora \
    -m open-llama-3b-v2-q8_0.gguf \
    -o open-llama-3b-v2-q8_0-english2tokipona-chat.gguf \
    --lora lora-open-llama-3b-v2-q8_0-english2tokipona-chat-LATEST.gguf

[cyanic-selkie]

@ngxson Just curious, what would be involved in supporting quantized adapter and base model?