LoRA integration with llama-export-lora.exe results in blank/gibberish output

[AkiMatsushita]

I'm experiencing an issue when integrating a LoRA model with a base model using llama-export-lora.exe. After merging the LoRA into the base model's gguf file, the resulting model either produces blank output or gibberish (e.g., "---------") regardless of the prompt used.

Steps to reproduce:

1. I trained a LoRA model using the peft library with the following settings:
   • Base model: tokyotech-llm/Llama-3.1-Swallow-8B-v0.1
   • r: 4
   • lora_alpha: 16
   • lora_dropout: 0.1
2. I converted the LoRA to gguf format using convert_lora_to_gguf.py with the --outtype f16 option.
   Command: python convert_lora_to_gguf.py --base ./base_model --outfile ./lora_model.gguf --outtype f16 ./lora_model
3. I merged the LoRA into the base model's gguf file using llama-export-lora.exe.
   Command: llama-export-lora.exe -m base_model.gguf --lora lora_model.gguf -o merged_model.gguf
4. I tried to run the merged model using llama.cpp.
   Command: ./main -m merged_model.gguf -p "Test prompt"

Expected behavior:

The merged model should generate text based on the given prompt, incorporating the knowledge learned by the LoRA.

Actual behavior:

The merged model produces either blank output or gibberish (e.g., "---------"). This happens even with prompts unrelated to the training data.

Environment:

• OS: Windows 11
• llama.cpp version: commit hash 9abe9ee
• Compiler: w64devkit 1.16.0
• Base model: tokyotech-llm/Llama-3.1-Swallow-8B-v0.1
• LoRA settings: r=4, lora_alpha=16, lora_dropout=0.1
• llama-export-lora.exe version: 4145

Additional context:

• The original LoRA model works correctly when loaded with transformers and peft.

Thank you for your time and assistance.