Fix LoRa weight merging in export

[antimatter15]

I was trying to export the model for use with llama.cpp but noticed that it just copied weights from the base model verbatim. I'm suspect there's a "right" way to do this, but this was the approach that worked for me.

[tloen]

Have you tested that the existing code doesn't work? Assuming you're not loading the foundation model in 8bit, the call to .eval() should merge the LoRA weights into the base weights.

[antimatter15]

Yeah I've tested the existing code and it didn't seem to work. Looking at the code for the underlying Peft implementation I don't see why .eval() would merge weights. All the weight merging logic is located inside the .train() method.

…

On Thu, Mar 16, 2023 at 1:04 AM Eric J. Wang ***@***.***> wrote: Have you tested that the existing code doesn't work? Assuming you're not loading the foundation model in 8bit, the call to .eval() should merge the LoRA weights into the base weights. — Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAHKZSF7CV2AF2JIOFQ2W3W4LCSNANCNFSM6AAAAAAV425TNQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[tloen]

eval() should be an alias for train(false), and merge_weights should default to true. I'll look into it tomorrow morning.

[tloen]

Can't hurt