MultiheadAttention out_projection

[ghost]

Hello,

thanks for this implementation - very useful.

I had a question regarding MultiheadAttention class - it seems like out_proj.weight is not updated or I am missing something?

Thanks!

[Baijiong-Lin]

Yep, I also find this question. It is because the out_proj in MultiheadAttention is NonDynamicallyQuantizableLinear rather than a simple Linear layer.

https://github.com/pytorch/pytorch/blob/dbb96ef30da4e50bdbecb56dfb9b2c43b8a39e9d/torch/nn/modules/activation.py#L1008

[ghost]

Yep, so o_lora_A and o_lora_B are not currently used in the package?
I think NonDynamicallyQuantizableLinear can be replaced with a linear layer with some warning being thrown if quantization is used...

[Baijiong-Lin]

Sounds like a good idea. I will try to fix it (maybe after two weeks, i am busy with some ddls currently).

[marcomistretta]

did you manage to fix it?

[marcomistretta]

P.S. thanks for this implementation!

[mounchiliu]

If I set enable_lora: list = ['q', 'k', 'v','o'], the problem mentioned in #7 still exists. This may be due to the need to pass the with_nn parameter during the recursive calls.
if module_name == name: return set_param(mod, rest, param, mode=mode, with_nn=with_nn)

[Baijiong-Lin]

@mounchiliu Thanks for your suggestion. I have fixed this problem.

[Baijiong-Lin]

@marcomistretta @ghost Sorry for the late reply. The LoRA of out_proj is not updated because of the wrong init of the LoRA rather than the use of NonDynamicallyQuantizableLinear. I have fixed this problem.