Are the three linear layers--q,k,v , merged in TensorRT-LLM for llama model?( potential bad influence on quantization accuracy)

[brisker]

here:
https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llama/hf_llama_convert.py#L37

https://github.com/NVIDIA/TensorRT-LLM/blob/main/examples/llama/convert.py#L88 (the same output quantization scale for q,k,v)

It seems that the qkv ,three linear layers' quantization scales are merged and shared, but this seems to have potential bad influence on quantization model accuracy.

Will this be fixed in the future?

[byshiue]

From our experiment, it could keep good accuracy even if we merge the scales. Do you encounter any accuracy issue?

[brisker]

@byshiue
yes, the accuracy drops too much. The details can be found here:

#967

[brisker]

@byshiue
It seems that llama2 quantization(not matter smoothquant or kv-cache quantization) has some bugs, which causes consistent bad accuracy ,and qkv scale merged or not may not be the main reason (but still a reason).

[byshiue]

I think we could discuss this issue in #967, do you agree?

[nv-guomingz]

Hi @brisker do u still have further issue or question now? If not, we'll close it soon.