Add tensor core GQA dispatch for [4,5,6,8]

[lzhangzz]

Benchmarks results of various CTA_H

A100 80G, batch size 128, seq len 1024, w/o split-kv, in microseconds

32:8	1	2	4
SIMT	513.70	316.26	319.70
TC	509.66	318.91	308.35

48:8	1	2	3	6
SIMT	755.17	382.27	354.37	367.46*
TC	756.35	401.28	319.58	309.31

64:8	1	2	4	8
SIMT	1020	497.92	406.27	985.82*
TC	993.98	517.86	319.39	305.12

80:8	1	2	5
SIMT	1260	616	485.70
TC	1230	635.52	318.75

^*register spill

Conclusion: use TC when head_num / kv_head_num > 2

[zhyncs]

Awesome! May you write an advanced guide for gemm tuning at https://github.com/InternLM/lmdeploy?tab=readme-ov-file#tutorials

[jjjjohnson]

what does TC stands for?

[zhyncs]

what does TC stands for?

tensor core