vulkan: fix mmq out of bounds reads

[0cc4m]

Fixes #17056

I also cleaned up some legacy indexing and buffer size code in the matmul functions.

Draft because I caused a mul_mat_id regression, still trying to find the reason.

[mimi89999]

When building I got multiple warnings about unused variables:

[22/99] Building CXX object ggml/src/ggml-vulkan/CMakeFiles/ggml-vulkan.dir/ggml-vulkan.cpp.o
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp: In function ‘void ggml_vk_mul_mat_vec_q_f16(ggml_backend_vk_context*, vk_context&, const ggml_cgraph*, int)’:
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:6588:20: warning: unused variable ‘ne22’ [-Wunused-variable]
 6588 |     const uint64_t ne22 = dst->ne[2];
      |                    ^~~~
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:6589:20: warning: unused variable ‘ne23’ [-Wunused-variable]
 6589 |     const uint64_t ne23 = dst->ne[3];
      |                    ^~~~
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp: In function ‘void ggml_vk_mul_mat_id_q_f16(ggml_backend_vk_context*, vk_context&, const ggml_tensor*, const ggml_tensor*, const ggml_tensor*, ggml_tensor*)’:
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:7171:20: warning: unused variable ‘ne03’ [-Wunused-variable]
 7171 |     const uint64_t ne03 = src0->ne[3];
      |                    ^~~~
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:7186:20: warning: unused variable ‘ne22’ [-Wunused-variable]
 7186 |     const uint64_t ne22 = dst->ne[2];
      |                    ^~~~
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:7187:20: warning: unused variable ‘ne23’ [-Wunused-variable]
 7187 |     const uint64_t ne23 = dst->ne[3];
      |                    ^~~~
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp: In function ‘void ggml_vk_mul_mat_vec_id_q_f16(ggml_backend_vk_context*, vk_context&, const ggml_cgraph*, int)’:
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:7441:20: warning: unused variable ‘ne02’ [-Wunused-variable]
 7441 |     const uint64_t ne02 = src0->ne[2];
      |                    ^~~~
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:7442:20: warning: unused variable ‘ne03’ [-Wunused-variable]
 7442 |     const uint64_t ne03 = src0->ne[3];
      |                    ^~~~
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:7446:20: warning: unused variable ‘ne12’ [-Wunused-variable]
 7446 |     const uint64_t ne12 = src1->ne[2];
      |                    ^~~~
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:7447:20: warning: unused variable ‘ne13’ [-Wunused-variable]
 7447 |     const uint64_t ne13 = src1->ne[3];
      |                    ^~~~
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:7458:20: warning: unused variable ‘ne22’ [-Wunused-variable]
 7458 |     const uint64_t ne22 = dst->ne[2];
      |                    ^~~~
/home/michel/llm/llama.cpp/ggml/src/ggml-vulkan/ggml-vulkan.cpp:7459:20: warning: unused variable ‘ne23’ [-Wunused-variable]
 7459 |     const uint64_t ne23 = dst->ne[3];
      |                    ^~~~
[96/99] Linking CXX executable bin/llama-export-lora

The change also didn't fix #17056 for me.

[mimi89999]

However, this change did fix [MUL_MAT] NMSE = 0.098834385 > 0.000500000 MUL_MAT(type_a=q4_0,type_b=f32,m=576,n=512,k=576,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): FAIL for me: MUL_MAT(type_a=q4_0,type_b=f32,m=576,n=512,k=576,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK

[0cc4m]

Yeah, that's the problem I found in the issue. I added that test. I guess you have another problem. I'll try to find it.

[0cc4m]

I can reproduce another issue with the model you posted in your issue. I'll look into it, thank you for testing the fix.