Merge vulkan code from mainline up to commit of 6/28/2025

[firecoperana]

• Vulkan Optimizations and Fixes (#8959)

• Optimize Vulkan REPEAT performance

.....................................................................................

vulkan: lock accesses of pinned_memory vector (#14333)

vulkan: handle noncontig in the final case of ggml_vk_get_cpy_pipeline (#14378)

Fix cuda build error

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[firecoperana]

Test Qwen 2.5 7B Q4_K_S and it runs fine, but for deepseek model, I was getting "GGGGGGG" output with -mla 1 -amb 512. Probably related to deepseek related optimization.

[ubergarm]

For deepseek often one wants to compile with -DGGML_CUDA_IQK_FORCE_BF16=1 to avoid overflowing fp16 accumulator which manifests as gibberish, nans, or GGG typically I believe.

I just tried to compile but got an error, might be because I just updated my rig and now seem to have gcc version 15.1.1 20250425 (GCC)... I'll fuss with it a bit but put it here in the meantime.

Details inside:

👈 build command and logs

# attempt to build clean
$ rm -rf ./build
$ cmake -B build -DGGML_VULKAN=ON -DGGML_CUDA=OFF -DGGML_RPC=OFF -DGGML_BLAS=OFF -DGGML_CCACHE=OFF
$ cmake --build build --config Release -j $(nproc)

-- The C compiler identification is GNU 15.1.1
-- The CXX compiler identification is GNU 15.1.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: /usr/bin/git (found version "2.50.0")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- OpenMP found
-- Using optimized iqk matrix multiplications
-- Enabling IQK Flash Attention kernels
-- Using llamafile
-- Found Vulkan: /lib/libvulkan.so (found version "1.4.313") found components: glslc glslangValidator
-- Vulkan found
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- x86 detected
-- ARCH_FLAGS = -march=native
-- Configuring done (0.5s)
-- Generating done (0.0s)
-- Build files have been written to: /mnt/astrodata/llm/ik_llama.cpp/build
[  0%] Generating build details from Git
[  0%] Building CXX object ggml/src/vulkan-shaders/CMakeFiles/vulkan-shaders-gen.dir/vulkan-shaders-gen.cpp.o
[  1%] Building C object examples/gguf-hash/CMakeFiles/sha256.dir/deps/sha256/sha256.c.o
[  3%] Building C object examples/gguf-hash/CMakeFiles/xxhash.dir/deps/xxhash/xxhash.c.o
[  3%] Building C object examples/gguf-hash/CMakeFiles/sha1.dir/deps/sha1/sha1.c.o
-- Found Git: /usr/bin/git (found version "2.50.0")
In function ‘SHA1Update’,
    inlined from ‘SHA1Final’ at /mnt/astrodata/llm/ik_llama.cpp/examples/gguf-hash/deps/sha1/sha1.c:265:5:
/mnt/astrodata/llm/ik_llama.cpp/examples/gguf-hash/deps/sha1/sha1.c:219:13: warning: ‘SHA1Transform’ reading 64 bytes from a region of size 0 [-Wstringop-overread]
  219 |             SHA1Transform(context->state, &data[i]);
      |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/mnt/astrodata/llm/ik_llama.cpp/examples/gguf-hash/deps/sha1/sha1.c:219:13: note: referencing argument 2 of type ‘const unsigned char[64]’
/mnt/astrodata/llm/ik_llama.cpp/examples/gguf-hash/deps/sha1/sha1.c: In function ‘SHA1Final’:
/mnt/astrodata/llm/ik_llama.cpp/examples/gguf-hash/deps/sha1/sha1.c:54:6: note: in a call to function ‘SHA1Transform’
   54 | void SHA1Transform(
      |      ^~~~~~~~~~~~~
In function ‘SHA1Update’,
    inlined from ‘SHA1Final’ at /mnt/astrodata/llm/ik_llama.cpp/examples/gguf-hash/deps/sha1/sha1.c:269:9:
/mnt/astrodata/llm/ik_llama.cpp/examples/gguf-hash/deps/sha1/sha1.c:219:13: warning: ‘SHA1Transform’ reading 64 bytes from a region of size 0 [-Wstringop-overread]
  219 |             SHA1Transform(context->state, &data[i]);
      |             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/mnt/astrodata/llm/ik_llama.cpp/examples/gguf-hash/deps/sha1/sha1.c:219:13: note: referencing argument 2 of type ‘const unsigned char[64]’
/mnt/astrodata/llm/ik_llama.cpp/examples/gguf-hash/deps/sha1/sha1.c: In function ‘SHA1Final’:
/mnt/astrodata/llm/ik_llama.cpp/examples/gguf-hash/deps/sha1/sha1.c:54:6: note: in a call to function ‘SHA1Transform’
   54 | void SHA1Transform(
      |      ^~~~~~~~~~~~~
[  3%] Built target sha256
[  3%] Built target sha1
[  3%] Built target xxhash
[  3%] Generating build details from Git
-- Found Git: /usr/bin/git (found version "2.50.0")
[  4%] Building CXX object common/CMakeFiles/build_info.dir/build-info.cpp.o
[  5%] Linking CXX executable ../../../bin/vulkan-shaders-gen
[  5%] Built target build_info
[  5%] Built target vulkan-shaders-gen
[  6%] Generate vulkan shaders
ggml_vulkan: Generating and compiling shaders to SPIR-V
[  6%] Building C object ggml/src/CMakeFiles/ggml.dir/ggml.c.o
[  7%] Building C object ggml/src/CMakeFiles/ggml.dir/ggml-quants.c.o
[  8%] Building CXX object ggml/src/CMakeFiles/ggml.dir/ggml-vulkan.cpp.o
[  8%] Building CXX object ggml/src/CMakeFiles/ggml.dir/ggml-vulkan-shaders.cpp.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml.dir/llamafile/sgemm.cpp.o
[  9%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/iqk_gemm_kquants.cpp.o
[ 10%] Building C object ggml/src/CMakeFiles/ggml.dir/ggml-alloc.c.o
[ 10%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/iqk_mul_mat.cpp.o
[ 11%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/iqk_flash_attn.cpp.o
[ 11%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/fa/iqk_fa_576_512.cpp.o
[ 11%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/iqk_gemm_iquants.cpp.o
[ 11%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/fa/iqk_fa_256_256.cpp.o
[ 12%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/fa/iqk_fa_192_128.cpp.o
[ 12%] Building C object ggml/src/CMakeFiles/ggml.dir/ggml-backend.c.o
[ 14%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/iqk_gemm_ktquants.cpp.o
[ 14%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/fa/iqk_fa_128_128.cpp.o
[ 15%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/fa/iqk_fa_64_64.cpp.o
[ 16%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/iqk_gemm_legacy_quants.cpp.o
[ 16%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/fa/iqk_fa_96_96.cpp.o
[ 17%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/iqk_gemm_floats.cpp.o
[ 17%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/iqk_gemm_1bit.cpp.o
[ 18%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/iqk_gemm_iqk_quants.cpp.o
[ 18%] Building CXX object ggml/src/CMakeFiles/ggml.dir/iqk/iqk_quantize.cpp.o
[ 19%] Building C object ggml/src/CMakeFiles/ggml.dir/ggml-aarch64.c.o
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c: In function ‘ggml_compute_forward’:
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c:19814:5: warning: enumeration value ‘GGML_OP_SIN’ not handled in switch [-Wswitch]
19814 |     switch (tensor->op) {
      |     ^~~~~~
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c:19814:5: warning: enumeration value ‘GGML_OP_COS’ not handled in switch [-Wswitch]
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c:19814:5: warning: enumeration value ‘GGML_OP_COUNT_EQUAL’ not handled in switch [-Wswitch]
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c:19814:5: warning: enumeration value ‘GGML_OP_CONV_2D_DW’ not handled in switch [-Wswitch]
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c:19814:5: warning: enumeration value ‘GGML_OP_RWKV_WKV6’ not handled in switch [-Wswitch]
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c:19814:5: warning: enumeration value ‘GGML_OP_OPT_STEP_ADAMW’ not handled in switch [-Wswitch]
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c: In function ‘ggml_compute_backward’:
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c:20395:5: warning: enumeration value ‘GGML_OP_SIN’ not handled in switch [-Wswitch]
20395 |     switch (tensor->op) {
      |     ^~~~~~
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c:20395:5: warning: enumeration value ‘GGML_OP_COS’ not handled in switch [-Wswitch]
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c:20395:5: warning: enumeration value ‘GGML_OP_COUNT_EQUAL’ not handled in switch [-Wswitch]
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c:20395:5: warning: enumeration value ‘GGML_OP_CONV_2D_DW’ not handled in switch [-Wswitch]
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c:20395:5: warning: enumeration value ‘GGML_OP_RWKV_WKV6’ not handled in switch [-Wswitch]
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml.c:20395:5: warning: enumeration value ‘GGML_OP_OPT_STEP_ADAMW’ not handled in switch [-Wswitch]
In file included from /usr/include/vulkan/vulkan_hpp_macros.hpp:35,
                 from /usr/include/vulkan/vulkan.hpp:11,
                 from /mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-vulkan.cpp:8:
/usr/include/c++/15.1.1/ciso646:46:4: warning: #warning "<ciso646> is deprecated in C++17, use <version> to detect implementation-specific macros" [-Wcpp]
   46 | #  warning "<ciso646> is deprecated in C++17, use <version> to detect implementation-specific macros"
      |    ^~~~~~~
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-vulkan.cpp: In function ‘void ggml_vk_print_gpu_info(size_t)’:
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-vulkan.cpp:3541:18: warning: unused variable ‘subgroup_size’ [-Wunused-variable]
 3541 |     const size_t subgroup_size = (default_subgroup_size != 0) ? default_subgroup_size : subgroup_props.subgroupSize;
      |                  ^~~~~~~~~~~~~
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-vulkan.cpp:3542:16: warning: unused variable ‘uma’ [-Wunused-variable]
 3542 |     const bool uma = props2.properties.deviceType == vk::PhysicalDeviceType::eIntegratedGpu;
      |                ^~~
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-vulkan.cpp: In function ‘void ggml_vk_instance_init()’:
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-vulkan.cpp:3644:12: warning: unused variable ‘num_available_devices’ [-Wunused-variable]
 3644 |     size_t num_available_devices = vk_instance.instance.enumeratePhysicalDevices().size();
      |            ^~~~~~~~~~~~~~~~~~~~~
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-backend.c:269:16: warning: no previous prototype for ‘ggml_backend_tensor_memset’ [-Wmissing-prototypes]
  269 | GGML_CALL void ggml_backend_tensor_memset(struct ggml_tensor* tensor, uint8_t value, size_t offset, size_t size) {
      |                ^~~~~~~~~~~~~~~~~~~~~~~~~~
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-backend.c: In function ‘ggml_backend_multi_buffer_context_interface’:
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-backend.c:1022:34: error: initialization of ‘_Bool (*)(struct ggml_backend_buffer *, const struct ggml_tensor *, struct ggml_tensor *)’ from incompatible pointer type ‘void (*)(struct ggml_backend_buffer *, uint8_t)’ {aka ‘void (*)(struct ggml_backend_buffer *, unsigned char)’} [-Wincompatible-pointer-types]
 1022 |         /* .clear           = */ ggml_backend_multi_buffer_clear,
      |                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-backend.c:1022:34: note: (near initialization for ‘multi_backend_buffer_i.cpy_tensor’)
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-backend.c:1006:23: note: ‘ggml_backend_multi_buffer_clear’ declared here
 1006 | GGML_CALL static void ggml_backend_multi_buffer_clear(ggml_backend_buffer_t buffer, uint8_t value) {
      |                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-backend.c:1024:5: warning: missing initializer for field ‘reset’ of ‘struct ggml_backend_buffer_i’ [-Wmissing-field-initializers]
 1024 |     };
      |     ^
In file included from /mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-backend.c:1:
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-backend-impl.h:50:34: note: ‘reset’ declared here
   50 |         void         (*GGML_CALL reset)      (ggml_backend_buffer_t buffer); // reset any internal state due to tensor initialization, such as tensor extras
      |                                  ^~~~~
make[2]: *** [ggml/src/CMakeFiles/ggml.dir/build.make:222: ggml/src/CMakeFiles/ggml.dir/ggml-backend.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:2044: ggml/src/CMakeFiles/ggml.dir/all] Error 2
make: *** [Makefile:146: all] Error 2

EDIT

fwiw i just forced that explicitly to void * and it compiles but then segfaults towards the end of starting up

#/ggml/src/ggml-backend.c ~around line 1020 or so
-        /* .clear           = */ ggml_backend_multi_buffer_clear,
+        /* .clear           = */ (GGML_CALL void *) ggml_backend_multi_buffer_clear, // ubergarm hack

[ikawrakow]

Please no new ops, new enum values, and no refactoring of the CPU backend. I think the Vulkan back-end can be updated to the latest without using the new back-end formalism in mainline.

[ikawrakow]

Btw, currently working on my M2-Max laptop, and Safari disintegrates into pieces when trying to view the changes in this PR.

[firecoperana]

For deepseek often one wants to compile with -DGGML_CUDA_IQK_FORCE_BF16=1 to avoid overflowing fp16 accumulator which manifests as gibberish, nans, or GGG typically I believe.

I just tried to compile but got an error, might be because I just updated my rig and now seem to have gcc version 15.1.1 20250425 (GCC)... I'll fuss with it a bit but put it here in the meantime.

Details inside:
👈 build command and logs

EDIT

fwiw i just forced that explicitly to void * and it compiles but then segfaults towards the end of starting up

#/ggml/src/ggml-backend.c ~around line 1020 or so
-        /* .clear           = */ ggml_backend_multi_buffer_clear,
+        /* .clear           = */ (GGML_CALL void *) ggml_backend_multi_buffer_clear, // ubergarm hack

Pull again. Fixed it.

[ubergarm]

@firecoperana

Heya thanks again for digging into this! I have two different rigs on which I'm testing. It does now build on the AMD RX 7900 XTX Ubuntu 24.04 box now!

So good news I was able to compile and run firecoperana/Merge_mainline_vulkan@495103bd with vulkan backend! However, only seemed to run without -fa. If I try to use -fa it segfaults after its mostly loaded and right before llama-server would start listening for inputs.

Seems like something is still off as the speeds are off from mainline. Could be I'm using the AMDVLK driver as installed from apt-get install libvulkan-dev 1.4.313.0~rc1-1lunarg24.04-1 or that I'm compiling it wrong? Details in the fold:

👈 sweep-bench comparisons Qwen3-14B-Q4_0 dense no FA

# checkout Merge_mainline_vulkan
$ git rev-parse --short HEAD
495103bd

# build
cmake -B build -DGGML_HIP=OFF -DGGML_HIPBLAS=OFF -DGGML_VULKAN=ON -DGGML_RPC=OFF -DGGML_CCACHE=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j $(nproc)

# test
model=/home/w/projects/models/ubergarm/Qwen3-14B-GGUF/Qwen3-14B-Q4_0.gguf
sudo ./build/bin/llama-sweep-bench \
  --model "$model" \
  -ctk f16 -ctv f16 \
  -c 16896 \
  -ngl 99 \
  --warmup-batch \
  --threads 1

ik_llama.cpp firecoperana/Merge_mainline_vulkan@495103bd FA=0

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
512	128	0	1.363	375.67	3.786	33.81
512	128	512	1.365	375.16	3.817	33.53
512	128	1024	1.414	362.06	3.844	33.30
512	128	1536	1.444	354.69	3.971	32.23
512	128	2048	1.429	358.21	3.965	32.28
512	128	2560	1.447	353.93	4.036	31.71
512	128	3072	1.462	350.17	4.099	31.23
512	128	3584	1.492	343.12	4.137	30.94
512	128	4096	1.499	341.62	4.233	30.24
512	128	4608	1.518	337.27	4.311	29.69
512	128	5120	1.525	335.71	4.355	29.39
512	128	5632	1.567	326.74	4.440	28.83
512	128	6144	1.556	329.11	4.508	28.39
512	128	6656	1.579	324.18	4.534	28.23
512	128	7168	1.596	320.79	4.600	27.83
512	128	7680	1.623	315.45	4.685	27.32
512	128	8192	1.640	312.19	4.775	26.80

llama.cpp@27208bf6 FA=0

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
512	128	0	0.323	1585.78	1.822	70.27
512	128	512	0.334	1533.43	1.859	68.86
512	128	1024	0.369	1386.13	1.907	67.11
512	128	1536	0.382	1338.94	1.956	65.43
512	128	2048	0.374	1369.21	1.995	64.15
512	128	2560	0.391	1308.08	2.081	61.50
512	128	3072	0.396	1293.44	2.148	59.58
512	128	3584	0.422	1214.46	2.202	58.12
512	128	4096	0.422	1214.09	2.278	56.20
512	128	4608	0.435	1176.88	2.344	54.61
512	128	5120	0.441	1159.87	2.407	53.17
512	128	5632	0.482	1061.18	2.472	51.77
512	128	6144	0.465	1100.88	2.549	50.21
512	128	6656	0.483	1060.17	2.602	49.20
512	128	7168	0.494	1037.17	2.661	48.10
512	128	7680	0.523	979.25	2.724	46.99
512	128	8192	0.538	951.01	2.820	45.39

On my local rig with a CUDA and ARCH linux installing extra/vulkan-utility-libraries 1.4.313.0-1 (vulkan-devel) was having a compiling issue still complaining about RPC during linking. It might be because that super new gcc 15.1.1 though given I just updated everything...

$ cmake -B build -DGGML_VULKAN=ON -DGGML_CUDA=OFF -DGGML_RPC=OFF -DGGML_BLAS=OFF -DGGML_CCACHE=ON -DCMAKE_BUILD_TYPE=Debug
$ cmake --build build --config Debug -j $(nproc)

[ 24%] Building CXX object src/CMakeFiles/llama.dir/llama.cpp.o
[ 24%] Building CXX object src/CMakeFiles/llama.dir/unicode.cpp.o
[ 25%] Linking CXX executable ../../bin/llama-gguf
/mnt/astrodata/llm/ik_llama.cpp/src/unicode.cpp: In function ‘std::wstring unicode_wstring_from_utf8(const std::string&)’:
/mnt/astrodata/llm/ik_llama.cpp/src/unicode.cpp:232:10: warning: ‘template<class _Codecvt, class _Elem, class _Wide_alloc, class _Byte_alloc> class std::__cxx11::wstring_convert’ is deprecated [-Wdeprecated-declarations]
  232 |     std::wstring_convert<std::codecvt_utf8<wchar_t>> conv;
      |          ^~~~~~~~~~~~~~~
In file included from /usr/include/c++/15.1.1/locale:47,
                 from /usr/include/c++/15.1.1/regex:43,
                 from /mnt/astrodata/llm/ik_llama.cpp/src/unicode.cpp:12:
/usr/include/c++/15.1.1/bits/locale_conv.h:262:33: note: declared here
  262 |     class _GLIBCXX17_DEPRECATED wstring_convert
      |                                 ^~~~~~~~~~~~~~~
[ 25%] Linking CXX executable ../../bin/llama-gguf-hash
[ 26%] Linking CXX shared library libllama.so
/usr/bin/ld: ../../ggml/src/libggml.so: undefined reference to `ggml_backend_rpc_init'
collect2: error: ld returned 1 exit status
make[2]: *** [examples/gguf/CMakeFiles/llama-gguf.dir/build.make:102: bin/llama-gguf] Error 1
make[1]: *** [CMakeFiles/Makefile2:3314: examples/gguf/CMakeFiles/llama-gguf.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
/usr/bin/ld: ../../ggml/src/libggml.so: undefined reference to `ggml_backend_rpc_init'
collect2: error: ld returned 1 exit status
make[2]: *** [examples/gguf-hash/CMakeFiles/llama-gguf-hash.dir/build.make:108: bin/llama-gguf-hash] Error 1
make[1]: *** [CMakeFiles/Makefile2:3151: examples/gguf-hash/CMakeFiles/llama-gguf-hash.dir/all] Error 2
[ 26%] Built target llama
make: *** [Makefile:146: all] Error 2

However, if I enable the RPC backend with -DGGML_RPC=ON it compiles now! Though starting up it throws some errors and isn't working yet

model=/mnt/astrodata/llm/models/ubergarm/Qwen3-14B-GGUF/Qwen3-14B-Q4_0.gguf

./build/bin/llama-sweep-bench \
  --model "$model" \
  -c 16896 \
  -ngl 99 \
  --warmup-batch \
  --threads 1

llm_load_tensors: ggml ctx size =    0.40 MiB
llm_load_tensors: offloading 40 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 41/41 layers to GPU
llm_load_tensors:    Vulkan0 buffer size =  7697.69 MiB
llm_load_tensors:        CPU buffer size =   417.30 MiB
.........................................................................................
llama_new_context_with_model: n_ctx      = 16896
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: mla_attn   = 0
llama_new_context_with_model: attn_max_b = 0
llama_new_context_with_model: fused_moe  = 0
llama_new_context_with_model: ser        = -1, 0
llama_new_context_with_model: freq_base  = 1000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:    Vulkan0 KV buffer size =  2640.00 MiB
llama_new_context_with_model: KV self size  = 2640.00 MiB, K (f16): 1320.00 MiB, V (f16): 1320.00 MiB
llama_new_context_with_model: Vulkan_Host  output buffer size =     0.58 MiB
ggml_backend_sched_backend_from_buffer: warning: no backend supports op NONE with a weight with buffer type Vulkan0 used in tensor blk.0.attn_norm.weight, the weight will need to be copied
ggml_backend_sched_backend_from_buffer: warning: no backend supports op NONE with a weight with buffer type Vulkan0 used in tensor blk.0.attn_q_norm.weight, the weight will need to be copied
ggml_backend_sched_backend_from_buffer: warning: no backend supports op NONE with a weight with buffer type Vulkan0 used in tensor blk.0.attn_k_norm.weight, the weight will need to be copied
ggml_backend_sched_backend_from_buffer: warning: no backend supports op NONE with a weight with buffer type Vulkan0 used in tensor blk.0.ffn_norm.weight, the weight will need to be copied
ggml_backend_sched_backend_from_buffer: warning: no backend supports op NONE with a weight with buffer type Vulkan0 used in tensor blk.1.attn_norm.weight, the weight will need to be copied

Lemme know if there is a certain version of the vulkan backend that might work better or happy to try more iterations! Thanks!

[firecoperana]

I noticed something odd too and suspect it's related to vulkan shader. When I run llama server in visual studio, I can match the performance of the mainline, but if I run in command line, I was only getting 1/3 to 1/2 of the speed for token generation. If you have time, you can do some troubleshooting, as I'm not familiar with vulkan at all.

"warning: no backend supports op NONE with a weight with buffer type Vulkan0 used in tensor blk.0.attn_norm.weight" happens because vulkan does not support fused rms norm. It only shows in debug version.

[ikawrakow]

"warning: no backend supports op NONE with a weight with buffer type Vulkan0 used in tensor blk.0.attn_norm.weight" happens because vulkan does not support fused rms norm. It only shows in debug version.

We will worry about the missing fused ops after we get the PR in. There is quite a bit left to do to have the ik_llama.cpp advantages available also with Vulkan:

• Implement fused ops
• Implement GEMM/GEMV for all quantization types added in ik_llama.cpp
• Port the ik_llama.cpp improvements related to "indirect" GEMM and GEMV (as needed for MoE models).

[ikawrakow]

Tested on my RTX-4080. If I remove the fused ops (GGML_OP_FUSED_RMS_NORM and GGML_OP_FUSED_MUL_UNARY) and don't use flash attention, I get this for LlaMA-3.1-8B

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
1024	256	0	2.074	493.73	2.602	98.37
1024	256	1024	1.074	953.71	3.198	80.05
1024	256	2048	0.968	1058.33	3.069	83.41
1024	256	3072	0.907	1128.89	3.187	80.32
1024	256	4096	0.941	1088.54	3.368	76.00
1024	256	5120	0.962	1064.06	3.531	72.51
1024	256	6144	0.993	1030.96	3.742	68.42
1024	256	7168	1.037	987.64	3.963	64.60
1024	256	8192	1.098	932.90	4.223	60.62
1024	256	9216	1.156	885.58	4.474	57.22
1024	256	10240	1.216	842.27	4.711	54.34
1024	256	11264	1.271	805.53	4.949	51.73
1024	256	12288	1.323	774.28	5.201	49.22
1024	256	13312	1.381	741.70	5.457	46.92
1024	256	14336	1.440	711.14	5.709	44.84
1024	256	15360	1.469	696.92	5.962	42.94

Flash attention seems to be running on the CPU, so performance drops further with that. TG is on par with mainline for short context, but PP is ~3X lower.

[ikawrakow]

If I change the LOG_DEBUG to LOG_INFO in ggml_vk_print_gpu_info, I see this line:

ggml_vulkan: 0 = NVIDIA GeForce RTX 4080 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 49152 | int dot: 0 | matrix cores: none

On mainline I see this:

ggml_vulkan: 0 = NVIDIA GeForce RTX 4080 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat

So, for some reason int dot products and cooperative matrix are not enabled. I guess, this may explain the lower performance.

[ikawrakow]

Why do we need this check? I don't have coopmat2 available, but if I comment out this check I get FA enabled, and it gives me a nice boost in performance.

[firecoperana]

Removed.

[ikawrakow]

OK, I'm learning. Need to build using

cmake .. -DGGML_VULKAN=ON -DGGML_VULKAN_COOPMAT2_GLSLC_SUPPORT=1 -DGGML_VULKAN_COOPMAT_GLSLC_SUPPORT=1 -DGGML_VULKAN_INTEGER_DOT_GLSLC_SUPPORT=1

Then I need to comment out the check for coopmat2 on line 9476 in ggml-vulkan.cpp to get FA enabled. With that, I almost match the Vulkan performance in mainline:

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
1024	256	0	0.334	3070.20	2.433	105.20
1024	256	1024	0.340	3012.31	2.596	98.60
1024	256	2048	0.342	2995.49	2.751	93.07
1024	256	3072	0.334	3069.60	2.890	88.58
1024	256	4096	0.339	3023.88	3.048	84.00
1024	256	5120	0.352	2909.64	3.240	79.02
1024	256	6144	0.369	2774.90	3.427	74.71
1024	256	7168	0.377	2716.14	3.618	70.76
1024	256	8192	0.388	2636.59	3.793	67.50
1024	256	9216	0.413	2479.99	3.989	64.18
1024	256	10240	0.437	2343.03	4.199	60.96
1024	256	11264	0.460	2225.86	4.408	58.08
1024	256	12288	0.487	2102.61	4.614	55.48
1024	256	13312	0.503	2037.31	4.821	53.10
1024	256	14336	0.535	1915.62	5.036	50.84
1024	256	15360	0.553	1853.00	5.247	48.79

PP is on par with mainline, TG is on par (or even slightly better) for short context, but performance somehow decreases faster with context length, so we end up with ~70% of mainline TG performance at 16k tokens.

I'm told in this comment that I need to update my Nvidia driver to 575, which will give me coopmat2 and almost a factor of 2 speedup.

[firecoperana]

OK, I'm learning. Need to build using

cmake .. -DGGML_VULKAN=ON -DGGML_VULKAN_COOPMAT2_GLSLC_SUPPORT=1 -DGGML_VULKAN_COOPMAT_GLSLC_SUPPORT=1 -DGGML_VULKAN_INTEGER_DOT_GLSLC_SUPPORT=1

Then I need to comment out the check for coopmat2 on line 9476 in ggml-vulkan.cpp to get FA enabled. With that, I almost match the Vulkan performance in mainline:
PP TG N_KV T_PP s S_PP t/s T_TG s S_TG t/s
1024 256 0 0.334 3070.20 2.433 105.20
1024 256 1024 0.340 3012.31 2.596 98.60
1024 256 2048 0.342 2995.49 2.751 93.07
1024 256 3072 0.334 3069.60 2.890 88.58
1024 256 4096 0.339 3023.88 3.048 84.00
1024 256 5120 0.352 2909.64 3.240 79.02
1024 256 6144 0.369 2774.90 3.427 74.71
1024 256 7168 0.377 2716.14 3.618 70.76
1024 256 8192 0.388 2636.59 3.793 67.50
1024 256 9216 0.413 2479.99 3.989 64.18
1024 256 10240 0.437 2343.03 4.199 60.96
1024 256 11264 0.460 2225.86 4.408 58.08
1024 256 12288 0.487 2102.61 4.614 55.48
1024 256 13312 0.503 2037.31 4.821 53.10
1024 256 14336 0.535 1915.62 5.036 50.84
1024 256 15360 0.553 1853.00 5.247 48.79

PP is on par with mainline, TG is on par (or even slightly better) for short context, but performance somehow decreases faster with context length, so we end up with ~70% of mainline TG performance at 16k tokens.

I'm told in this comment that I need to update my Nvidia driver to 575, which will give me coopmat2 and almost a factor of 2 speedup.

The new commit should remove the need to add these in cmake command. Also disable the fused ops for now.

[ubergarm]

The new commit should remove the need to add these in cmake command. Also disable the fused ops for now.

Thanks I was having trouble getting it setup before, the recent commit fixed it right up. First the amazing news, check this out on the AMD RX 7900 XTX it is up to snuff in early testing:

Very nice! I want to try some more models tomorrow but this is getting exciting!

I also got it to build and detect things properly on my local ARCH linux NVIDIA 3090TI FE rig, however when it starts up it throws an error:

ggml_vulkan: 0 = NVIDIA GeForce RTX 3090 Ti (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
/mnt/astrodata/llm/ik_llama.cpp/ggml/src/ggml-vulkan.cpp:2031: GGML_ASSERT((GGML_KQ_MASK_PAD % rows_cols[0]) == 0) failed

Amazing progress in a short time!

I tried a couple small R1-0528 quants but not quite there yet:

👈 AMD 7900 XTX DeepSeek-R1-0528 Log

cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_HIPBLAS=0 -DGGML_VULKAN=1 -DGGML_RPC=0 -DGGML_CCACHE=1 -DGGML_SCHED_MAX_COPIES=1 -DGGML_CUDA_IQK_FORCE_BF16=1
cmake --build build --config Release -j $(nproc)

model=/home/w/projects/models/ubergarm/DeepSeek-R1-0528-GGUF/DeepSeek-R1-0528-IQ1_S.gguf
#model=/home/w/projects/models/ubergarm/DeepSeek-R1-0528-GGUF/DeepSeek-R1-0528-IQ1_S_R4-00001-of-00003.gguf
./build/bin/llama-sweep-bench \
  --model "$model" \
  --ctx-size 4608 \
  -fa \
  -mla 3 -amb 512 \
  -ctk q8_0 \
  -ngl 99 \
  -ot "blk\.(3|4|5|6|7|8)\.ffn_.*=Vulkan0" \
  -ot exps=CPU \
  --threads 16 \
  --no-mmap
  # -fmoe # leave this off for now

ggml_vulkan: 0 = Radeon RX 7900 XTX (AMD open-source driver) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat

llama_new_context_with_model: n_ctx      = 4608
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 1
llama_new_context_with_model: mla_attn   = 3
llama_new_context_with_model: attn_max_b = 512
llama_new_context_with_model: fused_moe  = 0
llama_new_context_with_model: ser        = -1, 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 0.025
llama_kv_cache_init:    Vulkan0 KV buffer size =   164.06 MiB
llama_new_context_with_model: KV self size  =  164.06 MiB, c^KV (q8_0):  164.06 MiB, kv^T: not used
llama_new_context_with_model: Vulkan_Host  output buffer size =     0.49 MiB
llama_new_context_with_model:    Vulkan0 compute buffer size =   982.00 MiB
llama_new_context_with_model: Vulkan_Host compute buffer size =   480.91 MiB
llama_new_context_with_model: graph nodes  = 4641
llama_new_context_with_model: graph splits = 826

main: n_kv_max = 4608, n_batch = 2048, n_ubatch = 512, flash_attn = 1, n_gpu_layers = 99, n_threads = 16, n_threads_batch = 16

|    PP |     TG |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |
|-------|--------|--------|----------|----------|----------|----------|
ggml_compute_forward_dup_q: CPU#cache_k_l0 (view)#0 -> cache_k_l0 (view) (copy) is of type f16
ggml_compute_forward_dup_q: CPU#cache_k_l0 (view)#0 -> cache_k_l0 (view) (copy) is of type f16
/home/w/projects/ik_llama.cpp/ggml/src/ggml.c:10783: /home/w/projects/ik_llama.cpp/ggml/src/ggml.c:10783: fatal errorggml_compute_forward_dup_q: CPU#c
ache_k_l0 (view)#0 -> cache_k_l0 (view) (copy) is of type f16
fatal error

[ikawrakow]

I don't quite understand why ik_llama.cpp would run faster than mainline. None of the additions that make it run faster on CPU/CUDA are implemented in the Vulkan port.

I tried a couple small R1-0528 quants but not quite there yet:

Of course not. The Vulkan backend does not support DeepSeek flash attention, so no, no -mla 3 is possible. -fmoe is not there either. Neither are all the additions to concatenating, copying, and transposing tensors necessary to make FlashMLA-3 work.