Skip to content

Fixed cooperative matrix shaders on Vulkan builds when CROSS_COMPILE ON #2914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sandrohanea wants to merge 6 commits into from
Closed

Fixed cooperative matrix shaders on Vulkan builds when CROSS_COMPILE ON #2914

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
04ab3d3
Fixed COOPMAT on Vulkan builds with CROSS_COMPILE ON
Mar 20, 2025
e6ab39f
Fixed a warning to retrigger CI (failed transiently last time)
Mar 21, 2025
62e5be8
Simplified the condition to just pass the DGGML_VULKAN_COOPMAT_GLSLC_…
Mar 26, 2025
eaa2808
Add status messages for GL_NV_cooperative_matrix2 support in Vulkan
Mar 26, 2025
3275375
Enhance Vulkan CMake configuration to allow explicit glslc executable…
Mar 26, 2025
3519d2d
WIP Refactor Vulkan CMake configuration to improve glslc executable h…
Mar 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 98 additions & 23 deletions ggml/src/ggml-vulkan/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,15 @@
cmake_minimum_required(VERSION 3.19)
cmake_policy(SET CMP0114 NEW)

find_package(Vulkan COMPONENTS glslc REQUIRED)
# First find glslc on the host system (not target) to ensure we have control over which one is used
find_program(Vulkan_GLSLC_EXECUTABLE glslc NO_CMAKE_FIND_ROOT_PATH)
if(NOT Vulkan_GLSLC_EXECUTABLE)
message(WARNING "glslc not found on host system. Will attempt to find via Vulkan package")
endif()

# Find Vulkan package with required components
find_package(Vulkan REQUIRED)
message(STATUS "Using glslc: ${Vulkan_GLSLC_EXECUTABLE}")

function(detect_host_compiler)
if (CMAKE_HOST_SYSTEM_NAME STREQUAL "Windows")
Expand All @@ -22,33 +30,97 @@ if (Vulkan_FOUND)
ggml-vulkan.cpp
../../include/ggml-vulkan.h
)
if (CMAKE_CROSSCOMPILING)
# Check if user has explicitly set these options
Copy link
Contributor

@mtmcp mtmcp Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure what is the need for these Android checks? Perhaps I confused matters mentioning Android, but I believe it was working correctly with the feature detection—unless I am missing something. Feature detection by attempting to compile the shaders should be able to work on all systems.

if(NOT DEFINED GGML_VULKAN_COOPMAT_GLSLC_SUPPORT OR NOT DEFINED GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)
# Try to detect target capabilities based on architecture/platform
message(STATUS "Cross-compiling: attempting to auto-detect target shader capabilities")

# Set defaults based on known platforms
if(ANDROID)
# Most modern Android devices support cooperative matrices
message(STATUS "Android target detected: enabling COOPMAT support by default")
set(DETECTED_COOPMAT_SUPPORT ON)

# Check Android API level or NDK version for COOPMAT2 support
# Android NDK r25+ and modern devices generally support this
if(ANDROID_PLATFORM_LEVEL GREATER_EQUAL 29 OR (DEFINED ANDROID_NDK_REVISION AND ANDROID_NDK_REVISION GREATER_EQUAL 25))
message(STATUS "Modern Android target detected: enabling COOPMAT2 support by default")
set(DETECTED_COOPMAT2_SUPPORT ON)
else()
set(DETECTED_COOPMAT2_SUPPORT OFF)
endif()
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "aarch64|ARM64")
# ARM64 generally supports cooperative matrices on newer GPUs
message(STATUS "ARM64 target detected: enabling COOPMAT support by default")
set(DETECTED_COOPMAT_SUPPORT ON)
set(DETECTED_COOPMAT2_SUPPORT OFF) # Be conservative with COOPMAT2
else()
# For other platforms, default to OFF to be safe
set(DETECTED_COOPMAT_SUPPORT OFF)
set(DETECTED_COOPMAT2_SUPPORT OFF)
endif()

# Set the options based on detection if not already set
if(NOT DEFINED GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)
option(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT "Enable coopmat shaders" ${DETECTED_COOPMAT_SUPPORT})
endif()

if(NOT DEFINED GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)
option(GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT "Enable coopmat2 shaders" ${DETECTED_COOPMAT2_SUPPORT})
endif()
else()
# Options already set, use those values
option(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT "Enable coopmat shaders" ${GGML_VULKAN_COOPMAT_GLSLC_SUPPORT})
option(GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT "Enable coopmat2 shaders" ${GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT})
endif()

message(STATUS "Cross-compile COOPMAT support: ${GGML_VULKAN_COOPMAT_GLSLC_SUPPORT}")
message(STATUS "Cross-compile COOPMAT2 support: ${GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT}")
else()
# Compile a test shader to determine whether GL_KHR_cooperative_matrix is supported.
# If it's not, there will be an error to stderr.
# If it's supported, set a define to indicate that we should compile those shaders
execute_process(COMMAND ${Vulkan_GLSLC_EXECUTABLE} -o - -fshader-stage=compute --target-env=vulkan1.3 "${CMAKE_CURRENT_SOURCE_DIR}/vulkan-shaders/test_coopmat_support.comp"
OUTPUT_VARIABLE glslc_output
ERROR_VARIABLE glslc_error)

if (${glslc_error} MATCHES ".*extension not supported: GL_KHR_cooperative_matrix.*")
message(STATUS "GL_KHR_cooperative_matrix not supported by glslc")
set(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT OFF CACHE INTERNAL "Not enable coopmat shaders")
else()
message(STATUS "GL_KHR_cooperative_matrix supported by glslc")
set(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT ON CACHE INTERNAL "Enable coopmat shaders")
endif()

# Compile a test shader to determine whether GL_KHR_cooperative_matrix is supported.
# If it's not, there will be an error to stderr.
# If it's supported, set a define to indicate that we should compile those shaders
execute_process(COMMAND ${Vulkan_GLSLC_EXECUTABLE} -o - -fshader-stage=compute --target-env=vulkan1.3 "${CMAKE_CURRENT_SOURCE_DIR}/vulkan-shaders/test_coopmat_support.comp"
OUTPUT_VARIABLE glslc_output
ERROR_VARIABLE glslc_error)
# Compile a test shader to determine whether GL_NV_cooperative_matrix2 is supported.
# If it's not, there will be an error to stderr.
# If it's supported, set a define to indicate that we should compile those shaders
execute_process(COMMAND ${Vulkan_GLSLC_EXECUTABLE} -o - -fshader-stage=compute --target-env=vulkan1.3 "${CMAKE_CURRENT_SOURCE_DIR}/vulkan-shaders/test_coopmat2_support.comp"
OUTPUT_VARIABLE glslc_output
ERROR_VARIABLE glslc_error)

if (${glslc_error} MATCHES ".*extension not supported: GL_KHR_cooperative_matrix.*")
message(STATUS "GL_KHR_cooperative_matrix not supported by glslc")
else()
message(STATUS "GL_KHR_cooperative_matrix supported by glslc")
add_compile_definitions(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)
if (${glslc_error} MATCHES ".*extension not supported: GL_NV_cooperative_matrix2.*")
message(STATUS "GL_NV_cooperative_matrix2 not supported by glslc")
set(GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT OFF CACHE INTERNAL "Not enable coopmat2 shaders")
else()
message(STATUS "GL_NV_cooperative_matrix2 supported by glslc")
set(GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT ON CACHE INTERNAL "Enable coopmat2 shaders")
endif()
endif()

# Compile a test shader to determine whether GL_NV_cooperative_matrix2 is supported.
# If it's not, there will be an error to stderr.
# If it's supported, set a define to indicate that we should compile those shaders
execute_process(COMMAND ${Vulkan_GLSLC_EXECUTABLE} -o - -fshader-stage=compute --target-env=vulkan1.3 "${CMAKE_CURRENT_SOURCE_DIR}/vulkan-shaders/test_coopmat2_support.comp"
OUTPUT_VARIABLE glslc_output
ERROR_VARIABLE glslc_error)

if (${glslc_error} MATCHES ".*extension not supported: GL_NV_cooperative_matrix2.*")
message(STATUS "GL_NV_cooperative_matrix2 not supported by glslc")
if (GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)
message(STATUS "GGML_VULKAN_COOPMAT_GLSLC_SUPPORT enabled")
add_compile_definitions(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)
else()
message(STATUS "GL_NV_cooperative_matrix2 supported by glslc")
message(STATUS "GGML_VULKAN_COOPMAT_GLSLC_SUPPORT disabled")
endif()

if (GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)
message(STATUS "GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT enabled")
add_compile_definitions(GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)
else()
message(STATUS "GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT disabled")
endif()

target_link_libraries(ggml-vulkan PRIVATE Vulkan::Vulkan)
Expand Down Expand Up @@ -119,6 +191,9 @@ if (Vulkan_FOUND)
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/vulkan-shaders
CMAKE_ARGS -DCMAKE_TOOLCHAIN_FILE=${HOST_CMAKE_TOOLCHAIN_FILE}
-DCMAKE_INSTALL_PREFIX=${CMAKE_BINARY_DIR}
-DGGML_VULKAN_COOPMAT_GLSLC_SUPPORT=${GGML_VULKAN_COOPMAT_GLSLC_SUPPORT}
-DGGML_VULKAN_COOPMAT2_GLSLC_SUPPORT=${GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT}
-DGGML_VULKAN_GLSLC_EXECUTABLE=${Vulkan_GLSLC_EXECUTABLE}
BUILD_COMMAND ${CMAKE_COMMAND} --build .
INSTALL_COMMAND ${CMAKE_COMMAND} --install .
INSTALL_DIR ${CMAKE_BINARY_DIR}
Expand Down Expand Up @@ -159,4 +234,4 @@ if (Vulkan_FOUND)

else()
message(WARNING "Vulkan not found")
endif()
endif()
2 changes: 1 addition & 1 deletion ggml/src/ggml-vulkan/ggml-vulkan.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1394,7 +1394,7 @@ static std::array<uint32_t, 2> fa_rows_cols(uint32_t D, uint32_t clamp, ggml_typ
return {64, 32};
}
return {64, 64};
};
}

static bool ggml_vk_matmul_shmem_support(const vk_device& device, const std::vector<uint32_t>& warptile, bool mul_mat_id, ggml_type src0_type) {

Expand Down
31 changes: 28 additions & 3 deletions ggml/src/ggml-vulkan/vulkan-shaders/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,36 @@
find_package (Threads REQUIRED)
find_program(GLSLC_EXECUTABLE glslc)
if(NOT GLSLC_EXECUTABLE)
message(FATAL_ERROR "glslc not found.")

# Use glslc from the parent CMake context or find it on the host system
if(DEFINED Vulkan_GLSLC_EXECUTABLE)
set(GLSLC_EXECUTABLE ${Vulkan_GLSLC_EXECUTABLE})
message(STATUS "Using glslc from parent CMake: ${GLSLC_EXECUTABLE}")
# Also define this at compile time to set the default value in the shader generator
add_compile_definitions(VULKAN_GLSLC_EXECUTABLE="${GLSLC_EXECUTABLE}")
Copy link
Contributor

@mtmcp mtmcp Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, I don't think we need this functionality at all. The vulkan-shaders-gen binary has a command-line switch which receives a path to the specified glslc. There is really no need to provide a statically compiled path to glslc, since it is already being passed in at runtime.

else()
# Find glslc on the host system, not the target
find_program(GLSLC_EXECUTABLE glslc NO_CMAKE_FIND_ROOT_PATH)
if(NOT GLSLC_EXECUTABLE)
message(FATAL_ERROR "glslc not found. Please set Vulkan_GLSLC_EXECUTABLE in parent CMake.")
endif()
message(STATUS "Found host glslc: ${GLSLC_EXECUTABLE}")
endif()

option(GGML_VULKAN_COOPMAT_GLSLC_SUPPORT "Enable coopmat shaders" OFF)
option(GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT "Enable coopmat2 shaders" OFF)

message(STATUS "GGML_VULKAN_COOPMAT_GLSLC_SUPPORT: ${GGML_VULKAN_COOPMAT_GLSLC_SUPPORT}")
message(STATUS "GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT: ${GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT}")

set(TARGET vulkan-shaders-gen)
add_executable(${TARGET} vulkan-shaders-gen.cpp)
if (GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)
Copy link
Contributor

@jeffbolznv jeffbolznv Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In ggml-org/llama.cpp#11695 (comment) I had guessed that we may need to pass through a cmake variable telling this makefile which version of glslc to use? Is that not necessary? It's just the #defines that weren't making it through?

Copy link
Contributor Author

@sandrohanea sandrohanea Mar 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not 100% sure if we need additional version to be used (and in my case it was working as same version as the host system was used). I can confirm that only providing the compile definitions is working for my build => https://github.com/sandrohanea/whisper.net/actions/runs/13977409227/job/39134498314

target_compile_definitions(vulkan-shaders-gen PRIVATE GGML_VULKAN_COOPMAT_GLSLC_SUPPORT)
endif()

if (GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)
target_compile_definitions(vulkan-shaders-gen PRIVATE GGML_VULKAN_COOPMAT2_GLSLC_SUPPORT)
endif()

install(TARGETS ${TARGET} RUNTIME)
target_compile_features(${TARGET} PRIVATE cxx_std_17)
target_link_libraries(vulkan-shaders-gen PUBLIC Threads::Threads)
12 changes: 11 additions & 1 deletion ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@



#include <iostream>
#include <fstream>
#include <sstream>
Expand Down Expand Up @@ -35,7 +36,16 @@
std::mutex lock;
std::vector<std::pair<std::string, std::string>> shader_fnames;

std::string GLSLC = "glslc";
// Default glslc path, can be overridden at compile time or runtime
#ifdef VULKAN_GLSLC_EXECUTABLE
// If VULKAN_GLSLC_EXECUTABLE is defined at compile time, use it
#define DEFAULT_GLSLC_EXECUTABLE VULKAN_GLSLC_EXECUTABLE
Copy link
Contributor

@mtmcp mtmcp Mar 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See note above. I don't think there's any benefit to bake an executable path into the binary when it is already supplied as a command-line switch to vulkan-shaders-gen.

#else
// Otherwise default to "glslc"
#define DEFAULT_GLSLC_EXECUTABLE "glslc"
#endif

std::string GLSLC = DEFAULT_GLSLC_EXECUTABLE;
std::string input_dir = "vulkan-shaders";
std::string output_dir = "/tmp";
std::string target_hpp = "ggml-vulkan-shaders.hpp";
Expand Down
Loading