Update LLVM

[rjodinchr]

libclc has to be built through the runtimes feature of LLVM. It needs the native target to build.
Disabling zlib as build is failing on libclc when enabled.

Add patches to llvm while we wait for them to get merged

Fix #1519

[dneto0]

Link error on Windows. Some warnings, then an error linking ErrorHandler.obj

T:\src\github\clspv\build\third_party\llvm\Release\bin\opt.exe : warning : failed to create target machine for 'spir-unknown-unknown': unable to get target for 'spir-unknown-unknown', see --version and --triple. [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\builtins.opt.clspv--.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
    Building Custom Rule T:/src/github/clspv/third_party/llvm/libclc/CMakeLists.txt
    Building Custom Rule T:/src/github/clspv/third_party/llvm/libclc/utils/CMakeLists.txt
cl : command line warning D9002: ignoring unknown option '-fno-rtti' [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
cl : command line warning D9002: ignoring unknown option '-fno-exceptions' [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
    prepare-builtins.cpp
LINK : warning LNK4044: unrecognized option '/lpsapi'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/lshell32'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/lole32'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/luuid'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/ladvapi32'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/lws2_32'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/lntdll'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/ldelayimp'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LLVMSupport.lib(ErrorHandling.obj) : error LNK2019: unresolved external symbol __imp_RtlGetLastNtStatus referenced in function "class std::error_code __cdecl llvm::mapLastWindowsError(void)" (?mapLastWindowsError@llvm@@YA?AVerror_code@std@@XZ) [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
shell32.lib(SHELL32.dll) : error LNK2001: unresolved external symbol __delayLoadHelper2 [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
ole32.lib(ole32.dll) : error LNK2001: unresolved external symbol __delayLoadHelper2 [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
T:\src\github\clspv\build\third_party\llvm\Release\bin\prepare_builtins.exe : fatal error LNK1120: 2 unresolved externals [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
    Generating obj.libclc.dir/clspv64--/clc-convert.cl.bc

[rjodinchr]

@frasercrmck have you ever seen something like that compiling libclc as a runtime on windows?

[rjodinchr]

It seems that we have unresolved symbols because some link options have been ignored.

From what I understand those options are using a unix-like syntax that is supported by clang, but not by MSVC.

Most probably, libclc cmake is using it because most of its targets are built with clang to generate bc files. But prepare-builtins is a host tool, thus it tries to compile it with MSVC, but it is using the wrong syntax for link arguments.

[dneto0]

Discussed the Windows build situation with @rjodinchr:

• The libclc build has been moved to be a "runtime" in the LLVM build scheme.
• In the build, clspv, clang and other LLVM host tools are built first, then libclc is built.
• The libclc build must use the host tools that it just built.
• But in the Windows/cmake setup, the libclc build is using the CMake external build mechanism.
• The external build ends up using the msbuild path, and the msvc doesn't understand the link flags passed to it.

Overall, we're in a state where the Windows/msbuild flow is broken upstream: anybody working on libclc on windows with MSBuild will have this problem. Also it seems the libclc CI is disabled upstream, so others are not catching it.

It's unreasonable for our clspv project to support this configuration as a blocker, so I will not block PRs based on the Windows build. I'll apply Required bits to the linux and macos builds, and not add them to the windows builds.

[frasercrmck]

I'm sorry that you're experiencing these problems.

You're right that we unfortunately don't have CI, I believe due to a fairly recent change which silently dropped libclc from testing. It was unlucky timing with the push towards using a runtimes build. I was under the impression that libclc was to be forced into the runtimes build but perhaps I misread the situation and we could have left it as it was.

It's unlikely I'm going to be able to look into any of these issues in the future. CC @wenju-he who might want to look into this for our downstream? Am I right in thinking that this will affect DPC++ in time?

[wenju-he]

The libclc build must use the host tools that it just built.

Right, in second-stage runtime build libclc uses in-tree just-built clang. The first stage LLVM build is using MSVC compiler.

The external build ends up using the msbuild path, and the msvc doesn't understand the link flags passed to it.

I haven't seen this issue in our downstream GPU compiler.
Not sure if this relates to which cmake generator is used. A few days ago we switched from MSVC generator to Ninja generator on windows to workaround libclc flaky error CMake error : Cannot restore timestamp due to a cmake bug (https://gitlab.kitware.com/cmake/cmake/-/issues/23495).

Am I right in thinking that this will affect DPC++ in time?

I'm currently working on enabling libclc as runtime build in our downstream GPU compiler. I haven't seen any issues with libclc, except that we can't set CLANG_BUILD_TOOLS to OFF. We used to set CLANG_BUILD_TOOLS to OFF to reduce build time. CLANG_BUILD_TOOLS is ON by default, and clang target is available in libclc runtime build because clang target is globally registered at https://github.com/llvm/llvm-project/blob/a5bff94ffd1b81a3562f02f05980ee87cc4164df/clang/cmake/modules/AddClang.cmake#L198

Enabling libclc as runtime in DPC++ compiler build is my next step. Not sure if I'll see the same issue as in this PR.

[wenju-he]

Link error on Windows. Some warnings, then an error linking ErrorHandler.obj

T:\src\github\clspv\build\third_party\llvm\Release\bin\opt.exe : warning : failed to create target machine for 'spir-unknown-unknown': unable to get target for 'spir-unknown-unknown', see --version and --triple. [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\builtins.opt.clspv--.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
    Building Custom Rule T:/src/github/clspv/third_party/llvm/libclc/CMakeLists.txt
    Building Custom Rule T:/src/github/clspv/third_party/llvm/libclc/utils/CMakeLists.txt
cl : command line warning D9002: ignoring unknown option '-fno-rtti' [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
cl : command line warning D9002: ignoring unknown option '-fno-exceptions' [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
    prepare-builtins.cpp
LINK : warning LNK4044: unrecognized option '/lpsapi'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/lshell32'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/lole32'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/luuid'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/ladvapi32'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/lws2_32'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/lntdll'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LINK : warning LNK4044: unrecognized option '/ldelayimp'; ignored [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
LLVMSupport.lib(ErrorHandling.obj) : error LNK2019: unresolved external symbol __imp_RtlGetLastNtStatus referenced in function "class std::error_code __cdecl llvm::mapLastWindowsError(void)" (?mapLastWindowsError@llvm@@YA?AVerror_code@std@@XZ) [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
shell32.lib(SHELL32.dll) : error LNK2001: unresolved external symbol __delayLoadHelper2 [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
ole32.lib(ole32.dll) : error LNK2001: unresolved external symbol __delayLoadHelper2 [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
T:\src\github\clspv\build\third_party\llvm\Release\bin\prepare_builtins.exe : fatal error LNK1120: 2 unresolved externals [T:\src\github\clspv\build\third_party\llvm\runtimes\runtimes-bins\libclc\utils\prepare_builtins.vcxproj] [T:\src\github\clspv\build\third_party\llvm\runtimes\libclc.vcxproj]
    Generating obj.libclc.dir/clspv64--/clc-convert.cl.bc

I reproduced the fail using MSVC generator:

cmake -G "Visual Studio 17 2022" -A x64 -Thost=x64 -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" -DLLVM_ENABLE_RUNTIMES="libclc" -DLLVM_INCLUDE_TESTS=OFF -DLLVM_BUILD_TESTS=OFF -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_TARGETS_TO_BUILD="X86;AMDGPU" -DCMAKE_INSTALL_PREFIX=install -DCMAKE_BUILD_TYPE=Release -DLIBCLC_TARGETS_TO_BUILD="amdgcn--amdhsa" ../llvm

The error is not reproducible with Ninja generator.

[wenju-he]

prepare_builtins link error on windows is probably a mix of two different issues which are not related to libclc.
The issues are at https://github.com/llvm/llvm-project/blob/dfa5bbeafaff2bbb211cde2980cc5f29906d8fac/llvm/cmake/modules/LLVMExternalProjectUtils.cmake#L228-L230 :

        set(compiler_args -DCMAKE_C_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang-cl${CMAKE_EXECUTABLE_SUFFIX}
                          -DCMAKE_CXX_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang-cl${CMAKE_EXECUTABLE_SUFFIX}
                          -DCMAKE_ASM_COMPILER=${LLVM_RUNTIME_OUTPUT_INTDIR}/clang-cl${CMAKE_EXECUTABLE_SUFFIX})

Issue 1: MSVC generator is multi-configuration generator, and LLVM_RUNTIME_OUTPUT_INTDIR contains build/$(Configuration)/bin. $(Configuration) can only be resolved at build time but not at cmake configure time. Therefore, in 2nd stage runtime build, CMAKE_C_COMPILER and CMAKE_CXX_COMPILER are not resolved and default msvc cl compiler is chosen as fallback.
Issue 2: However, what confuses me is that CMAKE_ASM_COMPILER is correctly resolved, e.g. to build/Release/bin/clang-cl.exe. The mismatch between CMAKE_CXX_COMPILER and CMAKE_ASM_COMPILER leads to the prepare_builtins link error. I don't know if it is an undocumented cmake behavior regarding differences between CMAKE_CXX_COMPILER and CMAKE_ASM_COMPILER.

Workaround for the link error is to use -DRUNTIMES_CMAKE_ARGS="-DCMAKE_C_COMPILER=cl;-DCMAKE_CXX_COMPILER=cl;-DCMAKE_ASM_COMPILER=cl" flag.

2nd stage runtime build should use freshly built clang, so the proper fix is to resolve $(Configuration) correctly. I can only think of writing ${LLVM_RUNTIME_OUTPUT_INTDIR}/clang-cl${CMAKE_EXECUTABLE_SUFFIX} to a cmake cache file within a COMMAND in add_custom_command, and append -C new-cache-file.cmake to ${cmake_args} in ExternalProject_Add. COMMAND is executed at build time and $(Configuration) can be resolved. However, I'm not sure if it is a clean fix.

[rjodinchr]

Configuring libclc appends during the runtime build stage of the LLVM cmake invocation. At that point all the binaries exist with the configuration resolved I believe. That's why it is able to pick CMAKE_ASM_COMPILER.

It feels that when using the visual studio generator, it ignores hints on CMAKE_C_COMPILER, thus choosing the one from the generator.

Workaround for the link error is to use -DRUNTIMES_CMAKE_ARGS="-DCMAKE_C_COMPILER=cl;-DCMAKE_CXX_COMPILER=cl;-DCMAKE_ASM_COMPILER=cl" flag.

Is this an idea, or did you try it and see it working? If it works, it can be good enough for clspv I guess.

[wenju-he]

Configuring libclc appends during the runtime build stage of the LLVM cmake invocation. At that point all the binaries exist with the configuration resolved I believe. That's why it is able to pick CMAKE_ASM_COMPILER.

Yes, all binaries are available in runtime cmake configure and build stage. I find that all binaries in https://github.com/llvm/llvm-project/blob/dfa5bbeafaff2bbb211cde2980cc5f29906d8fac/llvm/cmake/modules/LLVMExternalProjectUtils.cmake#L228-L283 are correctly found by cmake except for CMAKE_C_COMPILER and CMAKE_CXX_COMPILER.

It feels that when using the visual studio generator, it ignores hints on CMAKE_C_COMPILER, thus choosing the one from the generator.

You're right, see https://gitlab.kitware.com/cmake/cmake/-/issues/23385#note_1166850 : "Visual Studio generators do not support setting CMAKE_<LANG>_COMPILER directly. That is only supported by command-line generators like Ninja and NMake Makefiles."
So this probably means that freshly built clang can't be used for llvm runtime build when visual studio generator is used.

Workaround for the link error is to use -DRUNTIMES_CMAKE_ARGS="-DCMAKE_C_COMPILER=cl;-DCMAKE_CXX_COMPILER=cl;-DCMAKE_ASM_COMPILER=cl" flag.

Is this an idea, or did you try it and see it working? If it works, it can be good enough for clspv I guess.

Yes, I tried it and it works.

Ninja generator (not Ninja Multi-Config) also works and it can find and use the freshly built clang as compiler.

[rjodinchr]

Yes, I tried it and it works.

I'm having trouble understanding how it can work if CMAKE_C_COMPILER is ignored by Visual Studio. It feels like either it should get the correct clang from the CMAKE_C_COMPILER value already passed in by externalProject, or setting CMAKE_C_COMPILER in RUNTIMES_CMAKE_ARGS should not have any effect.

Am I missing something?

[wenju-he]

Yes, I tried it and it works.

I'm having trouble understanding how it can work if CMAKE_C_COMPILER is ignored by Visual Studio. It feels like either it should get the correct clang from the CMAKE_C_COMPILER value already passed in by externalProject, or setting CMAKE_C_COMPILER in RUNTIMES_CMAKE_ARGS should not have any effect.

Am I missing something?

-DRUNTIMES_CMAKE_ARGS="-DCMAKE_C_COMPILER=cl;-DCMAKE_CXX_COMPILER=cl;-DCMAKE_ASM_COMPILER=cl" flag will force using cl as CMAKE_ASM_COMPILER, then it matches with CMAKE_C_COMPILER which is cl too. The match will make the build pass.
So basically -DRUNTIMES_CMAKE_ARGS="-DCMAKE_ASM_COMPILER=cl" should work as well.

[wenju-he]

CMAKE_C_COMPILER will be cl regardless if CMAKE_C_COMPILER is set in -DRUNTIMES_CMAKE_ARGS for visual studio generator.

[wenju-he]

It feels like either it should get the correct clang from the CMAKE_C_COMPILER value already passed in by externalProject

Yes, this is correct per my understanding.
It is probably just that the CMAKE_C_COMPILER flag is ignored. But CMAKE_ASM_COMPILER flag is not ignored, so there is mismatch between C/CXX compiler and ASM compiler that caused the build error.

[rjodinchr]

Thanks a lot @wenju-he for all that information.
I'm trying both solutions right now (switching to ninja & defining RUNTIMES_CMAKE_ARGS) in clspv.
I only need one of them to work, I'll let you know my findings.

[rjodinchr]

Windows bots are back working in clspv by settings RUNTIMES_CMAKE_ARGS. Thanks again @wenju-he