[CUDA] Cannot find libdevice when compiling files in non-C drive

[yuxianch]

Describe the bug
Cannot find libdevice when compiling files with CUDA backend in non-C drive

To Reproduce
Supposing CUDA is installed in C drive, run the following command in non-C drive.

touch test.cpp
clang-cl /TP  -fsycl -fsycl-unnamed-lambda /EHsc -sycl-std=2020 -fsycl-targets=nvptx64-nvidia-cuda -std:c++17  -c test.cpp

It will show the following error:

clang-cl: error: cannot find libdevice for sm_50; provide path to different CUDA installation via '--cuda-path', or pass '-nocudalib' to build without linking with libdevice

If providing CUDA installation by "--cuda-path", then the error will be gone.

$ clang-cl /TP  -fsycl -fsycl-unnamed-lambda /EHsc -sycl-std=2020 -fsycl-targets=nvptx64-nvidia-cuda -std:c++17  -c test.cpp --cuda-path=C:/Program\ Files/NVIDIA\ GPU\ Computing\ Toolkit/CUDA/v11.4
warning: linking module
      'path_of_compiler_build\lib\clang\14.0.0\../../clc\remangled-l32-signed_char.libspirv-nvptx64--nvidiacl.bc':
      Linking two modules of different target triples:
      'path_of_compiler_build\lib\clang\14.0.0\../../clc\remangled-l32-signed_char.libspirv-nvptx64--nvidiacl.bc'
      is 'nvptx64-unknown-nvidiacl' whereas 'test.cpp' is 'nvptx64-nvidia-cuda'
 [-Wlinker-warnings]
1 warning generated.

Environment (please complete the following information):

• OS: Windows
• Target device and vendor: Nvidia GPU
• DPC++ version: https://github.com/intel/llvm.git 2869ca7
• CUDA version:

Cuda compilation tools, release 11.4, V11.4.120
Build cuda_11.4.r11.4/compiler.30300941_0

Additional context
This issue can be only be reproduced on non-C drive(where CUDA is installed).

[jchlanda]

Hi @yuxianch
Thank you for your report. I've looked into this a bit and am not sure if displayed behaviour is a bug, or desired feature. It boils down to the fact that candidate CUDA installation path is provided in a relative form, without the driver specifier: https://github.com/intel/llvm/blob/sycl/clang/lib/Driver/ToolChains/Cuda.cpp#L141 and the implementation of exists (vfs::FileSystem) makes the assumption that the drive should be the same as the one of clang-cl. Please note, that specifying clang-cl working directory to a location on c: fixes the issue. We could easily add a code to set the default drive to c:, while still respecting SysRoot, if provided:

diff --git a/clang/lib/Driver/ToolChains/Cuda.cpp b/clang/lib/Driver/ToolChains/Cuda.cpp
index 0058def1aacf..07ef60e01bdd 100644
--- a/clang/lib/Driver/ToolChains/Cuda.cpp
+++ b/clang/lib/Driver/ToolChains/Cuda.cpp
@@ -136,10 +136,11 @@ CudaInstallationDetector::CudaInstallationDetector(
     Candidates.emplace_back(
         Args.getLastArgValue(clang::driver::options::OPT_cuda_path_EQ).str());
   } else if (HostTriple.isOSWindows()) {
+    std::string DriveOrSysRoot = D.SysRoot.empty() ? "C:/" : D.SysRoot;
     for (const char *Ver : Versions)
       Candidates.emplace_back(
-          D.SysRoot + "/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v" +
-          Ver);
+          DriveOrSysRoot +
+          "/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v" + Ver);
   } else {
     if (!Args.hasArg(clang::driver::options::OPT_cuda_path_ignore_env)) {
       // Try to find ptxas binary. If the executable is located in a directory

but I am worried that there is code that relies on that implicit drive detection.

I'd be interested to hear your thoughts.

[yuxianch]

Hi @jchlanda
I am wondering could we check CUDA_PATH which is defined by the NVIDIA CUDA installer on Windows?

> echo %CUDA_PATH%
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4

[jchlanda]

That's a good idea. I'll add it to the Candidates vector, and since it's defined by the installer I'll make sure that it goes in first, so it'll end up being picked up, even if other versions are available on the system.