BLT CUDA targets: Compiling the CUDA compiler identification source file "CMakeCUDACompilerId.cu" failed.

[pgrete]

In the process of updating our CI infrastructure (see parthenon-hpc-lab/parthenon#1162) we somehow cannot make it work with Ascent anymore.
The Dockerfile in said PR results in a working Ascent build (current develop), but when trying to add it as depdency in our code we see the following error:

$ cmake -B build-ascent -DCMAKE_BUILD_TYPE=Release -DMACHINE_VARIANT=cuda-mpi -DMACHINE_CFG=$(pwd)/cmake/machinecfg/CI.cmake -DPARTHENON_ENABLE_ASCENT=ON -DAscent_DIR=/usr/local/ascent-checkout/lib/cmake/ascent

...

-- Building performance tests.
-- Building regression tests.
-- Creating BLT MPI targets...
-- FindMPI Enabled  (ENABLE_FIND_MPI == ON)
-- Found MPI_C: /opt/openmpi/lib/libmpi.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")  
-- BLT MPI Compile Flags:  
-- BLT MPI Include Paths:  /opt/openmpi/include
-- BLT MPI Libraries:      /opt/openmpi/lib/libmpi.so
-- BLT MPI Link Flags:     SHELL:-Wl,-rpath -Wl,/opt/openmpi/lib -Wl,--enable-new-dtags
-- MPI Executable:       /opt/openmpi/bin/mpiexec
-- MPI Num Proc Flag:    -n
-- MPI Command Append:   
-- Creating BLT CUDA targets...
CMake Error at /usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:726 (message):
  Compiling the CUDA compiler identification source file
  "CMakeCUDACompilerId.cu" failed.

  Compiler: /usr/local/cuda/bin/nvcc

  Build flags:

  Id flags:
  --keep;--keep-dir;tmp;-ccbin=/parthenon/external/Kokkos/bin/nvcc_wrapper -v

  

  The output was:

  2

  #$ _NVVM_BRANCH_=nvvm

  #$ _SPACE_=

  #$ _CUDART_=cudart

  #$ _HERE_=/usr/local/cuda/bin

  #$ _THERE_=/usr/local/cuda/bin

  #$ _TARGET_SIZE_=

  #$ _TARGET_DIR_=

  #$ _TARGET_DIR_=targets/x86_64-linux

  #$ TOP=/usr/local/cuda/bin/..

  #$ NVVMIR_LIBRARY_DIR=/usr/local/cuda/bin/../nvvm/libdevice

  #$
  LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:/opt/openmpi/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64


  #$
  PATH=/usr/local/cuda/bin/../nvvm/bin:/usr/local/cuda/bin:/opt/openmpi/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin


  #$ INCLUDES="-I/usr/local/cuda/bin/../targets/x86_64-linux/include"

  #$ LIBRARIES= "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib/stubs"
  "-L/usr/local/cuda/bin/../targets/x86_64-linux/lib"

  #$ CUDAFE_FLAGS=

  #$ PTXAS_FLAGS=

  #$ rm tmp/a_dlink.reg.c

  #$ "/parthenon/external/Kokkos/bin"/nvcc_wrapper -D__CUDA_ARCH_LIST__=520
  -E -x c++ -D__CUDACC__ -D__NVCC__
  "-I/usr/local/cuda/bin/../targets/x86_64-linux/include"
  -D__CUDACC_VER_MAJOR__=12 -D__CUDACC_VER_MINOR__=0
  -D__CUDACC_VER_BUILD__=140 -D__CUDA_API_VER_MAJOR__=12
  -D__CUDA_API_VER_MINOR__=0 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include
  "cuda_runtime.h" -m64 "CMakeCUDACompilerId.cu" -o
  "tmp/CMakeCUDACompilerId.cpp4.ii"

  <command-line>: warning: "__CUDA_ARCH_LIST__" redefined

  <command-line>: note: this is the location of the previous definition

  #$ cudafe++ --c++17 --gnu_version=110400 --display_error_number
  --orig_src_file_name "CMakeCUDACompilerId.cu" --orig_src_path_name
  "/parthenon/build-ascent/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu"
  --allow_managed --m64 --parse_templates --gen_c_file_name
  "tmp/CMakeCUDACompilerId.cudafe1.cpp" --stub_file_name
  "CMakeCUDACompilerId.cudafe1.stub.c" --gen_module_id_file
  --module_id_file_name "tmp/CMakeCUDACompilerId.module_id"
  "tmp/CMakeCUDACompilerId.cpp4.ii"

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(94):
  error: identifier "__match32_any_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(98):
  error: identifier "__match32_any_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(103):
  error: identifier "__match64_any_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(104):
  error: identifier "__match32_any_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(109):
  error: identifier "__match64_any_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(110):
  error: identifier "__match32_any_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(114):
  error: identifier "__match64_any_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(118):
  error: identifier "__match64_any_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(122):
  error: identifier "__match32_any_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(126):
  error: identifier "__match64_any_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(133):
  error: identifier "__match32_all_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(137):
  error: identifier "__match32_all_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(142):
  error: identifier "__match64_all_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(143):
  error: identifier "__match32_all_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(148):
  error: identifier "__match64_all_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(149):
  error: identifier "__match32_all_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(153):
  error: identifier "__match64_all_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(157):
  error: identifier "__match64_all_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(161):
  error: identifier "__match32_all_sync" is undefined

  

  /usr/local/cuda/bin/../targets/x86_64-linux/include/crt/sm_70_rt.hpp(165):
  error: identifier "__match64_all_sync" is undefined

  

  20 errors detected in the compilation of "CMakeCUDACompilerId.cu".

  # --error 0x2 --

  

  

Call Stack (most recent call first):
  /usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:6 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
  /usr/share/cmake-3.22/Modules/CMakeDetermineCompilerId.cmake:48 (__determine_compiler_id_test)
  /usr/share/cmake-3.22/Modules/CMakeDetermineCUDACompiler.cmake:298 (CMAKE_DETERMINE_COMPILER_ID)
  /usr/local/ascent-checkout/lib/cmake/ascent/thirdparty/BLTSetupCUDA.cmake:67 (enable_language)
  /usr/local/ascent-checkout/lib/cmake/ascent/BLTSetupTargets.cmake:97 (include)
  /usr/local/ascent-checkout/lib/cmake/ascent/AscentConfig.cmake:156 (include)
  CMakeLists.txt:370 (find_package)


-- Configuring incomplete, errors occurred!
See also "/parthenon/build-ascent/CMakeFiles/CMakeOutput.log".
See also "/parthenon/build-ascent/CMakeFiles/CMakeError.log".

This can directly be reproduced in the docker container (following a git clone https://github.com/parthenon-hpc-lab/parthenon.git and git submodule update --init).

I don't understand why/how the error is being triggered so any advice/hint is appreciated.

[cyrush]

@pgrete thanks for the report, I will try to do a build using your container recipe later this week to see if I can help.

[pgrete]

That'd be great. Thanks.
FWIW the machine I tried to build the container on/for has an ampere GPU with compute capabilities 80 (in contrast to the sm_70 that seem to be related to the issue).