[Bug] ERROR model_metadata.py:162: FAILED to read metadata section in legacy model lib when running Qwen-1_8B-Chat-q4f16_1-MLC

[tlopex]

🐛 Bug

When I want to initiate Qwen model after I compiled it, I met this bug: FAILED to read metadata section in legacy model lib

To Reproduce

Steps to reproduce the behavior:

1.python -m mlc_chat convert_weight ./dist/models/Qwen-1_8B-Chat/ --quantization q4f16_1 -o dist/Qwen-1_8B-Chat-q4f16_1-MLC
2.python -m mlc_chat gen_config ./dist/models/Qwen-1_8B-Chat/ --quantization q4f16_1 --conv-template qwen -o dist/Qwen-1_8B-Chat-q4f16_1-MLC/
3.python -m mlc_chat compile ./dist/Qwen-1_8B-Chat-q4f16_1-MLC/mlc-chat-config.json --device cuda -o dist/libs/Qwen-1_8B-Chat-q4f16_1-cuda.so
4.

python
>>> from mlc_chat import ChatModule
>>> cm = ChatModule(model="./dist/Qwen-1_8B-Chat-q4f16_1-MLC",model_lib_path="./dist/libs/Qwen-1_8B-Chat-q4f16_1-cuda.so")

Traceback

>>> cm = ChatModule(model="./dist/Qwen-1_8B-Chat-q4f16_1-MLC",model_lib_path="./dist/libs/Qwen-1_8B-Chat-q4f16_1-cuda.so")
[2024-02-08 14:40:19] INFO auto_device.py:76: Found device: cuda:0
[2024-02-08 14:40:19] INFO auto_device.py:85: Not found device: rocm:0
[2024-02-08 14:40:19] INFO auto_device.py:85: Not found device: metal:0
[2024-02-08 14:40:20] INFO auto_device.py:85: Not found device: vulkan:0
[2024-02-08 14:40:20] INFO auto_device.py:85: Not found device: opencl:0
[2024-02-08 14:40:20] INFO auto_device.py:33: Using device: cuda:0
[2024-02-08 14:40:20] INFO chat_module.py:370: Using model folder: /home/tlopex/mlc-llm/dist/Qwen-1_8B-Chat-q4f16_1-MLC
[2024-02-08 14:40:20] INFO chat_module.py:371: Using mlc chat config: /home/tlopex/mlc-llm/dist/Qwen-1_8B-Chat-q4f16_1-MLC/mlc-chat-config.json
[2024-02-08 14:40:20] INFO chat_module.py:513: Using library model: ./dist/libs/Qwen-1_8B-Chat-q4f16_1-cuda.so
[2024-02-08 14:40:21] ERROR model_metadata.py:162: FAILED to read metadata section in legacy model lib.
Traceback (most recent call last):
  File "/home/tlopex/mlc-llm/python/mlc_chat/cli/model_metadata.py", line 160, in main
    metadata = _extract_metadata(parsed.model_lib)
  File "/home/tlopex/mlc-llm/python/mlc_chat/cli/model_metadata.py", line 26, in _extract_metadata
    return json.loads(VirtualMachine(load_module(model_lib), device("cpu"))["_metadata"]())
  File "/home/tlopex/relax/python/tvm/runtime/relax_vm.py", line 97, in __init__
    self._setup_device(device, memory_cfg)
  File "/home/tlopex/relax/python/tvm/runtime/relax_vm.py", line 133, in _setup_device
    self.module["vm_initialization"](*init_args)
  File "/home/tlopex/relax/python/tvm/_ffi/_ctypes/packed_func.py", line 239, in __call__
    raise_last_ffi_error()
  File "/home/tlopex/relax/python/tvm/_ffi/base.py", line 481, in raise_last_ffi_error
    raise py_err
tvm.error.InternalError: Traceback (most recent call last):
  2: tvm::runtime::relax_vm::VirtualMachineImpl::_Init(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
  1: tvm::runtime::relax_vm::VirtualMachineImpl::Init(std::vector<DLDevice, std::allocator<DLDevice> > const&, std::vector<tvm::runtime::memory::AllocatorType, std::allocator<tvm::runtime::memory::AllocatorType> > const&)
  0: tvm::runtime::relax_vm::VirtualMachineImpl::InitFuncPool()
  File "/home/tlopex/relax/src/runtime/relax_vm/vm.cc", line 676
InternalError: Check failed: (func.defined()) is false: Error: Cannot find PackedFunc flashinfer.single_prefill in either Relax VM kernel library, or in TVM runtime PackedFunc registry, or in global Relax functions of the VM executable
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tlopex/mlc-llm/python/mlc_chat/chat_module.py", line 780, in __init__
    self._reload(self.model_lib_path, self.model_path, user_chat_config_json_str)
  File "/home/tlopex/mlc-llm/python/mlc_chat/chat_module.py", line 1008, in _reload
    self._reload_func(lib, model_path, app_config_json, kv_cache_config.asjson())
  File "/home/tlopex/relax/python/tvm/_ffi/_ctypes/packed_func.py", line 239, in __call__
    raise_last_ffi_error()
  File "/home/tlopex/relax/python/tvm/_ffi/base.py", line 481, in raise_last_ffi_error
    raise py_err
  File "/home/tlopex/mlc-llm/cpp/llm_chat.cc", line 1613, in mlc::llm::LLMChatModule::GetFunction(tvm::runtime::String const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
    chat_->Reload(args[0], args[1], args[2], args[3]);
  File "/home/tlopex/mlc-llm/cpp/llm_chat.cc", line 594, in mlc::llm::LLMChat::Reload(tvm::runtime::TVMArgValue, tvm::runtime::String, tvm::runtime::String, tvm::runtime::String)
    this->ft_.Init(reload_lib, device_, model_config);
  File "/home/tlopex/mlc-llm/cpp/llm_chat.cc", line 171, in Init
    this->local_vm->GetFunction("vm_initialization")(
tvm.error.InternalError: Traceback (most recent call last):
  5: mlc::llm::LLMChatModule::GetFunction(tvm::runtime::String const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
        at /home/tlopex/mlc-llm/cpp/llm_chat.cc:1613
  4: mlc::llm::LLMChat::Reload(tvm::runtime::TVMArgValue, tvm::runtime::String, tvm::runtime::String, tvm::runtime::String)
        at /home/tlopex/mlc-llm/cpp/llm_chat.cc:594
  3: Init
        at /home/tlopex/mlc-llm/cpp/llm_chat.cc:171
  2: tvm::runtime::relax_vm::VirtualMachineImpl::_Init(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
  1: tvm::runtime::relax_vm::VirtualMachineImpl::Init(std::vector<DLDevice, std::allocator<DLDevice> > const&, std::vector<tvm::runtime::memory::AllocatorType, std::allocator<tvm::runtime::memory::AllocatorType> > const&)
  0: tvm::runtime::relax_vm::VirtualMachineImpl::InitFuncPool()
  File "/home/tlopex/relax/src/runtime/relax_vm/vm.cc", line 676
InternalError: Check failed: (func.defined()) is false: Error: Cannot find PackedFunc flashinfer.single_prefill in either Relax VM kernel library, or in TVM runtime PackedFunc registry, or in global Relax functions of the VM executable

Environment

• Platform : CUDA
• Operating system : Ubuntu 20.04
• Device: RTX3080TI
• How you installed MLC-LLM : source
• How you installed TVM-Unity : source
• Python version : 3.8.10
• GPU driver version (if applicable): 535.154.05
• CUDA/cuDNN version (if applicable):12.2

[tlopex]

I found that I will encounter the same bug when I launched stableLM and mistral. And I have reinstalled MLC-LLM but the bug is still here.

[Hzfengsy]

Please enable FlashInfer when building mlc_chat if you are building from the source.

[tlopex]

@Hzfengsy Thanks for your reply! I built mlc_chat from the source following the Doc https://llm.mlc.ai/docs/install/mlc_llm.html#install-mlc-packages , but I cannot find the place where I can enable FlashInfer. Could you tell me where I can enable FlashInfer?

[yashsinghcodes]

I'm getting same error, using linux vulkan. I used mlc_chat prebuilt package

[Hzfengsy]

@tlopex Thanks for reporting this. We should update the file gen_cmake_config.py to support FlashInfer (as well as NCCL), we can open this issue until the fix.

For now, you could update config.cmake manually, refer here

[Hzfengsy]

@yashsinghcodes FlashInfer is only enabled on cuda devices. Please share your error msg on your env

[tlopex]

@Hzfengsy Thanks for your solution!
I tried manually update my config.cmake , but I still encountered problem when building mlc_chat
This is my config.cmake

set(TVM_HOME /home/tlopex/relax)
set(CMAKE_BUILD_TYPE RelWithDebInfo)
set(USE_CUDA ON)
set(USE_CUTLASS OFF)
set(USE_CUBLAS OFF)
set(USE_ROCM OFF)
set(USE_VULKAN OFF)
set(USE_METAL OFF)
set(USE_OPENCL OFF)
set(USE_NCCL ON)
set(USE_FLASHINFER ON)
set(CMAKE_CUDA_ARCHITECTURES )
set(CMAKE_CUDA_FLAGS "--expt-relaxed-constexpr")
set(USE_CUDA ON)
set(USE_CUTLASS OFF)
set(USE_CUBLAS OFF)
set(USE_NCCL OFF)
set(USE_FLASHINFER ON)
set(CMAKE_CUDA_ARCHITECTURES )
set(CMAKE_CUDA_FLAGS "--expt-relaxed-constexpr")

And this is my error Info

tlopex@tlopex-OMEN-by-HP-Laptop-17-ck1xxx:~/mlc-llm/build$ cmake .. && cmake --build . --parallel $(nproc) && cd ..
-- Hide private symbols
-- TVM_HOME: /home/tlopex/relax
-- Hide private symbols...
-- Forbidding undefined symbols in shared library, using -Wl,--no-undefined on platform Linux
-- Didn't find the path to CCACHE, disabling ccache
-- VTA build with VTA_HW_PATH=/home/tlopex/relax/3rdparty/vta-hw
-- Build VTA runtime with target: sim
CMake Warning (dev) at /home/tlopex/relax/cmake/utils/FindCUDA.cmake:44 (find_package):
  Policy CMP0146 is not set: The FindCUDA module is removed.  Run "cmake
  --help-policy CMP0146" for policy details.  Use the cmake_policy command to
  set the policy and suppress this warning.

Call Stack (most recent call first):
  /home/tlopex/relax/cmake/modules/CUDA.cmake:19 (find_cuda)
  /home/tlopex/relax/CMakeLists.txt:537 (include)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Found CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-12.2
-- Found CUDA_CUDA_LIBRARY=/usr/local/cuda-12.2/targets/x86_64-linux/lib/stubs/libcuda.so
-- Found CUDA_CUDART_LIBRARY=/usr/local/cuda-12.2/lib64/libcudart.so
-- Found CUDA_NVRTC_LIBRARY=/usr/local/cuda-12.2/lib64/libnvrtc.so
-- Found CUDA_CUDNN_INCLUDE_DIRS=
-- Found CUDA_CUDNN_LIBRARY=
-- Found CUDA_CUBLAS_LIBRARY=/usr/local/cuda-12.2/lib64/libcublas.so
-- Found CUDA_CURAND_LIBRARY=/usr/local/cuda-12.2/lib64/libcurand.so
-- Found CUDA_CUBLASLT_LIBRARY=/usr/local/cuda-12.2/lib64/libcublasLt.so
-- Found CUDA_NVTX_LIBRARY=/usr/local/cuda-12.2/lib64/libnvToolsExt.so
-- Found CUDA_nvToolsExt_LIBRARY=/usr/local/cuda-12.2/lib64/libnvToolsExt.so
-- Build with CUDA 12.2 support
-- Build with contrib.random
-- Build with contrib.sort
-- Build with contrib.hybriddump
-- Git found: /usr/bin/git
-- Found TVM_GIT_COMMIT_HASH=484d42526a9992c372d1687ff68a86bda0d1c238
-- Found TVM_GIT_COMMIT_TIME=2024-02-07 16:36:37 -0500
-- Building with TVM Map...
-- Build with thread support...
-- Added "-fuse-ld=lld" to linker flags 
-- Build with FlashInfer
-- CMAKE_CUDA_ARCHITECTURES is 52
-- Compile tvm binding.
-- FlashInfer uses TVM home /home/tlopex/relax.
-- system-nameLinux
CMake Deprecation Warning at 3rdparty/tokenizers-cpp/msgpack/CMakeLists.txt:1 (CMAKE_MINIMUM_REQUIRED):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
CMake Deprecation Warning at 3rdparty/tokenizers-cpp/sentencepiece/CMakeLists.txt:15 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- VERSION: 0.2.00
CMake Warning at CMakeLists.txt:99 (message):
  Cannot find libflash_attn.  The model must not have been built with
  --use-flash-attn-mqa option.


-- Configuring done (0.2s)
-- Generating done (0.1s)
-- Build files have been written to: /home/tlopex/mlc-llm/build
[  0%] Building CUDA object tvm/3rdparty/flashinfer/CMakeFiles/flashinfer_tvm.dir/src/tvm_wrapper.cu.o
[  0%] Built target tokenizers_c
[  7%] Building CXX object tvm/CMakeFiles/tvm_libinfo_objs.dir/src/support/libinfo.cc.o
[  7%] Built target tokenizer_cpp_objs
[ 28%] Built target mlc_llm_objs
[ 57%] Built target sentencepiece-static
[ 57%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/builtin_fp16.cc.o
[ 57%] Built target tokenizers_cpp
[ 57%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/c_runtime_api.cc.o
[ 57%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/const_loader_module.cc.o
[ 64%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/container.cc.o
[ 64%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/cpu_device_api.cc.o
[ 64%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/debug.cc.o
[ 64%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/disco/bcast_session.cc.o
[ 64%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/disco/builtin.cc.o
[ 64%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/disco/disco_worker.cc.o
[ 64%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/disco/loader.cc.o
[ 64%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/disco/process_session.cc.o
[ 64%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/disco/session.cc.o
[ 71%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/disco/threaded_session.cc.o
[ 71%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/dso_library.cc.o
[ 71%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/file_utils.cc.o
[ 71%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/library_module.cc.o
[ 71%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/logging.cc.o
[ 71%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/metadata.cc.o
[ 71%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/memory/memory_manager.cc.o
[ 71%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/minrpc/minrpc_logger.cc.o
[ 71%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/module.cc.o
[ 71%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/name_transforms.cc.o
[ 78%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/ndarray.cc.o
In file included from /usr/local/cuda-12.2/include/cuda/barrier:14,
                 from /usr/local/cuda-12.2/include/cuda/pipeline:56,
                 from /home/tlopex/relax/3rdparty/flashinfer/include/flashinfer/decode.cuh:26,
                 from /home/tlopex/relax/3rdparty/flashinfer/include/flashinfer.cuh:20,
                 from /home/tlopex/relax/3rdparty/flashinfer/src/tvm_wrapper.cu:23:
/usr/local/cuda-12.2/include/cuda/std/barrier:15:4: error: #error "CUDA synchronization primitives are only supported for sm_70 and up."
   15 | #  error "CUDA synchronization primitives are only supported for sm_70 and up."
      |    ^~~~~
In file included from /usr/local/cuda-12.2/include/cuda/std/detail/libcxx/include/atomic:727,
                 from /usr/local/cuda-12.2/include/cuda/std/detail/libcxx/include/barrier:60,
                 from /usr/local/cuda-12.2/include/cuda/std/barrier:22,
                 from /usr/local/cuda-12.2/include/cuda/barrier:14,
                 from /usr/local/cuda-12.2/include/cuda/pipeline:56,
                 from /home/tlopex/relax/3rdparty/flashinfer/include/flashinfer/decode.cuh:26,
                 from /home/tlopex/relax/3rdparty/flashinfer/include/flashinfer.cuh:20,
                 from /home/tlopex/relax/3rdparty/flashinfer/src/tvm_wrapper.cu:23:
/usr/local/cuda-12.2/include/cuda/std/detail/libcxx/include/support/atomic/atomic_cuda.h:12:4: error: #error "CUDA atomics are only supported for sm_60 and up on *nix and sm_70 and up on Windows."
   12 | #  error "CUDA atomics are only supported for sm_60 and up on *nix and sm_70 and up on Windows."
      |    ^~~~~
In file included from /usr/local/cuda-12.2/include/cuda/std/detail/libcxx/include/barrier:487,
                 from /usr/local/cuda-12.2/include/cuda/std/barrier:22,
                 from /usr/local/cuda-12.2/include/cuda/barrier:14,
                 from /usr/local/cuda-12.2/include/cuda/pipeline:56,
                 from /home/tlopex/relax/3rdparty/flashinfer/include/flashinfer/decode.cuh:26,
                 from /home/tlopex/relax/3rdparty/flashinfer/include/flashinfer.cuh:20,
                 from /home/tlopex/relax/3rdparty/flashinfer/src/tvm_wrapper.cu:23:
/usr/local/cuda-12.2/include/cuda/std/detail/libcxx/include/__cuda/barrier.h:19:4: error: #error "CUDA synchronization primitives are only supported for sm_70 and up."
   19 | #  error "CUDA synchronization primitives are only supported for sm_70 and up."
      |    ^~~~~
In file included from /home/tlopex/relax/3rdparty/flashinfer/include/flashinfer/prefill.cuh:32,
                 from /home/tlopex/relax/3rdparty/flashinfer/include/flashinfer/handler.cuh:25,
                 from /home/tlopex/relax/3rdparty/flashinfer/include/flashinfer.cuh:21,
                 from /home/tlopex/relax/3rdparty/flashinfer/src/tvm_wrapper.cu:23:
/home/tlopex/relax/3rdparty/flashinfer/include/flashinfer/mma.cuh:64:2: error: #error "Unsupported CUDA architecture for ldmatrix instruction"
   64 | #error "Unsupported CUDA architecture for ldmatrix instruction"
      |  ^~~~~
/home/tlopex/relax/3rdparty/flashinfer/include/flashinfer/mma.cuh:83:2: error: #error "Unsupported CUDA architecture for ldmatrix instruction"
   83 | #error "Unsupported CUDA architecture for ldmatrix instruction"
      |  ^~~~~
/home/tlopex/relax/3rdparty/flashinfer/include/flashinfer/mma.cuh:282:2: error: #error "Unsupported CUDA architecture for mma instruction"
  282 | #error "Unsupported CUDA architecture for mma instruction"
      |  ^~~~~
/home/tlopex/relax/3rdparty/flashinfer/include/flashinfer/mma.cuh:322:2: error: #error "Unsupported CUDA architecture for mma instruction"
  322 | #error "Unsupported CUDA architecture for mma instruction"
      |  ^~~~~
/home/tlopex/relax/3rdparty/flashinfer/include/flashinfer/mma.cuh:442:2: error: #error "Unsupported CUDA architecture for mma instruction"
  442 | #error "Unsupported CUDA architecture for mma instruction"
      |  ^~~~~
make[2]: *** [tvm/3rdparty/flashinfer/CMakeFiles/flashinfer_tvm.dir/build.make:77：tvm/3rdparty/flashinfer/CMakeFiles/flashinfer_tvm.dir/src/tvm_wrapper.cu.o] 错误 1
make[1]: *** [CMakeFiles/Makefile2:536：tvm/3rdparty/flashinfer/CMakeFiles/flashinfer_tvm.dir/all] 错误 2
make[1]: *** 正在等待未完成的任务....
[ 78%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/nvtx.cc.o
[ 78%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/object.cc.o
[ 78%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/packed_func.cc.o
[ 78%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/profiling.cc.o
[ 78%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/registry.cc.o
[ 78%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/relax_vm/builtin.cc.o
[ 78%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/relax_vm/bytecode.cc.o
[ 78%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/relax_vm/executable.cc.o
[ 85%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/relax_vm/lm_support.cc.o
[ 85%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/relax_vm/ndarray_cache_support.cc.o
[ 85%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/relax_vm/paged_kv_cache.cc.o
[ 85%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/relax_vm/vm.cc.o
[ 85%] Built target tvm_libinfo_objs
[ 85%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/source_utils.cc.o
[ 85%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/static_library.cc.o
[ 85%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/system_library.cc.o
[ 85%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/thread_pool.cc.o
[ 85%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/threading_backend.cc.o
[ 92%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/vm/bytecode.cc.o
[ 92%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/vm/executable.cc.o
[ 92%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/vm/vm.cc.o
[ 92%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/workspace_pool.cc.o
[ 92%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/cuda/cuda_device_api.cc.o
[ 92%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/cuda/cuda_module.cc.o
[ 92%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/cuda/l2_cache_flush.cc.o
[ 92%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/relax_vm/cuda/cuda_graph_builtin.cc.o
[ 92%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/contrib/random/random.cc.o
[100%] Building CXX object tvm/CMakeFiles/tvm_runtime_objs.dir/src/runtime/contrib/sort/sort.cc.o
[100%] Built target tvm_runtime_objs
make: *** [Makefile:156：all] 错误 2

It said that

error: #error "CUDA synchronization primitives are only supported for sm_70 and up.

However, my GPU is RTX3080Ti and I checked from NVIDIA's official website that it's Compute Capability should be 8.6(sm_86), so I cannot understand why it would happen.

[CharlieFRuan]

Hi @tlopex @yashsinghcodes please refer to this comment here: #1551 (comment)

Apologies for the inconvenience; I will update the doc correspondingly soon

[tlopex]

@CharlieFRuan Thanks! That solved my problem!