[Bug] [llama2-7B] fail to execute Llama-2-7b-chat-hf-q4f16_1-MLC

[SunCrazy]

🐛 Bug

When I execute Llama-2-7b-chat-hf-q4f16_1-MLC, some errors are raised as follow:

(relax-py10) [chenf@b11b0623:/mnt/ssd/chenf/project/mlc_models/example/llama2]$ /mnt/ssd/chenf/opensource/mlc-llm/build/mlc_chat_cli --model-lib-path prebuilt_libs/Llama-2-7b-chat-hf-q4f16_1-cuda.so --model Llama-2-7b-chat-hf-q4f16_1-MLC/
Use MLC config: "/mnt/ssd/chenf/project/mlc_models/example/llama2/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json"
Use model weights: "/mnt/ssd/chenf/project/mlc_models/example/llama2/Llama-2-7b-chat-hf-q4f16_1-MLC/ndarray-cache.json"
Use model library: "prebuilt_libs/Llama-2-7b-chat-hf-q4f16_1-cuda.so"
You can use the following special commands:
  /help               print the special commands
  /exit               quit the cli
  /stats              print out the latest stats (token/sec)
  /reset              restart a fresh chat
  /reload [model]  reload model `model` from disk, or reload the current model if `model` is not specified

Loading model...
Loading finished
Running system prompts...
[11:32:03] /mnt/ssd/chenf/opensource/mlc-llm/3rdparty/tvm/src/runtime/relax_vm/builtin.cc:310: Check failed: (static_cast<int64_t>(ptr->size()) == size) is false: ValueError: ErrorContext(fn=prefill, loc=param[3], param=params, annotation=R.Tuple(R.Tensor((v, 512), dtype="uint32"), R.Tensor((v, 128), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((12288, 512), dtype="uint32"), R.Tensor((12288, 128), dtype="float16"), R.Tensor((4096, 512), dtype="uint32"), R.Tensor((4096, 128), dtype="float16"), R.Tensor((22016, 512), dtype="uint32"), R.Tensor((22016, 128), dtype="float16"), R.Tensor((4096, 1376), dtype="uint32"), R.Tensor((4096, 344), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((4096,), dtype="float16"), R.Tensor((v, 512), dtype="uint32"), R.Tensor((v, 128), dtype="float16"), R.Tensor((cache_len, 128), dtype="float16"), R.Tensor((cache_len, 128), dtype="float16")))  expect a Tuple with 327 elements,  but get a Tuple with 0 elements.
Stack trace:
  [bt] (0) /mnt/ssd/chenf/opensource/mlc-llm/build/tvm/libtvm_runtime.so(tvm::runtime::Backtrace[abi:cxx11]()+0x2c) [0x7f58a951c3ac]
  [bt] (1) /mnt/ssd/chenf/opensource/mlc-llm/build/mlc_chat_cli(tvm::runtime::detail::LogFatal::Entry::Finalize()+0x3d) [0x56421e368aad]
  [bt] (2) /mnt/ssd/chenf/opensource/mlc-llm/build/tvm/libtvm_runtime.so(tvm::runtime::relax_vm::CheckTupleInfo(tvm::runtime::ObjectRef, long, tvm::runtime::Optional<tvm::runtime::String>)+0x29f) [0x7f58a957515f]
  [bt] (3) /mnt/ssd/chenf/opensource/mlc-llm/build/tvm/libtvm_runtime.so(tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::runtime::TypedPackedFunc<void (tvm::runtime::ObjectRef, long, tvm::runtime::Optional<tvm::runtime::String>)>::AssignTypedLambda<void (*)(tvm::runtime::ObjectRef, long, tvm::runtime::Optional<tvm::runtime::String>)>(void (*)(tvm::runtime::ObjectRef, long, tvm::runtime::Optional<tvm::runtime::String>), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)+0x216) [0x7f58a9589c96]
  [bt] (4) /mnt/ssd/chenf/opensource/mlc-llm/build/tvm/libtvm_runtime.so(tvm::runtime::relax_vm::VirtualMachineImpl::InvokeClosurePacked(tvm::runtime::ObjectRef const&, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)+0x7d) [0x7f58a95d271d]
  [bt] (5) /mnt/ssd/chenf/opensource/mlc-llm/build/tvm/libtvm_runtime.so(tvm::runtime::relax_vm::VirtualMachineImpl::RunInstrCall(tvm::runtime::relax_vm::VMFrame*, tvm::runtime::relax_vm::Instruction)+0x90b) [0x7f58a95d471b]
  [bt] (6) /mnt/ssd/chenf/opensource/mlc-llm/build/tvm/libtvm_runtime.so(tvm::runtime::relax_vm::VirtualMachineImpl::RunLoop()+0x235) [0x7f58a95d3ba5]
  [bt] (7) /mnt/ssd/chenf/opensource/mlc-llm/build/tvm/libtvm_runtime.so(tvm::runtime::relax_vm::VirtualMachineImpl::InvokeBytecode(long, std::vector<tvm::runtime::TVMRetValue, std::allocator<tvm::runtime::TVMRetValue> > const&)+0x2a4) [0x7f58a95d56b4]
  [bt] (8) /mnt/ssd/chenf/opensource/mlc-llm/build/tvm/libtvm_runtime.so(+0x218cba) [0x7f58a95d5cba]

To Reproduce

Steps to reproduce the behavior:

1. compile mlc runtime from source (use tvm in submodule); mlc-llm commit: 5e23900
2. download model and prebuild_lib from provided url;
3. execute command: /mnt/ssd/chenf/opensource/mlc-llm/build/mlc_chat_cli --model-lib-path prebuilt_libs/Llama-2-7b-chat-hf-q4f16_1-cuda.so --model Llama-2-7b-chat-hf-q4f16_1-MLC/

Environment

• Platform (e.g. WebGPU/Vulkan/IOS/Android/CUDA): CUDA
• Operating system (e.g. Ubuntu/Windows/MacOS/...): Ubuntu 20.04
• Device (e.g. iPhone 12 Pro, PC+RTX 3090, ...): Tesla V100
• How you installed MLC-LLM (conda, source): source
• How you installed TVM-Unity (pip, source): source
• Python version (e.g. 3.10): 3.10
• GPU driver version (if applicable):
• CUDA/cuDNN version (if applicable):
• TVM Unity Hash Tag (python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))", applicable if you compile models):

USE_NVTX: OFF
USE_GTEST: AUTO
SUMMARIZE: OFF
USE_IOS_RPC: OFF
USE_MSC: OFF
USE_ETHOSU: OFF
CUDA_VERSION: 11.6
USE_LIBBACKTRACE: AUTO
DLPACK_PATH: 3rdparty/dlpack/include
USE_TENSORRT_CODEGEN: OFF
USE_THRUST: OFF
USE_TARGET_ONNX: OFF
USE_AOT_EXECUTOR: ON
BUILD_DUMMY_LIBTVM: OFF
USE_CUDNN: OFF
USE_TENSORRT_RUNTIME: OFF
USE_ARM_COMPUTE_LIB_GRAPH_EXECUTOR: OFF
USE_CCACHE: AUTO
USE_ARM_COMPUTE_LIB: OFF
USE_CPP_RTVM: OFF
USE_OPENCL_GTEST: /path/to/opencl/gtest
USE_MKL: OFF
USE_PT_TVMDSOOP: OFF
MLIR_VERSION: NOT-FOUND
USE_CLML: OFF
USE_STACKVM_RUNTIME: OFF
USE_GRAPH_EXECUTOR_CUDA_GRAPH: OFF
ROCM_PATH: /opt/rocm
USE_DNNL: OFF
USE_VITIS_AI: OFF
USE_MLIR: OFF
USE_RCCL: OFF
USE_LLVM: /mnt/ssd/chenf/software/miniconda3/envs/relax-py10/bin/llvm-config
USE_VERILATOR: OFF
USE_TF_TVMDSOOP: OFF
USE_THREADS: ON
USE_MSVC_MT: OFF
BACKTRACE_ON_SEGFAULT: OFF
USE_GRAPH_EXECUTOR: ON
USE_NCCL: OFF
USE_ROCBLAS: OFF
GIT_COMMIT_HASH: 7dfc863df8b6c9227a03547e5a0bf23f44c3f62d
USE_VULKAN: OFF
USE_RUST_EXT: OFF
USE_CUTLASS: OFF
USE_CPP_RPC: OFF
USE_HEXAGON: OFF
USE_CUSTOM_LOGGING: OFF
USE_UMA: OFF
USE_FALLBACK_STL_MAP: OFF
USE_SORT: ON
USE_RTTI: ON
GIT_COMMIT_TIME: 2024-01-04 15:14:07 +0800
USE_HEXAGON_SDK: /path/to/sdk
USE_BLAS: none
USE_ETHOSN: OFF
USE_LIBTORCH: OFF
USE_RANDOM: ON
USE_CUDA: ON
USE_COREML: OFF
USE_AMX: OFF
BUILD_STATIC_RUNTIME: OFF
USE_CMSISNN: OFF
USE_KHRONOS_SPIRV: OFF
USE_CLML_GRAPH_EXECUTOR: OFF
USE_TFLITE: OFF
USE_HEXAGON_GTEST: /path/to/hexagon/gtest
PICOJSON_PATH: 3rdparty/picojson
USE_OPENCL_ENABLE_HOST_PTR: OFF
INSTALL_DEV: OFF
USE_PROFILER: ON
USE_NNPACK: OFF
LLVM_VERSION: 17.0.6
USE_OPENCL: OFF
COMPILER_RT_PATH: 3rdparty/compiler-rt
RANG_PATH: 3rdparty/rang/include
USE_SPIRV_KHR_INTEGER_DOT_PRODUCT: OFF
USE_OPENMP: none
USE_BNNS: OFF
USE_CUBLAS: OFF
USE_METAL: OFF
USE_MICRO_STANDALONE_RUNTIME: OFF
USE_HEXAGON_EXTERNAL_LIBS: OFF
USE_ALTERNATIVE_LINKER: AUTO
USE_BYODT_POSIT: OFF
USE_HEXAGON_RPC: OFF
USE_MICRO: OFF
DMLC_PATH: 3rdparty/dmlc-core/include
INDEX_DEFAULT_I64: ON
USE_RELAY_DEBUG: OFF
USE_RPC: ON
USE_TENSORFLOW_PATH: none
TVM_CLML_VERSION:
USE_MIOPEN: OFF
USE_ROCM: OFF
USE_PAPI: OFF
USE_CURAND: OFF
TVM_CXX_COMPILER_PATH: /usr/bin/c++
HIDE_PRIVATE_SYMBOLS: ON

• Any other relevant information:

[junrushao]

The C++-based CLI is less maintained, and we could instead use Python APIs.

Could you instead do:

rm -rf prebuilt_libs/

And then use the Python script below:

from mlc_chat import ChatConfig, ChatModule, callback
from mlc_chat.support import logging

logging.enable_logging()

MODEL = "/mnt/ssd/chenf/project/mlc_models/example/llama2/Llama-2-7b-chat-hf-q4f16_1-MLC/"

def main():
    cm = ChatModule(
        MODEL,
        chat_config=ChatConfig(context_window_size=1024),
    )
    cm.generate(
        "What is the meaning of life?",
        progress_callback=callback.StreamToStdout(callback_interval=2),
    )

if __name__ == "__main__":
    main()

[SunCrazy]

The C++-based CLI is less maintained, and we could instead use Python APIs.

Could you instead do:

rm -rf prebuilt_libs/

And then use the Python script below:

from mlc_chat import ChatConfig, ChatModule, callback
from mlc_chat.support import logging

logging.enable_logging()

MODEL = "/mnt/ssd/chenf/project/mlc_models/example/llama2/Llama-2-7b-chat-hf-q4f16_1-MLC/"

def main():
    cm = ChatModule(
        MODEL,
        chat_config=ChatConfig(context_window_size=1024),
    )
    cm.generate(
        "What is the meaning of life?",
        progress_callback=callback.StreamToStdout(callback_interval=2),
    )

if __name__ == "__main__":
    main()

@junrushao Thanks for your reply! I change to execute the python script above, but get new errors:

I guess I didn't compile a dependency when compiling TVM, right？

[2024-01-07 17:08:31] INFO auto_device.py:76: Found device: cuda:0
[2024-01-07 17:08:32] INFO auto_device.py:85: Not found device: rocm:0
[2024-01-07 17:08:32] INFO auto_device.py:85: Not found device: metal:0
[2024-01-07 17:08:32] INFO auto_device.py:85: Not found device: vulkan:0
[2024-01-07 17:08:33] INFO auto_device.py:85: Not found device: opencl:0
[2024-01-07 17:08:33] INFO auto_device.py:33: Using device: cuda:0
[2024-01-07 17:08:33] INFO chat_module.py:366: Using model folder: /mnt/ssd/chenf/project/mlc_models/example/llama2/Llama-2-7b-chat-hf-q4f16_1-MLC
[2024-01-07 17:08:33] INFO chat_module.py:367: Using mlc chat config: /mnt/ssd/chenf/project/mlc_models/example/llama2/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json
[2024-01-07 17:08:33] INFO chat_module.py:756: Model lib not found. Now compiling model lib on device...
[2024-01-07 17:08:33] INFO llama_model.py:79: Overriding prefill_chunk_size from 4096 to 1024 (context_window_size)
[2024-01-07 17:08:33] INFO jit.py:83: Compiling using commands below:
[2024-01-07 17:08:33] INFO jit.py:84: /mnt/ssd/chenf/software/miniconda3/envs/relax-py10/bin/python -m mlc_chat compile /mnt/ssd/chenf/project/mlc_models/example/llama2/Llama-2-7b-chat-hf-q4f16_1-MLC --opt 'flashinfer=1;cublas_gemm=1;cudagraph=0' --overrides 'context_window_size=1024;prefill_chunk_size=4096;tensor_parallel_shards=1' --device cuda:0 --output /tmp/tmpqjedea0v/lib.so
[2024-01-07 17:08:34] INFO auto_config.py:69: Found model configuration: /mnt/ssd/chenf/project/mlc_models/example/llama2/Llama-2-7b-chat-hf-q4f16_1-MLC/mlc-chat-config.json
[2024-01-07 17:08:34] INFO auto_target.py:75: Detecting target device: cuda:0
[2024-01-07 17:08:34] INFO auto_target.py:77: Found target: {"thread_warp_size": 32, "arch": "sm_70", "max_threads_per_block": 1024, "max_num_threads": 1024, "kind": "cuda", "max_shared_memory_per_block": 49152, "tag": "", "keys": ["cuda", "gpu"]}
[2024-01-07 17:08:34] INFO auto_target.py:94: Found host LLVM triple: x86_64-conda-linux-gnu
[2024-01-07 17:08:34] INFO auto_target.py:95: Found host LLVM CPU: skylake-avx512
[2024-01-07 17:08:34] INFO auto_target.py:242: Generating code for CUDA architecture: sm_70
[2024-01-07 17:08:34] INFO auto_target.py:243: To produce multi-arch fatbin, set environment variable MLC_MULTI_ARCH. Example: MLC_MULTI_ARCH=70,72,75,80,86,87,89,90
[2024-01-07 17:08:34] INFO auto_config.py:151: Found model type: llama. Use `--model-type` to override.
[2024-01-07 17:08:34] WARNING compiler_flags.py:67: flashinfer is not supported on CUDA arch < 80
Compiling with arguments:
  --config          LlamaConfig(hidden_size=4096, intermediate_size=11008, num_attention_heads=32, num_hidden_layers=32, rms_norm_eps=1e-06, vocab_size=32000, position_embedding_base=10000, context_window_size=4096, prefill_chunk_size=4096, num_key_value_heads=32, head_dim=128, tensor_parallel_shards=1, max_batch_size=1, kwargs={})
  --quantization    GroupQuantize(name='q4f16_1', kind='group-quant', group_size=32, quantize_dtype='int4', storage_dtype='uint32', model_dtype='float16', num_elem_per_storage=8, num_storage_per_group=4, max_int_value=7)
  --model-type      llama
  --target          {"thread_warp_size": 32, "host": {"mtriple": "x86_64-conda-linux-gnu", "tag": "", "kind": "llvm", "mcpu": "skylake-avx512", "keys": ["cpu"]}, "arch": "sm_70", "max_threads_per_block": 1024, "max_num_threads": 1024, "kind": "cuda", "max_shared_memory_per_block": 49152, "tag": "", "keys": ["cuda", "gpu"]}
  --opt             flashinfer=0;cublas_gemm=0;cudagraph=0
  --system-lib-prefix ""
  --output          /tmp/tmpqjedea0v/lib.so
  --overrides       context_window_size=1024;sliding_window_size=None;prefill_chunk_size=4096;attention_sink_size=None;max_batch_size=None;tensor_parallel_shards=1
[2024-01-07 17:08:34] INFO compiler_flags.py:118: Overriding context_window_size from 4096 to 1024
[2024-01-07 17:08:34] INFO compiler_flags.py:118: Overriding prefill_chunk_size from 4096 to 4096
[2024-01-07 17:08:34] INFO compiler_flags.py:118: Overriding tensor_parallel_shards from 1 to 1
[2024-01-07 17:08:34] INFO llama_model.py:79: Overriding prefill_chunk_size from 4096 to 1024 (context_window_size)
[2024-01-07 17:08:34] INFO compile.py:131: Creating model from: LlamaConfig(hidden_size=4096, intermediate_size=11008, num_attention_heads=32, num_hidden_layers=32, rms_norm_eps=1e-06, vocab_size=32000, position_embedding_base=10000, context_window_size=4096, prefill_chunk_size=4096, num_key_value_heads=32, head_dim=128, tensor_parallel_shards=1, max_batch_size=1, kwargs={})
[2024-01-07 17:08:34] INFO compile.py:141: Exporting the model to TVM Unity compiler
[2024-01-07 17:08:39] INFO compile.py:147: Running optimizations using TVM Unity
[2024-01-07 17:08:39] INFO compile.py:160: Registering metadata: {'model_type': 'llama', 'quantization': 'q4f16_1', 'context_window_size': 1024, 'sliding_window_size': -1, 'attention_sink_size': -1, 'prefill_chunk_size': 1024, 'tensor_parallel_shards': 1, 'kv_cache_bytes': 536870912}
[2024-01-07 17:08:39] INFO pipeline.py:35: Running TVM Relax graph-level optimizations
[2024-01-07 17:08:42] INFO pipeline.py:35: Lowering to TVM TIR kernels
[2024-01-07 17:08:48] INFO pipeline.py:35: Running TVM TIR-level optimizations
[2024-01-07 17:08:57] INFO pipeline.py:35: Running TVM Dlight low-level optimizations
[2024-01-07 17:09:09] INFO pipeline.py:35: Lowering to VM bytecode
[2024-01-07 17:09:10] INFO estimate_memory_usage.py:55: [Memory usage] Function `_initialize_effect`: 0.00 MB
[2024-01-07 17:09:10] INFO estimate_memory_usage.py:55: [Memory usage] Function `batch_decode`: 0.09 MB
[2024-01-07 17:09:10] INFO estimate_memory_usage.py:55: [Memory usage] Function `batch_prefill`: 88.51 MB
[2024-01-07 17:09:10] INFO estimate_memory_usage.py:55: [Memory usage] Function `create_flashinfer_paged_kv_cache`: 0.00 MB
[2024-01-07 17:09:10] INFO estimate_memory_usage.py:55: [Memory usage] Function `decode`: 8.36 MB
[2024-01-07 17:09:10] INFO estimate_memory_usage.py:55: [Memory usage] Function `embed`: 8.00 MB
[2024-01-07 17:09:10] INFO estimate_memory_usage.py:55: [Memory usage] Function `prefill`: 245.63 MB
[2024-01-07 17:09:10] INFO estimate_memory_usage.py:55: [Memory usage] Function `softmax_with_temperature`: 0.12 MB
[2024-01-07 17:09:11] INFO pipeline.py:35: Compiling external modules
[2024-01-07 17:09:11] INFO pipeline.py:35: Compilation complete! Exporting to disk
[2024-01-07 17:09:22] INFO compile.py:175: Generated: /tmp/tmpqjedea0v/lib.so
[2024-01-07 17:09:22] INFO jit.py:87: Using compiled model lib: /iothome/chenf/.cache/mlc_chat/model_lib/beca1ee3070ef1b4cd8bbddad9a1c09d.so
[2024-01-07 17:09:23] ERROR model_metadata.py:93: FAILED to read metadata section in legacy model lib.
Traceback (most recent call last):
  File "/mnt/ssd/chenf/opensource/mlc-llm/python/mlc_chat/cli/model_metadata.py", line 91, in main
    metadata = _extract_metadata(parsed.model_lib)
  File "/mnt/ssd/chenf/opensource/mlc-llm/python/mlc_chat/cli/model_metadata.py", line 24, in _extract_metadata
    return json.loads(VirtualMachine(load_module(model_lib), device("cpu"))["_metadata"]())
  File "/mnt/ssd/chenf/opensource/tvm/python/tvm/runtime/relax_vm.py", line 97, in __init__
    self._setup_device(device, memory_cfg)
  File "/mnt/ssd/chenf/opensource/tvm/python/tvm/runtime/relax_vm.py", line 133, in _setup_device
    self.module["vm_initialization"](*init_args)
  File "/mnt/ssd/chenf/opensource/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 239, in __call__
    raise_last_ffi_error()
  File "/mnt/ssd/chenf/opensource/tvm/python/tvm/_ffi/base.py", line 481, in raise_last_ffi_error
    raise py_err
  File "/mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc", line 839, in tvm::runtime::relax_vm::VirtualMachineImpl::_Init(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
    this->Init(devices, alloc_types);
  File "/mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc", line 462, in tvm::runtime::relax_vm::VirtualMachineImpl::Init(std::vector<DLDevice, std::allocator<DLDevice> > const&, std::vector<tvm::runtime::memory::AllocatorType, std::allocator<tvm::runtime::memory::AllocatorType> > const&)
    this->InitFuncPool();
  File "/mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc", line 676, in tvm::runtime::relax_vm::VirtualMachineImpl::InitFuncPool()
    ICHECK(func.defined())
tvm.error.InternalError: Traceback (most recent call last):
  2: tvm::runtime::relax_vm::VirtualMachineImpl::_Init(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
        at /mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc:839
  1: tvm::runtime::relax_vm::VirtualMachineImpl::Init(std::vector<DLDevice, std::allocator<DLDevice> > const&, std::vector<tvm::runtime::memory::AllocatorType, std::allocator<tvm::runtime::memory::AllocatorType> > const&)
        at /mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc:462
  0: tvm::runtime::relax_vm::VirtualMachineImpl::InitFuncPool()
        at /mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc:676
  File "/mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc", line 676
InternalError: Check failed: (func.defined()) is false: Error: Cannot find PackedFunc paged_kv_cache.attention_kernel_prefill in either Relax VM kernel library, or in TVM runtime PackedFunc registry, or in global Relax functions of the VM executable
Traceback (most recent call last):
  File "/mnt/ssd/chenf/project/mlc_models/example/llama2/run.py", line 19, in <module>
    main()
  File "/mnt/ssd/chenf/project/mlc_models/example/llama2/run.py", line 9, in main
    cm = ChatModule(
  File "/mnt/ssd/chenf/opensource/mlc-llm/python/mlc_chat/chat_module.py", line 774, in __init__
    self._reload(self.model_lib_path, self.model_path, user_chat_config_json_str)
  File "/mnt/ssd/chenf/opensource/mlc-llm/python/mlc_chat/chat_module.py", line 988, in _reload
    self._reload_func(lib, model_path, app_config_json)
  File "/mnt/ssd/chenf/opensource/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 239, in __call__
    raise_last_ffi_error()
  File "/mnt/ssd/chenf/opensource/tvm/python/tvm/_ffi/base.py", line 481, in raise_last_ffi_error
    raise py_err
  File "/mnt/ssd/chenf/opensource/mlc-llm/cpp/llm_chat.cc", line 1541, in mlc::llm::LLMChatModule::GetFunction(tvm::runtime::String const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
    chat_->Reload(args[0], args[1], args[2]);
  File "/mnt/ssd/chenf/opensource/mlc-llm/cpp/llm_chat.cc", line 557, in mlc::llm::LLMChat::Reload(tvm::runtime::TVMArgValue, tvm::runtime::String, tvm::runtime::String)
    this->ft_.Init(reload_lib, device_, this->num_shards_);
  File "/mnt/ssd/chenf/opensource/mlc-llm/cpp/llm_chat.cc", line 160, in Init
    this->local_vm->GetFunction("vm_initialization")(
  File "/mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc", line 839, in tvm::runtime::relax_vm::VirtualMachineImpl::_Init(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
    this->Init(devices, alloc_types);
  File "/mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc", line 462, in tvm::runtime::relax_vm::VirtualMachineImpl::Init(std::vector<DLDevice, std::allocator<DLDevice> > const&, std::vector<tvm::runtime::memory::AllocatorType, std::allocator<tvm::runtime::memory::AllocatorType> > const&)
    this->InitFuncPool();
  File "/mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc", line 676, in tvm::runtime::relax_vm::VirtualMachineImpl::InitFuncPool()
    ICHECK(func.defined())
tvm.error.InternalError: Traceback (most recent call last):
  5: mlc::llm::LLMChatModule::GetFunction(tvm::runtime::String const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
        at /mnt/ssd/chenf/opensource/mlc-llm/cpp/llm_chat.cc:1541
  4: mlc::llm::LLMChat::Reload(tvm::runtime::TVMArgValue, tvm::runtime::String, tvm::runtime::String)
        at /mnt/ssd/chenf/opensource/mlc-llm/cpp/llm_chat.cc:557
  3: Init
        at /mnt/ssd/chenf/opensource/mlc-llm/cpp/llm_chat.cc:160
  2: tvm::runtime::relax_vm::VirtualMachineImpl::_Init(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
        at /mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc:839
  1: tvm::runtime::relax_vm::VirtualMachineImpl::Init(std::vector<DLDevice, std::allocator<DLDevice> > const&, std::vector<tvm::runtime::memory::AllocatorType, std::allocator<tvm::runtime::memory::AllocatorType> > const&)
        at /mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc:462
  0: tvm::runtime::relax_vm::VirtualMachineImpl::InitFuncPool()
        at /mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc:676
  File "/mnt/ssd/chenf/opensource/tvm/src/runtime/relax_vm/vm.cc", line 676
InternalError: Check failed: (func.defined()) is false: Error: Cannot find PackedFunc paged_kv_cache.attention_kernel_prefill in either Relax VM kernel library, or in TVM runtime PackedFunc registry, or in global Relax functions of the VM executable

[MasterJH5574]

Thanks @SunCrazy for sharing the stack trace. Figured out the reason and am gonna work on a fix. Will report back after fixing.

[junrushao]

Potential fix: #1555

[BaeBae33]

Potential fix: #1555

still crash

[junrushao]

The ROCm backend is broken and I'm able to confirm with a single-line command below:

mlc_chat compile llama2_7b --opt O0 --device rocm:0 --output /tmp/lib.so --quantization q4f16_1

I submitted one of the fixes, which handles MoE operators (group GEMM) properly in our LLVM-ROCm backend: apache/tvm#16403.

Note it is still broken with failed LLVM verification, and we need a extra few changes to get it back:

  File "/home/junrushao/Projects/tvm-dev/src/target/llvm/codegen_llvm.cc", line 354
TVMError: LLVM module verification failed with the following errors:
Intrinsic has incorrect argument type!
ptr @llvm.amdgcn.ds.bpermute
Intrinsic has incorrect argument type!
ptr @llvm.amdgcn.ds.bpermute
Intrinsic has incorrect argument type!
ptr @llvm.amdgcn.ds.bpermute
Intrinsic has incorrect argument type!
ptr @llvm.amdgcn.ds.bpermute
Intrinsic has incorrect argument type!
ptr @llvm.amdgcn.ds.bpermute
Intrinsic has incorrect argument type!
ptr @llvm.amdgcn.ds.bpermute
Call parameter type does not match function signature!
float 1.000000e+04
 <4 x float>  %46 = call <4 x float> @__ocml_pow_f32(float 1.000000e+04, float %45)
Both operands to a binary operator are not of the same type!
  %47 = fdiv float %41, <4 x float> %46
Call parameter type does not match function signature!
  %47 = fdiv float %41, <4 x float> %46
 <4 x float>  %48 = call <4 x float> @__ocml_sin_f32(float %47)
fptrunc source and destination must both be a vector or neither
  %49 = fptrunc <4 x float> %48 to half
Call parameter type does not match function signature!
float 1.000000e+04
 <4 x float>  %283 = call <4 x float> @__ocml_pow_f32(float 1.000000e+04, float %282)
Both operands to a binary operator are not of the same type!
  %284 = fdiv float %278, <4 x float> %283
Call parameter type does not match function signature!
  %284 = fdiv float %278, <4 x float> %283
 <4 x float>  %285 = call <4 x float> @__ocml_cos_f32(float %284)
fptrunc source and destination must both be a vector or neither
  %286 = fptrunc <4 x float> %285 to half
Call parameter type does not match function signature!
float 1.000000e+04
 <4 x float>  %391 = call <4 x float> @__ocml_pow_f32(float 1.000000e+04, float %390)
Both operands to a binary operator are not of the same type!
  %392 = fdiv float %386, <4 x float> %391
Call parameter type does not match function signature!
  %392 = fdiv float %386, <4 x float> %391
 <4 x float>  %393 = call <4 x float> @__ocml_sin_f32(float %392)
fptrunc source and destination must both be a vector or neither
  %394 = fptrunc <4 x float> %393 to half
Call parameter type does not match function signature!
float 1.000000e+04
 <4 x float>  %748 = call <4 x float> @__ocml_pow_f32(float 1.000000e+04, float %747)
Both operands to a binary operator are not of the same type!
  %749 = fdiv float %743, <4 x float> %748
Call parameter type does not match function signature!
  %749 = fdiv float %743, <4 x float> %748
 <4 x float>  %750 = call <4 x float> @__ocml_cos_f32(float %749)
fptrunc source and destination must both be a vector or neither
  %751 = fptrunc <4 x float> %750 to half

[junrushao]

CC: @tqchen @spectrometerHBH

[zhongzhenluo]

Hi I got same error with vulkan backend

[junrushao]

Right. Vulkan/AMD are sharing the same LLVM layer, and thus they are broken at the same time.

[junrushao]

Confirmed that this PR fixes the ROCm issue, while @spectrometerHBH is working on a fix for Vulkan rght now. I'm cherry-picking this change to mlc-ai/relax so that it goes to our nightlies starting tonight

[junrushao]

Starting the next nightly (tomorrow), the following command should be recovered on ROCm:

mlc_chat chat HF://junrushao/Llama-2-7b-chat-hf-q4f16_1

[zhongzhenluo]

Thanks. I will looking forward for vulkan update

[spectrometerHBH]

Thanks. I will looking forward for vulkan update

Hi, some Vulkan fixes here: apache/tvm#16405. cc if it works for you.

[junrushao]

@zhongzhenluo Vulkan target is fixed by: mlc-ai/relax@9c8cf08