[Bug] InternalError: Check failed: (export_dso) is false: `export_dso` should be enabled for DSOExportable modules

[Matthieu-Tinycoaching]

Hello,

While running the tutorial_extensions_to_more_model_variants.ipynb notebook on google colab:
lib_path, model_path, chat_config_path = mlc_llm.build_model(build_args)

I got the following error message:

Weights exist at dist/models/vicuna-7b-v1.5, skipping download.
Using path "dist/models/vicuna-7b-v1.5" for model "vicuna-7b-v1.5"
Target configured: cuda -keys=cuda,gpu -arch=sm_75 -max_num_threads=1024 -max_shared_memory_per_block=49152 -max_threads_per_block=1024 -registers_per_block=65536 -thread_warp_size=32
Automatically using target for weight quantization: cuda -keys=cuda,gpu -arch=sm_75 -max_num_threads=1024 -max_shared_memory_per_block=49152 -max_threads_per_block=1024 -registers_per_block=65536 -thread_warp_size=32
Start computing and quantizing weights... This may take a while.
Finish computing and quantizing weights.
Total param size: 3.5313796997070312 GB
Start storing to cache dist/vicuna-7b-v1.5-q4f16_1/params
[0327/0327] saving param_326
All finished, 115 total shards committed, record saved to dist/vicuna-7b-v1.5-q4f16_1/params/ndarray-cache.json
Finish exporting chat config to dist/vicuna-7b-v1.5-q4f16_1/params/mlc-chat-config.json
---------------------------------------------------------------------------
InternalError                             Traceback (most recent call last)
[<ipython-input-9-b2f1f1fc4927>](https://localhost:8080/#) in <cell line: 1>()
----> 1 lib_path, model_path, chat_config_path = mlc_llm.build_model(build_args)

4 frames
[/usr/local/lib/python3.11/site-packages/tvm/_ffi/_ctypes/packed_func.py](https://localhost:8080/#) in __call__(self, *args)
    236             != 0
    237         ):
--> 238             raise get_last_ffi_error()
    239         _ = temp_args
    240         _ = args

InternalError: Traceback (most recent call last):
  9: TVMFuncCall
  8: _ZN3tvm7runtime13PackedFun
  7: tvm::runtime::TypedPackedFunc<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (tvm::runtime::ObjectRef const&)>::AssignTypedLambda<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(tvm::runtime::ObjectRef const&)>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(tvm::runtime::ObjectRef const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*) const
  6: tvm::SaveJSON[abi:cxx11](tvm::runtime::ObjectRef const&)
  5: tvm::JSONGraph::Create(tvm::runtime::ObjectRef const&)
  4: tvm::JSONAttrGetter::Get(tvm::runtime::Object*)
  3: tvm::__make_reflection4::{lambda(tvm::runtime::Object const*)#2}::_FUN(tvm::runtime::Object const)
  2: tvm::codegen::SerializeModuleToBytes[abi:cxx11](tvm::runtime::Module const&, bool)
  1: tvm::codegen::ModuleSerializer::SerializeModuleToBytes(dmlc::Stream*, bool)
  0: _ZN3tvm7runtime6deta
  File "/workspace/tvm/src/target/codegen.cc", line 87
InternalError: Check failed: (export_dso) is false: `export_dso` should be enabled for DSOExportable modules

Does anyone know how to deal with it?

[junrushao]

As a temporary workaround, use “-use-cache 0” and we have a bug fix coming up: apache/tvm#15693

[varshith15]

@junrushao "--use-cache 0" isn't working either

[Matthieu-Tinycoaching]

Hi,

Running this directly on my local computer with RTX 3090, I got the same error.

python3 -m mlc_llm.build --hf-path lmsys/vicuna-7b-v1.5 --target cuda --quantization q4f16_1

Returned me this message error:

Updated git hooks.
Git LFS initialized.
Clonage dans 'dist/models/vicuna-7b-v1.5'...
remote: Enumerating objects: 42, done.
remote: Total 42 (delta 0), reused 0 (delta 0), pack-reused 42
Dépaquetage des objets: 100% (42/42), 6.77 Kio | 693.00 Kio/s, fait.
Filtrage du contenu: 100% (3/3), 4.55 Gio | 1.98 Mio/s, fait.
Encountered 1 file(s) that may not have been copied correctly on Windows:
	pytorch_model-00001-of-00002.bin

See: `git lfs help smudge` for more details.
Downloaded weights to dist/models/vicuna-7b-v1.5
Using path "dist/models/vicuna-7b-v1.5" for model "vicuna-7b-v1.5"
Target configured: cuda -keys=cuda,gpu -arch=sm_86 -max_num_threads=1024 -max_shared_memory_per_block=49152 -max_threads_per_block=1024 -registers_per_block=65536 -thread_warp_size=32
Automatically using target for weight quantization: cuda -keys=cuda,gpu -arch=sm_86 -max_num_threads=1024 -max_shared_memory_per_block=49152 -max_threads_per_block=1024 -registers_per_block=65536 -thread_warp_size=32
Start computing and quantizing weights... This may take a while.
Finish computing and quantizing weights.
Total param size: 3.5323562622070312 GB
Start storing to cache dist/vicuna-7b-v1.5-q4f16_1/params
[0327/0327] saving param_326
All finished, 115 total shards committed, record saved to dist/vicuna-7b-v1.5-q4f16_1/params/ndarray-cache.json
Finish exporting chat config to dist/vicuna-7b-v1.5-q4f16_1/params/mlc-chat-config.json
[15:43:54] /workspace/tvm/include/tvm/topi/transform.h:1126: Warning: Fast mode segfaults when there are out-of-bounds indices. Make sure input indices are in bound
[15:43:54] /workspace/tvm/include/tvm/topi/transform.h:1126: Warning: Fast mode segfaults when there are out-of-bounds indices. Make sure input indices are in bound
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/matthieu/Deployment/mlc-llm/mlc_llm/build.py", line 13, in <module>
    main()
  File "/home/matthieu/Deployment/mlc-llm/mlc_llm/build.py", line 10, in main
    core.build_model_from_args(parsed_args)
  File "/home/matthieu/Deployment/mlc-llm/mlc_llm/core.py", line 590, in build_model_from_args
    pickle.dump(mod, outfile)
  File "/home/matthieu/anaconda3/envs/mlc-llm/lib/python3.11/site-packages/tvm/runtime/object.py", line 88, in __reduce__
    return (_new_object, (cls,), self.__getstate__())
                                 ^^^^^^^^^^^^^^^^^^^
  File "/home/matthieu/anaconda3/envs/mlc-llm/lib/python3.11/site-packages/tvm/runtime/object.py", line 93, in __getstate__
    return {"handle": _ffi_node_api.SaveJSON(self)}
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "tvm/_ffi/_cython/./packed_func.pxi", line 331, in tvm._ffi._cy3.core.PackedFuncBase.__call__
  File "tvm/_ffi/_cython/./packed_func.pxi", line 262, in tvm._ffi._cy3.core.FuncCall
  File "tvm/_ffi/_cython/./packed_func.pxi", line 251, in tvm._ffi._cy3.core.FuncCall3
  File "tvm/_ffi/_cython/./base.pxi", line 181, in tvm._ffi._cy3.core.CHECK_CALL
tvm.error.InternalError: Traceback (most recent call last):
  9: TVMFuncCall
  8: _ZN3tvm7runtime13PackedFun
  7: tvm::runtime::TypedPackedFunc<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (tvm::runtime::ObjectRef const&)>::AssignTypedLambda<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(tvm::runtime::ObjectRef const&)>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(tvm::runtime::ObjectRef const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*) const
  6: tvm::SaveJSON[abi:cxx11](tvm::runtime::ObjectRef const&)
  5: tvm::JSONGraph::Create(tvm::runtime::ObjectRef const&)
  4: tvm::JSONAttrGetter::Get(tvm::runtime::Object*)
  3: tvm::__make_reflection4::{lambda(tvm::runtime::Object const*)#2}::_FUN(tvm::runtime::Object const)
  2: tvm::codegen::SerializeModuleToBytes[abi:cxx11](tvm::runtime::Module const&, bool)
  1: tvm::codegen::ModuleSerializer::SerializeModuleToBytes(dmlc::Stream*, bool)
  0: _ZN3tvm7runtime6deta
  File "/workspace/tvm/src/target/codegen.cc", line 87
InternalError: Check failed: (export_dso) is false: `export_dso` should be enabled for DSOExportable modules

Same error with --use-cache 0

[KiwiHana]

NV 3060
/mlc-llm$ python build.py --model /mnt/c/kiwi/llm/llama2/Llama-2-7b-chat-hf --target cuda --quantization q4f16_1 --artifact-path "./" --use-cache 0

Using path "/mnt/c/kiwi/llm/llama2/Llama-2-7b-chat-hf" for model "Llama-2-7b-chat-hf"
Target configured: cuda -keys=cuda,gpu -arch=sm_86 -max_num_threads=1024 -max_shared_memory_per_block=49152 -max_threads_per_block=1024 -registers_per_block=65536 -thread_warp_size=32
Automatically using target for weight quantization: cuda -keys=cuda,gpu -arch=sm_86 -max_num_threads=1024 -max_shared_memory_per_block=49152 -max_threads_per_block=1024 -registers_per_block=65536 -thread_warp_size=32
Start computing and quantizing weights... This may take a while.
Finish computing and quantizing weights.
Total param size: 3.5323562622070312 GB
Start storing to cache ./Llama-2-7b-chat-hf-q4f16_1/params
[0327/0327] saving param_326
All finished, 115 total shards committed, record saved to ./Llama-2-7b-chat-hf-q4f16_1/params/ndarray-cache.json
Finish exporting chat config to ./Llama-2-7b-chat-hf-q4f16_1/params/mlc-chat-config.json
[14:05:36] /workspace/tvm/include/tvm/topi/transform.h:1126: Warning: Fast mode segfaults when there are out-of-bounds indices. Make sure input indices are in bound
[14:05:36] /workspace/tvm/include/tvm/topi/transform.h:1126: Warning: Fast mode segfaults when there are out-of-bounds indices. Make sure input indices are in bound
Traceback (most recent call last):
  File "/home/kiwi/llm/mlc-llm/build.py", line 4, in <module>
    main()
  File "/home/kiwi/llm/mlc-llm/mlc_llm/build.py", line 10, in main
    core.build_model_from_args(parsed_args)
  File "/home/kiwi/llm/mlc-llm/mlc_llm/core.py", line 590, in build_model_from_args
    pickle.dump(mod, outfile)
  File "/home/kiwi/llm/llm-perf-bench/Y/envs/python311/lib/python3.11/site-packages/tvm/runtime/object.py", line 88, in __reduce__
    return (_new_object, (cls,), self.__getstate__())
                                 ^^^^^^^^^^^^^^^^^^^
  File "/home/kiwi/llm/llm-perf-bench/Y/envs/python311/lib/python3.11/site-packages/tvm/runtime/object.py", line 93, in __getstate__
    return {"handle": _ffi_node_api.SaveJSON(self)}
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "tvm/_ffi/_cython/./packed_func.pxi", line 331, in tvm._ffi._cy3.core.PackedFuncBase.__call__
  File "tvm/_ffi/_cython/./packed_func.pxi", line 262, in tvm._ffi._cy3.core.FuncCall
  File "tvm/_ffi/_cython/./packed_func.pxi", line 251, in tvm._ffi._cy3.core.FuncCall3
  File "tvm/_ffi/_cython/./base.pxi", line 181, in tvm._ffi._cy3.core.CHECK_CALL
tvm.error.InternalError: Traceback (most recent call last):
  9: TVMFuncCall
  8: _ZN3tvm7runtime13PackedFun
  7: tvm::runtime::TypedPackedFunc<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (tvm::runtime::ObjectRef const&)>::AssignTypedLambda<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(tvm::runtime::ObjectRef const&)>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(tvm::runtime::ObjectRef const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*) const
  6: tvm::SaveJSON[abi:cxx11](tvm::runtime::ObjectRef const&)
  5: tvm::JSONGraph::Create(tvm::runtime::ObjectRef const&)
  4: tvm::JSONAttrGetter::Get(tvm::runtime::Object*)
  3: tvm::__make_reflection4::{lambda(tvm::runtime::Object const*)#2}::_FUN(tvm::runtime::Object const)
  2: tvm::codegen::SerializeModuleToBytes[abi:cxx11](tvm::runtime::Module const&, bool)
  1: tvm::codegen::ModuleSerializer::SerializeModuleToBytes(dmlc::Stream*, bool)
  0: _ZN3tvm7runtime6deta
  File "/workspace/tvm/src/target/codegen.cc", line 87
InternalError: Check failed: (export_dso) is false: `export_dso` should be enabled for DSOExportable modules

[junrushao]

It is a known bug that has been fixed from upstream as I indicated in this thread: #883 (comment).

This fix has been merged to our nightly TVM Unity wheel tonight, so I would expect it to be gone tomorrow.

[KiwiHana]

Hi How to install nightly TVM Unity wheel tomorrow?

It is a known bug that has been fixed from upstream as I indicated in this thread: #883 (comment).

This fix has been merged to our nightly TVM Unity wheel tonight, so I would expect it to be gone tomorrow.

[KiwiHana]

NV 3060 /mlc-llm$ python build.py --model /mnt/c/kiwi/llm/llama2/Llama-2-7b-chat-hf --target cuda --quantization q4f16_1 --artifact-path "./" --use-cache 0

Using path "/mnt/c/kiwi/llm/llama2/Llama-2-7b-chat-hf" for model "Llama-2-7b-chat-hf"
Target configured: cuda -keys=cuda,gpu -arch=sm_86 -max_num_threads=1024 -max_shared_memory_per_block=49152 -max_threads_per_block=1024 -registers_per_block=65536 -thread_warp_size=32
Automatically using target for weight quantization: cuda -keys=cuda,gpu -arch=sm_86 -max_num_threads=1024 -max_shared_memory_per_block=49152 -max_threads_per_block=1024 -registers_per_block=65536 -thread_warp_size=32
Start computing and quantizing weights... This may take a while.
Finish computing and quantizing weights.
Total param size: 3.5323562622070312 GB
Start storing to cache ./Llama-2-7b-chat-hf-q4f16_1/params
[0327/0327] saving param_326
All finished, 115 total shards committed, record saved to ./Llama-2-7b-chat-hf-q4f16_1/params/ndarray-cache.json
Finish exporting chat config to ./Llama-2-7b-chat-hf-q4f16_1/params/mlc-chat-config.json
[14:05:36] /workspace/tvm/include/tvm/topi/transform.h:1126: Warning: Fast mode segfaults when there are out-of-bounds indices. Make sure input indices are in bound
[14:05:36] /workspace/tvm/include/tvm/topi/transform.h:1126: Warning: Fast mode segfaults when there are out-of-bounds indices. Make sure input indices are in bound
Traceback (most recent call last):
  File "/home/kiwi/llm/mlc-llm/build.py", line 4, in <module>
    main()
  File "/home/kiwi/llm/mlc-llm/mlc_llm/build.py", line 10, in main
    core.build_model_from_args(parsed_args)
  File "/home/kiwi/llm/mlc-llm/mlc_llm/core.py", line 590, in build_model_from_args
    pickle.dump(mod, outfile)
  File "/home/kiwi/llm/llm-perf-bench/Y/envs/python311/lib/python3.11/site-packages/tvm/runtime/object.py", line 88, in __reduce__
    return (_new_object, (cls,), self.__getstate__())
                                 ^^^^^^^^^^^^^^^^^^^
  File "/home/kiwi/llm/llm-perf-bench/Y/envs/python311/lib/python3.11/site-packages/tvm/runtime/object.py", line 93, in __getstate__
    return {"handle": _ffi_node_api.SaveJSON(self)}
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "tvm/_ffi/_cython/./packed_func.pxi", line 331, in tvm._ffi._cy3.core.PackedFuncBase.__call__
  File "tvm/_ffi/_cython/./packed_func.pxi", line 262, in tvm._ffi._cy3.core.FuncCall
  File "tvm/_ffi/_cython/./packed_func.pxi", line 251, in tvm._ffi._cy3.core.FuncCall3
  File "tvm/_ffi/_cython/./base.pxi", line 181, in tvm._ffi._cy3.core.CHECK_CALL
tvm.error.InternalError: Traceback (most recent call last):
  9: TVMFuncCall
  8: _ZN3tvm7runtime13PackedFun
  7: tvm::runtime::TypedPackedFunc<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (tvm::runtime::ObjectRef const&)>::AssignTypedLambda<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(tvm::runtime::ObjectRef const&)>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (*)(tvm::runtime::ObjectRef const&), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*) const
  6: tvm::SaveJSON[abi:cxx11](tvm::runtime::ObjectRef const&)
  5: tvm::JSONGraph::Create(tvm::runtime::ObjectRef const&)
  4: tvm::JSONAttrGetter::Get(tvm::runtime::Object*)
  3: tvm::__make_reflection4::{lambda(tvm::runtime::Object const*)#2}::_FUN(tvm::runtime::Object const)
  2: tvm::codegen::SerializeModuleToBytes[abi:cxx11](tvm::runtime::Module const&, bool)
  1: tvm::codegen::ModuleSerializer::SerializeModuleToBytes(dmlc::Stream*, bool)
  0: _ZN3tvm7runtime6deta
  File "/workspace/tvm/src/target/codegen.cc", line 87
InternalError: Check failed: (export_dso) is false: `export_dso` should be enabled for DSOExportable modules

I install nightly TVM Unity wheel by the following CLI, but still have the same error.
pip install --pre --upgrade --user typing-extensions psutil scipy
pip install --pre --upgrade tvm
Installing collected packages: typing-extensions
Successfully installed typing-extensions-4.8.0rc1

[Matthieu-Tinycoaching]

Any news for this bug?

[junrushao]

I believe it’s been fixed a while ago

[tqchen]

@CharlieFRuan @MasterJH5574 can you help confirming the latest states main? I believe the wheels should also been updated

[CharlieFRuan]

Hi @Matthieu-Tinycoaching, we updated the wheels again. I ran the notebook and there was no error observed.

[Matthieu-Tinycoaching]

Hi @CharlieFRuan is this updated in the new nightly pip packages?

Or how have I to install it?

[CharlieFRuan]

@Matthieu-Tinycoaching Following these steps would suffice!