[Bug] Compiling RedPajama 3B model fails

[joechen24]

🐛 Bug

I am running
python3 -m mlc_llm.build --hf-path togethercomputer/RedPajama-INCITE-Chat-3B-v1 --target iphone --max-seq-len 768 --quantization q4f16_1 (follow https://llm.mlc.ai/docs/compilation/compile_models.html)

However, it failed with the following message:

Save a cached module to dist/RedPajama-INCITE-Chat-3B-v1-q4f16_1/mod_cache_before_build.pkl.
Compilation error:
/var/folders/j9/g70hnkws13b1lt5rylpnd0m40000gn/T/tmpguobvd47/my_lib.metal:145:8: error: redefinition of 'RedPajama_INCITE_Chat_3B_v1_q4f16_1_fused_fused_decode5_fused_NT_matmul10_add9_cast8_cast12_add7_kernel_args_t'
.........[Long error message].....
LLVM ERROR: Error opening '/var/folders/j9/g70hnkws13b1lt5rylpnd0m40000gn/T/tmpguobvd47/my_lib.air': No such file or directory!
Traceback (most recent call last):
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Users/xxx/mlc-llm/mlc_llm/build.py", line 46, in
main()
File "/Users/xxx/mlc-llm/mlc_llm/build.py", line 42, in main
core.build_model_from_args(parsed_args)
File "/Users/xxx/mlc-llm/mlc_llm/core.py", line 705, in build_model_from_args
build(mod, args)
File "/Users/xxx/mlc-llm/mlc_llm/core.py", line 597, in build
ex = relax.build(mod_deploy, args.target, system_lib=args.system_lib)
File "/Users/xxx/.venv/lib/python3.9/site-packages/tvm/relax/vm_build.py", line 343, in build
return _vmlink(builder, target, tir_mod, ext_libs, params, system_lib=system_lib)
File "/Users/xxx/.venv/lib/python3.9/site-packages/tvm/relax/vm_build.py", line 242, in _vmlink
lib = tvm.build(
File "/Users/xxx/.venv/lib/python3.9/site-packages/tvm/driver/build_module.py", line 281, in build
rt_mod_host = _driver_ffi.tir_to_runtime(annotated_mods, target_host)
File "tvm/_ffi/_cython/./packed_func.pxi", line 332, in tvm._ffi._cy3.core.PackedFuncBase.call
File "tvm/_ffi/_cython/./packed_func.pxi", line 263, in tvm._ffi._cy3.core.FuncCall
File "tvm/_ffi/_cython/./packed_func.pxi", line 252, in tvm._ffi._cy3.core.FuncCall3
File "tvm/_ffi/_cython/./base.pxi", line 182, in tvm._ffi._cy3.core.CHECK_CALL
File "/Users/xxx/.venv/lib/python3.9/site-packages/tvm/_ffi/base.py", line 476, in raise_last_ffi_error
raise py_err
tvm.error.InternalError: Traceback (most recent call last):
File "/Users/catalyst/Workspace/mlc-ai-package-self-runner/work/package/package/tvm/include/tvm/runtime/packed_func.h", line 836
InternalError: Check failed: type_code == kTVMStr (4 vs. 11) : expected str but got NULL

Environment

• Operating system (e.g. Ubuntu/Windows/MacOS/...): MacOS
• MLC LLM version: as of 10/31/2023, main @e0cd3f6
• TVM-Unity version: mlc-ai-nightly==0.12.dev1773

[MasterJH5574]

Hi @zhuochen24 thanks for reporting. I can reproduce it and pushed a hot fix mlc-ai/relax@3001b20.

If you are using an Apple Silicon Mac, you should now be able to update TVM Unity with

conda activate your-environment
python3 -m pip install --pre -U -f https://mlc.ai/wheels mlc-chat-nightly mlc-ai-nightly

If you are using an Intel Mac, please wait for a few hours, since the pip wheel for Intel Mac is still being built.

Please let me know how it goes, thanks :-)

[junrushao]

Also CC @Lunderberg would you mind taking a look? Seems relevant to your commit a while ago

[joechen24]

@MasterJH5574 Thanks a lot for the quick action! Unfortunately, it still doesn't work for me
I am using Apple Silicon Mac, and ran python3 -m pip install --pre -U -f https://mlc.ai/wheels mlc-ai-nightly.
Looking at https://mlc.ai/wheels, it seems mlc-ai-nightly hasn't been updated with your latest fix.
Should I wait a day for it to take effect or did the recent nightly build fail?
Thank you!

[MasterJH5574]

Ah my bad. The wheel was built but not linked to https://mlc.ai/wheels. Just triggered the process and it is now running. Will report back when it is done.

[MasterJH5574]

Okay I think it's up now. Would you mind trying pip install again?

[joechen24]

It worked! Thank you for the help @MasterJH5574 !

[tqchen]

going to keep it open until we fixed the metal regression on unity

[MasterJH5574]

going to keep it open until we fixed the metal regression on unity

Sent a formal fix here apache/tvm#16033. Need to bring it to unity branch before closing this issue.

[junrushao]

I believe this fix had been synced to our pip package, and this issue should be gone now