Runing RedPajama get out of memory error

[logicalpath]

[logicalpath]

I run this command:

(mlc-chat) eddie@eddie-XPS-15-9560:~/workspace/mlc$ mlc_chat_cli --local-id RedPajama-INCITE-Chat-3B-v1-q4f16_0
Use MLC config: "dist/prebuilt/mlc-chat-RedPajama-INCITE-Chat-3B-v1-q4f16_0/mlc-chat-config.json"
Use model weights: "dist/prebuilt/mlc-chat-RedPajama-INCITE-Chat-3B-v1-q4f16_0/ndarray-cache.json"
Use model library: "dist/prebuilt/lib/RedPajama-INCITE-Chat-3B-v1-q4f16_0-vulkan.so"
Loading model...
You can use the following special commands:
/help print the special commands
/exit quit the cli
/stats print out the latest stats (token/sec)
/reset restart a fresh chat
/reload [local_id] reload model local_id from disk, or reload the current model if local_id is not specified

I get a prompt and enter text:

: Write a poem for San Francisco

Then I get an error:

[10:18:16] /home/runner/work/package/package/tvm/src/runtime/relax_vm/pooled_allocator.h:64: Warning: PooledAllocator got InternalError during allocation:

---

An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html

---

Check failed: (__e == VK_SUCCESS) is false: Vulkan Error, code=-2: VK_ERROR_OUT_OF_DEVICE_MEMORY
[10:18:16] /home/runner/work/package/package/tvm/src/runtime/relax_vm/pooled_allocator.h:65: Warning: Trying to release all unused memory and reallocate...
terminate called after throwing an instance of 'tvm::runtime::InternalError'
what(): [10:18:16] /home/runner/work/package/package/tvm/include/tvm/runtime/device_api.h:291: unknown type =0
Stack trace:
[bt] (0) /home/eddie/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(tvm::runtime::Backtraceabi:cxx11+0x27) [0x7f19af02e0c7]
[bt] (1) /home/eddie/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(+0x65a7a) [0x7f19aeff1a7a]
[bt] (2) /home/eddie/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(+0x4e809) [0x7f19aefda809]
[bt] (3) /home/eddie/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(+0xee9d4) [0x7f19af07a9d4]
[bt] (4) /home/eddie/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(+0x4c4de) [0x7f19aefd84de]
[bt] (5) /home/eddie/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::runtime::TypedPackedFunc<tvm::runtime::relax_vm::Storage (void*, tvm::runtime::ShapeTuple, long, DLDataType)>::AssignTypedLambda<tvm::runtime::relax_vm::Storage ()(void, tvm::runtime::ShapeTuple, long, DLDataType)>(tvm::runtime::relax_vm::Storage ()(void, tvm::runtime::ShapeTuple, long, DLDataType), std::__cxx11::basic_string<char, std::char_traits, std::allocator >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)+0x25e) [0x7f19af08c58e]
[bt] (6) /home/eddie/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(tvm::runtime::relax_vm::VirtualMachineImpl::RunInstrCall(tvm::runtime::relax_vm::VMFrame*, tvm::runtime::relax_vm::Instruction)+0xa11) [0x7f19af0c06d1]
[bt] (7) /home/eddie/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(tvm::runtime::relax_vm::VirtualMachineImpl::RunLoop()+0x2e7) [0x7f19af0bda87]
[bt] (8) /home/eddie/miniconda3/envs/mlc-chat/bin/../lib/libtvm_runtime.so(tvm::runtime::relax_vm::VirtualMachineImpl::InvokeBytecode(long, std::vector<tvm::runtime::TVMRetValue, std::allocatortvm::runtime::TVMRetValue > const&)+0x246) [0x7f19af0be096]

Aborted (core dumped)

I'm running Linux Mint 21.2 with this GPU:

(from Vulkaninfo):

GPU1:
VkPhysicalDeviceProperties:

apiVersion        = 4206816 (1.3.224)
driverVersion     = 2203731008 (0x835a4440)
vendorID          = 0x10de
deviceID          = 0x1c8d
deviceType        = PHYSICAL_DEVICE_TYPE_DISCRETE_GPU
deviceName        = NVIDIA GeForce GTX 1050
pipelineCacheUUID = dcbcd2de-e45b-e401-58e7-40a8bbb24597

VkPhysicalDeviceLimits:

maxImageDimension1D                             = 32768
maxImageDimension2D                             = 32768
maxImageDimension3D                             = 16384
maxImageDimensionCube                           = 32768
maxImageArrayLayers                             = 2048
maxTexelBufferElements                          = 134217728
maxUniformBufferRange                           = 65536
maxStorageBufferRange                           = 429nvii4967295
maxPushConstantsSize                            = 256
maxMemoryAllocationCount                        = 4294967295

nvidia-smi yields this output:

mlc-chat) eddie@eddie-XPS-15-9560:~/workspace/mlc$ nvidia-smi
Mon May 29 10:46:30 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17 Driver Version: 525.105.17 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A |
| N/A 45C P3 N/A / N/A | 1113MiB / 4096MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 987 G /usr/lib/xorg/Xorg 842MiB |
| 0 N/A N/A 1674 G cinnamon 33MiB |
| 0 N/A N/A 2112 G /usr/lib/firefox/firefox 235MiB |
+-----------------------------------------------------------------------------+

Maybe the CUDA version reference means that the Vulkan drivers are not being used?

Any help appreciated.
Thanks,
Eddie

[yzh119]

The GPU memory of GTX 1050 (4GB) might be too small to run RedPajama inference, especially when your graphics environment takes 1GB of GPU memory.

One suggestion is to disable X-server when you run mlc_chat_cli, by running the following command (I assume you are using GDM, another display manager such as LightDM should work similarly):

sudo systemctl stop gdm

After that, you can ctrl + alt + f2 to enter another tty, and if you run nvidia-smi in this case, the GPU Memory Usage should be 1MiB/4096MiB, and you can try running mlc_chat_cli again and see if you will still encounter the OOM issue.

After you have fun with MLC-LLM, you can restart your X-server by:

sudo systemctl start gdm

[logicalpath]

Thank you @yzh119, that worked great!
I did:
sudo systemctl stop lightdm then ctrl + alt + f4 ran mlc_chat_cli and the MLC-LLM wrote me a nice poem about San Frnacisco.

I typed 'ctrl + alt + ff7` to exit text mode. Thank you for all this work!