Server cannot run with 3080 Ti

[fanglioc]

While trying to deploy the server with:
python -m ochat.serving.openai_api_server --model openchat/openchat_3.5,

I got the following error:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 224.00 MiB. GPU 0 has a total capacty of 11.74 GiB of which 140.00 MiB is free. Including non-PyTorch memory, this process has 11.38 GiB memory in use. Of the allocated memory 10.94 GiB is allocated by PyTorch, and 9.49 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Do i need to somehow specify the quantization?

Thanks!

[mfortini]

Same thing here with a 4060 and 8GB VRAM

[sffranke]

Same here with GTX 1060 6 GB / ubuntu 22.04

[Altenrion]

Same with Tesla T4 16 GB / ubuntu 22.04
Any updates from anyone?

[Hi-Angel]

@imoneoi given online demo has been broken for a while, any chance this problem could get some attention, so people could run the server locally instead?