GPU not working when using GPU docker image #3958
Unanswered
ll3N1GmAll
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
LocalAI version:
docker image: localai/localai:latest-aio-gpu-nvidia-cuda-12
Environment, CPU architecture, OS, and Version:
Hardware: bare metal, 16GB RAM, i3 3.2GHZ CPU, 8 Nvidia GPUs
Describe the bug
When trying to select a model from the drop down, my browser's "save password" box pops up with this in the password saved field "[object HTMLFormElement]" and no model stays populated in the drop down no matter which model I select. All image generation operations and chat prompts are painfully slow due to them not using GPUs as the docker image is supposed to. The nvidia container toolkit is installed and did not resolve this issue. Trying prior to this toolkit install and afterwards was tested, still GPUs are not being used. CPU spikes and the responses take ~30 minutes.
Some image generation tasks fail with error: "failed to load model with internal loader: grpc service not ready" - this happened with dreamshaper model
stablediffusion-cpp appears to work, slowly on CPU only
stablediffusion fails with "rpc error: code = Unavailable desc = error reading from server: EOF"
To Reproduce
Trying to generate images with any models (i have tried 3 models). All fail to use GPUs to respond to any text prompts. Chats run very slowly as well. I have tried 2 different browsers. The default FF based browser in ubuntu has the "save password" box pop-up issue. The LibreWolf FF based browser I tried also fails; but without the save password box pop-up.
Expected behavior
When selecting an image generation model from the drop-down, I expect it to work once selected without trying to save credentials to my browser's credential vault. I then expect to be able to interact with said model to generate images via the GPUs. I also expect the GPU docker container to make use of the available GPUs for chat prompts as well. I also expect image generation to work.
Logs
dreamshaper:
This was the log info that corresponded with the error above: "failed to load model with internal loader: grpc service not ready"
6:15AM INF Loading model 'dreamshaper' with backend diffusers
6:16AM ERR failed starting/connecting to the gRPC service error="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:36299: connect: connection refused""
6:16AM ERR Server error error="failed to load model with internal loader: grpc service not ready"
stablediffusion:
7:29AM ERR Server error error="rpc error: code = Unavailable desc = error reading from server: EOF"
Additional context
GPUs are used successfully by Ollama, AnythingLLM, LMStudio, & GPT4All.
Beta Was this translation helpful? Give feedback.
All reactions