Misc. bug: ALL gguf models fail to run (no log, docker exit code 139),

[orchidObsessed]

Name and Version

Docker:

ghcr.io/ggml-org/llama.cpp:server (b8419)

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

docker run -p 8080:8080 -v /models:/models ghcr.io/ggml-org/llama.cpp:server -m /models/mistral-7b-instruct-v0.1.Q5_K_M.gguf -c 512 --host 0.0.0.0 --port 8080

Problem description & steps to reproduce

Whether using the full, light, or server image, both latest and some previous ones, ALL gguf models fail to run (no log, docker exit code 139), either when using server or providing a prompt / interactive mode. When building a custom image that clones and compiles llama-cpp, all models run successfully.

Sample failing docker compose:

services:
  llama-server:
    image: ghcr.io/ggml-org/llama.cpp:server
    container_name: llm-mistral
    environment:
      # LLAMA_ARG_MODEL: /models/mistral-7b-instruct-v0.1.Q5_K_M.gguf
      LLAMA_ARG_MODEL_URL: "https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGUF/resolve/main/mistral-7b-instruct-v0.1.Q5_K_M.gguf"
      LLAMA_ARG_PORT: 5000
      LLAMA_ARG_HOST: 0.0.0.0
      LLAMA_ARG_NO_WEBUI: True
      LLAMA_ARG_CTX_SIZE: 0
      LLAMA_ARG_NO_MMAP: True
    volumes:
      - /models:/models
    ports:
      - "5000:5000"
    restart: unless-stopped

First Bad Commit

No response

Relevant log output

[fakeboboliu]

same issue.

the model loading seems fine and it shows "HTTP server is listening", but it dies immediately.

---

Update:

It is now working with the following patch. It seems that the default CPU configuration was not aware that someone might be poor enough to run it on a 15-year-old CPU that does not support AVX.

I'm building it remotely and transferring it to my server, thus the build configuration should be correctly set.
You may found no need to change this while building locally on the server itself.

diff --git a/ggml/src/CMakeLists.txt b/ggml/src/CMakeLists.txt
index 52817510..7001af19 100644
--- a/ggml/src/CMakeLists.txt
+++ b/ggml/src/CMakeLists.txt
@@ -308,15 +308,7 @@ if (GGML_CPU_ALL_VARIANTS)
     if (NOT GGML_BACKEND_DL)
         message(FATAL_ERROR "GGML_CPU_ALL_VARIANTS requires GGML_BACKEND_DL")
     endif()
-    ggml_add_cpu_backend_variant(sandybridge    AVX)
-    ggml_add_cpu_backend_variant(haswell        AVX F16C AVX2 BMI2 FMA)
-    ggml_add_cpu_backend_variant(skylakex       AVX F16C AVX2 BMI2 FMA AVX512)
-    ggml_add_cpu_backend_variant(icelake        AVX F16C AVX2 BMI2 FMA AVX512 AVX512_VBMI AVX512_VNNI)
-    ggml_add_cpu_backend_variant(alderlake      AVX F16C AVX2 BMI2 FMA AVX_VNNI)
-    if (NOT MSVC)
-        # MSVC doesn't support AMX
-        ggml_add_cpu_backend_variant(sapphirerapids AVX F16C AVX2 BMI2 FMA AVX512 AVX512_VBMI AVX512_VNNI AVX512_BF16 AMX_TILE AMX_INT8)
-    endif()
+    ggml_add_cpu_backend_variant(nothing)
 elseif (GGML_CPU)
     ggml_add_cpu_backend_variant_impl("")
 endif()

[orchidObsessed]

Awesome - I'll give this a try shortly and let you know if it works for me. For reference, for anyone who runs into this, I was running under a VM backed by a Broadwell-family CPU (Intel Xeon E5-2680 v4).

Additionally, just in case, this was my fairly light Dockerfile to work around the bug in the meantime - forgive the probably suboptimal file, I am by no means a Docker export 😅:

FROM debian:12.9

# Specify image
ENV MODEL=none
ENV PORT=4100

# Get the things we'll need
RUN apt update && apt install \
    build-essential \
    cmake \
    curl \
    git \
    python3 -y

# Create and change into the workdir
RUN mkdir /app && mkdir /models
WORKDIR /app

# Install llama-cpp
RUN git clone --depth=1 https://github.com/ggerganov/llama.cpp
WORKDIR /app/llama.cpp
RUN cmake -B build
RUN cmake --build build --config Release

# Chdir and run
WORKDIR /app/llama.cpp/build/bin
CMD ["sh", "-c", "./llama-server --model /models/${MODEL} --port ${PORT} --host 0.0.0.0"]

[fakeboboliu]

I was running under a VM backed by a Broadwell-family CPU (Intel Xeon E5-2680 v4).

Just cat /proc/cpuinfo, you may find no avx in Features:, that's what caused my issue.

BTW, I'm running in a VM with E7-4820 , a CPU has no avx in the first place.

[orchidObsessed]

It seems like our issues might be a little different. Your CPU doesn't support AVX but mine does, per Intel's documentation - I'm not sure if I'm allowed to link to documentation so I'm not providing links, but the specifications page for the E7-4820 does not list AVX, but the E5-2680 does.

Probably because of that, the CMAKE patch does not fix the issue for me. I still have to build from source using the Dockerfile above to get any models to load in a Docker container, otherwise I get a 139.

[Tianyue-Zhao]

I hit a similar issue where llama.cpp crashes in the ggml_backend_get_default_buffer_type function with a segmentation fault.
This function appears very simple and probably shouldn't be crashing.
I then tried running the same model on a computer with a new CPU and it worked there.
This makes me think that the crash is related to using an older CPU, which is on Broadwell architecture.
I wasn't able to try the patch that was posted above because it requires the GGML_CPU_ALL_VARIANTS flag, which requires the GGML_BACKEND_DL flag, which conflicts with the llama-llava-cli target that I was using.
Posting this to provide a little information that might be relevant.

[fakeboboliu]

@Tianyue-Zhao It should be helpful to build llama-cpp on the machine that would run it. Your native compiler will avoid using any instructions that aren't supported.

[Tianyue-Zhao]

Yes, I was building llama-cpp on each computer that I then ran it on.

However, I now realize that my issue is probably not due to CPU architecture, sorry about my previous comment.
I know this now because I created a new VM equipped with a very new CPU, but it still segfaults in the same way after building.

It does however work on my local computer.

[Tianyue-Zhao]

I eventually found that my issue was caused by #12907.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.