115 Illegal instruction Container exit - Solution in thread

[lishaojun616]

The anythingllm is installed in Ubuntu server.
In the system LLM set ,the system can connect to the Ollama server and get the models .
But when chat in workspace ,the docker is exited.
1.Show the info in browser:

2.and the docker logs:
"/usr/local/bin/docker-entrypoint.sh: line 7: 115 Illegal instruction (core dumped) node /app/server/index.js"

What's the problem?

[timothycarambat]

Docker engine - it appears
#1290 (comment)

That issue is specifically occurring on Mac, but it is the same on Linux/Ubuntu as well.

[SyuanYo]

Hi @lishaojun616 , have you managed to resolve the issue?
I am experiencing the same situation as you described in #1323 .

[joachimt-git]

Hello everyone,

I face the same problem.

Same setup, ubuntu 22.04 LTS, using ollama as llm.

I have installed the newest docker engine, build anything-llm with docker-compose-v2.

...
[TELEMETRY SENT] {
  event: 'workspace_created',
  distinctId: 'c060354a-b171-4702-b83a-9da2ef0612e4',
  properties: {
    multiUserMode: false,
    LLMSelection: 'ollama',
    Embedder: 'native',
    VectorDbSelection: 'lancedb',
    runtime: 'docker'
  }
}
[Event Logged] - workspace_created
[TELEMETRY SENT] {
  event: 'onboarding_complete',
  distinctId: 'c060354a-b171-4702-b83a-9da2ef0612e4',
  properties: { runtime: 'docker' }
}
[NativeEmbedder] Initialized
/usr/local/bin/docker-entrypoint.sh: line 7:   119 Illegal instruction     (core dumped) node /app/server/index.js
gitlab-runner@gradio:~$ docker --version
Docker version 26.1.3, build b72abbb

This issue is marked as closed. Is there a solution available?

Best regrads

Joachim

[xsn-cloud]

Encountering the same issue. Using Ubuntu Server 22.04 with Docker, Yarn, and Node installed as recommended in HOW_TO_USE_DOCKER.md#how-to-use-dockerized-anything-llm. Ollama is on another machine, serving at 0.0.0.0 (other remote apps function correctly with this setup, even in Docker).

EDIT: Forgot to mention:

Docker version 26.1.3, build b72abbb

Ubuntu is running in a VM

Experiencing the identical error as posted by @joachimt-git:

[Event Logged] - update_llm_provider
[Event Logged] - update_embedding_engine
[Event Logged] - update_vector_db
[TELEMETRY SENT] {
  event: 'enabled_multi_user_mode',
  distinctId: '20b022d1-14cc-490c-ab86-4f941a32f7bc',
  properties: { multiUserMode: true, runtime: 'docker' }
}
[Event Logged] - multi_user_mode_enabled
[TELEMETRY SENT] {
  event: 'login_event',
  distinctId: '20b022d1-14cc-490c-ab86-4f941a32f7bc::1',
  properties: { multiUserMode: false, runtime: 'docker' }
}
[Event Logged] - login_event
[TELEMETRY SENT] {
  event: 'workspace_created',
  distinctId: '20b022d1-14cc-490c-ab86-4f941a32f7bc::1',
  properties: {
    multiUserMode: true,
    LLMSelection: 'ollama',
    Embedder: 'native',
    VectorDbSelection: 'lancedb',
    runtime: 'docker'
  }
}
[Event Logged] - workspace_created
[TELEMETRY SENT] {
  event: 'onboarding_complete',
  distinctId: '20b022d1-14cc-490c-ab86-4f941a32f7bc',
  properties: { runtime: 'docker' }
}
[NativeEmbedder] Initialized
/usr/local/bin/docker-entrypoint.sh: line 7:   117 Illegal instruction     (core dumped) node /app/server/index.js

Any suggestions for resolving this?

How can I assist?

Thanks

[timothycarambat]

This is certainly a configuration issue. Considering all is well until the native embedder is called this might be arch related - but we support both ARM and x86. Regardless, here is my exact steps that fail to repro:

1. Obtain Ubuntu 22.04 LTS AWS instance - used t3.small - x86
2. curl -fsSL https://get.docker.com -o get-docker.sh
3. sudo sh get-docker.sh
4. sudo usermod -aG docker $USER
5. docker -v

Docker version 26.1.3, build b72abbb

6. docker pull mintplexlabs/anythingllm

Run:

export STORAGE_LOCATION=$HOME/anythingllm && \
mkdir -p $STORAGE_LOCATION && \
touch "$STORAGE_LOCATION/.env" && \
docker run -d -p 3001:3001 \
--cap-add SYS_ADMIN \
-v ${STORAGE_LOCATION}:/app/server/storage \
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
-e STORAGE_DIR="/app/server/storage" \
mintplexlabs/anythingllm

Access via instance IP on port 3001 - I get the interface, onboard, create workspace, and upload documents.

[Event Logged] - workspace_created
[TELEMETRY SENT] {
  event: 'onboarding_complete',
  distinctId: 'fe73dd18-d52e-4a4c-bb62-a6adba7491d1',
  properties: { runtime: 'docker' }
}
-- Working readme.pdf --
-- Parsing content from pg 1 --
-- Parsing content from pg 2 --
-- Parsing content from pg 3 --
-- Parsing content from pg 4 --
-- Parsing content from pg 5 --
[SUCCESS]: readme.pdf converted & ready for embedding.

[CollectorApi] Document readme.pdf uploaded processed and successfully. It is now available in documents.
[TELEMETRY SENT] {
  event: 'document_uploaded',
  distinctId: 'fe73dd18-d52e-4a4c-bb62-a6adba7491d1',
  properties: { runtime: 'docker' }
}
[Event Logged] - document_uploaded
Adding new vectorized document into namespace sample
[NativeEmbedder] Initialized
[RecursiveSplitter] Will split with { chunkSize: 1000, chunkOverlap: 20 }
Chunks created from document: 14
[NativeEmbedder] The native embedding model has never been run and will be downloaded right now. Subsequent runs will be faster. (~23MB)
[NativeEmbedder] Downloading Xenova/all-MiniLM-L6-v2 from https://huggingface.co/
....truncated
[NativeEmbedder - Downloading model] onnx/model_quantized.onnx 100%
[NativeEmbedder] Embedded Chunk 1 of 1
Inserting vectorized chunks into LanceDB collection.
Caching vectorized results of custom-documents/readme.pdf-d717ca8c-6ac0-4514-8d6d-94ac48760afe.json to prevent duplicated embedding.
[TELEMETRY SENT] {
  event: 'documents_embedded_in_workspace',
  distinctId: 'fe73dd18-d52e-4a4c-bb62-a6adba7491d1',
  properties: {
    LLMSelection: 'openai',
    Embedder: 'native',
    VectorDbSelection: 'lancedb',
    runtime: 'docker'
  }
}
[Event Logged] - workspace_documents_added

Considering all of this occurs on [NativeEmbedder] Initialized this to me would indicate a lack of resources to run the local embedder and if that is the case, you should allocate more resources to the container or use another embedder. That is the only way I could imagine a full core dump or illegal instruction occurring with Illegal instruction. Either that or the underlying chip arch is not found/supported for Xenova transformers.js.

[joachimt-git]

Hi Timothy,

I think what @xsn-cloud and I have in commun is that we both use ollama.

May that ne the cause of the failure?

Joachim

[timothycarambat]

It would not, since the exception is in the AnythingLLM container and if there was an illegal instruction in the Ollama program it would throw in that container/program. All AnythingLLM does is execute a fetch request to the Ollama instance, which would be permitted in any container

[xsn-cloud]

@timothycarambat Thanks for addressing this issue. Please let me know if there's anything I can assist you with.

I've conducted the following experiment, also considering that it might be an issue with Docker running on VMs and to verify resource issues:

UPDATE: Also tested it in Windows 10 (WSL, Docker for Windows, Docker version 26.1.1, build 4cf5afa): Same issue

• Clean install of Debian 12 on baremetal - Dual Xeon E5-2650 v2 @ 2.60GHz with 96GB of RAM
• Docker version 26.1.3, build b72abbb
• Followed the procedure exactly as you did in your previous comment (the same one you used on an Ubuntu 22.04 LTS AWS)
• No documents loaded
• During the onboarding, anything-llm successfully communicates with ollama, accurately retrieves the installed models, and allows the selection of the model without issues.
• Model selected: llama3 with 4K context window. Other models tested with same results.

This is the outcome after the onboarding when attempting to send a "hello" in a new chat. (docker logs -f [containerid]).

Please note that the container was restarted from scratch to ensure the clarity of the logs.

Collector hot directory and tmp storage wiped!
Document processor app listening on port 8888
Environment variables loaded from .env
Prisma schema loaded from prisma/schema.prisma

✔ Generated Prisma Client (v5.3.1) to ./node_modules/@prisma/client in 338ms

Start using Prisma Client in Node.js (See: https://pris.ly/d/client)

'''
import { PrismaClient } from '@prisma/client'
const prisma = new PrismaClient()
'''


or start using Prisma Client at the edge (See: https://pris.ly/d/accelerate)
'''
import { PrismaClient } from '@prisma/client/edge'
const prisma = new PrismaClient()
'''

See other ways of importing Prisma Client: http://pris.ly/d/importing-client

Environment variables loaded from .env
Prisma schema loaded from prisma/schema.prisma
Datasource "db": SQLite database "anythingllm.db" at "file:../storage/anythingllm.db"

20 migrations found in prisma/migrations


No pending migrations to apply.
┌─────────────────────────────────────────────────────────┐
│  Update available 5.3.1 -> 5.14.0                       │
│  Run the following to update                            │
│    npm i --save-dev prisma@latest                       │
│    npm i @prisma/client@latest                          │
└─────────────────────────────────────────────────────────┘
[TELEMETRY ENABLED] Anonymous Telemetry enabled. Telemetry helps Mintplex Labs Inc improve AnythingLLM.
prisma:info Starting a sqlite pool with 33 connections.
fatal: not a git repository (or any of the parent directories): .git
getGitVersion Command failed: git rev-parse HEAD
fatal: not a git repository (or any of the parent directories): .git

[TELEMETRY SENT] {
  event: 'server_boot',
  distinctId: '4f39e3fb-ac8c-4043-9586-c21ef46b0c47',
  properties: { commit: '--', runtime: 'docker' }
}
[CommunicationKey] RSA key pair generated for signed payloads within AnythingLLM services.
Primary server in HTTP mode listening on port 3001
[NativeEmbedder] Initialized
/usr/local/bin/docker-entrypoint.sh: line 7:   163 Illegal instruction     (core dumped) node /app/server/index.js

One more clarification: The error occurs after sending a message in the chatbox. Until then, the last message displayed is [NativeEmbedder] Initialized, and it remains unchanged until the message is sent.

Thanks a lot for your time.

[timothycarambat]

If you were to not use the native embedder, this problem would not surface. The only commonality between all of this is varying CPUs. Transformers.js which runs the native embedder, uses ONNX runtime and at this point the root cause has to be coming from there as this only occurs when using the native embedder and that is the supporting libraries to enable that functionality.

[jorgen-k]

I had the same issue running Docker in Ubuntu 24.04 VM on a Proxmox host. I switched the CPU in the guest to "host," and it fixed the problem. Just wanted to share in case anyone else is having the same struggle I did. Hope this helps!

[timothycarambat]

Does the CPU you swapped to support AVXv2?

[computersrmyfriends]

No, my CPU does not support AVX2 however it supports AVX

[jorgen-k]

Does the CPU you swapped to support AVXv2?

I am sorry, i do not know how to check that, i just changed o "host" being a Intel Core i9-9900K CPU