docker-compose up fails on Mac M1: no matching manifest for linux/arm64/v8 in the manifest list entries

[rickfish]

Describe the bug
On a Mac M1 (Apple chip), docker-compose build runs fine with this message at the beginning of the build:

elasticsearch uses an image, skipping

then when I try docker-compose up I get the following error:

➜ docker git:(main) ✗ docker-compose up
Pulling elasticsearch (elasticsearch:6.8.15)...
6.8.15: Pulling from library/elasticsearch
ERROR: no matching manifest for linux/arm64/v8 in the manifest list entries

Details
Conductor version: 3.7.1
Persistence implementation: default for docker-compose.yml
Queue implementation: default for docker-compose.yml
Workflow definition:
Task definition:
Event handler definition:

To Reproduce
Steps to reproduce the behavior:

1. Go to docker directory
2. Run docker-compose build
3. Run docker-compose up
4. See error

Expected behavior
Either build the elasticsearch container or be able to run the containers

Additional context
I tried to add the :osx-x86_64 in the grpc/build.gradle as instructed. When I do, a normal ./gradlew build runs fine but the docker-compose build fails with an 'Unterminated quoted string' error so I removed it. Then I see the behavior in this issue.

[apanicker-nflx]

@rickfish Looks like docker supported elasticsearch images for M1 mac are only available from v7.10.2
Additionally, the only supported OS/Arch for 6.8.15 is linux/amd64.

Looks like the options are -

1. You would need to build the ES image yourself and run the container using this image
2. Run ES locally outside of docker

[rickfish]

Oh, ok. thanks for looking into it.

[amimimor]

@rickfish Looks like docker supported elasticsearch images for M1 mac are only available from v7.10.2 Additionally, the only supported OS/Arch for 6.8.15 is linux/amd64.

Looks like the options are -

1. You would need to build the ES image yourself and run the container using this image
2. Run ES locally outside of docker

I tried and succeeded by adding a platform: "linux/amd64" entry to the elasticsearch service definition in docker-compose.yaml

[apanicker-nflx]

@amimimor That's great. Would you be willing to submit a PR with the change? This would be really beneficial to the community

[rickfish]

@amimimor that's great. Thanks. I will try it on my machine later this week.

[amimimor]

@rickfish thanks, but admittedly, I haven't yet managed to boot the entire docker-compose setup on my M1. I now have issues with conductor-ui missing a binary. Sometimes gradle is hanging, etc. It is quite an awful experience. Hopefully I'll manage to workaround this and then create a PR

[csears]

Is upgrading to ES7 on the roadmap?

I tried running with the ES7 Arm64 image and ran into type mapping errors due to a breaking change (in Elasticsearch 7).

[javaHelper]

I'm getting below error with the same setup and changes suggested

exception: Security must be explicitly enabled when using a [basic] license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node. (500)

[anandbhaskaran]

Would be awesome if someone publish this docker image for M1.

[alberttwong]

I also get the same error. I'm on Apple M2 pro chip.

[alberttwong]

added the platform line to docker-compose.yaml

  elasticsearch:
    image: elasticsearch:6.8.15
    container_name: elasticsearch
    platform: "linux/amd64"
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx1024m"
      - transport.host=0.0.0.0
      - discovery.type=single-node
      - xpack.security.enabled=false
    volumes:
      - esdata-conductor:/usr/share/elasticsearch/data
    networks:
      - internal
    ports:
      - 9200:9200
      - 9300:9300
    healthcheck:
      test: wget http://localhost:9200/ -O /dev/null
      interval: 5s
      timeout: 5s
      retries: 12
    logging:
      driver: "json-file"
      options:
        max-size: "1k"
        max-file: "3"

error message now.

elasticsearch     | OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
elasticsearch     | [2023-07-08T17:03:35,473][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [unknown] uncaught exception in thread [main]
elasticsearch     | org.elasticsearch.bootstrap.StartupException: org.elasticsearch.bootstrap.BootstrapException: java.io.IOException: Cannot run program "/usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller": error=0, Failed to exec spawn helper: pid: 102, exit value: 1
elasticsearch     | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | Caused by: org.elasticsearch.bootstrap.BootstrapException: java.io.IOException: Cannot run program "/usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller": error=0, Failed to exec spawn helper: pid: 102, exit value: 1
elasticsearch     | 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:167) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	... 6 more
elasticsearch     | Caused by: java.io.IOException: Cannot run program "/usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller": error=0, Failed to exec spawn helper: pid: 102, exit value: 1
elasticsearch     | 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1142) ~[?:?]
elasticsearch     | 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1073) ~[?:?]
elasticsearch     | 	at org.elasticsearch.bootstrap.Spawner.spawnNativeController(Spawner.java:118) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Spawner.spawnNativeControllers(Spawner.java:86) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:165) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	... 6 more
elasticsearch     | Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 102, exit value: 1
elasticsearch     | 	at java.lang.ProcessImpl.forkAndExec(Native Method) ~[?:?]
elasticsearch     | 	at java.lang.ProcessImpl.<init>(ProcessImpl.java:313) ~[?:?]
elasticsearch     | 	at java.lang.ProcessImpl.start(ProcessImpl.java:244) ~[?:?]
elasticsearch     | 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1109) ~[?:?]
elasticsearch     | 	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1073) ~[?:?]
elasticsearch     | 	at org.elasticsearch.bootstrap.Spawner.spawnNativeController(Spawner.java:118) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Spawner.spawnNativeControllers(Spawner.java:86) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:165) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.15.jar:6.8.15]
elasticsearch     | 	... 6 more
elasticsearch exited with code 1

[alberttwong]

Tips at https://stackoverflow.com/questions/68877644/how-to-run-elasticsearch-6-on-an-apple-silicon-mac didn't work for me. however I didn't try webhippie/elasticsearch:6.4

[alberttwong]

https://stackoverflow.com/questions/68957318/installing-elasticsearch-6-8-15-on-apple-m1-macbook-issues

[alberttwong]

TL-DR; Basically conductor won't work on Apple M1/M2 chips. You have to run it on a Windows machine.

[superbsky]

I was able to launch Conductor from Docker Compose using the webhippie/elasticsearch:6.8 image. The only adjustment I needed to make was to pass the xpack parameter using a different environment variable, ELASTICSEARCH_XPACK_SECURITY_ENABLED=false.

[alberttwong]

Can you share your compose yaml

[superbsky]

Literally, it's just two lines:

elasticsearch:
image: webhippie/elasticsearch:6.8
...
- ELASTICSEARCH_XPACK_SECURITY_ENABLED=false

For more details, please see here:
https://hub.docker.com/r/webhippie/elasticsearch/

[Krever]

I confirm, this does the trick and elastisearch starts just fine :)

I still struggle to run full compose due to redis socket timeout but its a separate issue