Issues when running quickstart on Apple silicon #2416

kevinhu · 2021-04-19T05:05:47Z

Describe the bug
When running the quickstart on a MacOS machine with the M1 chip, there are a few problems:

arm64 images aren't found by default for mysql, neo4j, and kibana. This can be fixed by manually specifying compatible ones (i.e. commenting out kibana, using arm64v8/mariadb for mysql, and neo4j/neo4j-arm64-experimental:4.2.5-arm64 for neo4j).
zookeeper and schema-registry crash with segmentation faults, potentially because of an issue with the qemu emulator on amd64 images. This can be mitigated for zookeeper, since an arm64 image exists, but I wasn't able to find one for schema-registry.

To Reproduce
Steps to reproduce the behavior:

Run ./docker/quickstart.sh.

Expected behavior
Containers all start up correctly and are healthy.

Desktop (please complete the following information):

OS: MacOS
Version: 11.2.3

The text was updated successfully, but these errors were encountered:

kevinhu · 2021-07-06T22:29:50Z

Update: retried quickstart on the latest version, and it seems that it's still unhappy with mysql, though the automatic switch to elasticsearch fixes the neo4j issue. Substituting mariadb for mysql still gives an error with the qemo emulator as before.

Logs:

➜ ./quickstart.sh
Quickstarting DataHub: version head
No Datahub Neo4j volume found, starting with elasticsearch as graph service
Pulling elasticsearch          ... done
Pulling elasticsearch-setup    ... done
Pulling mysql                  ... pulling from library/mysql
Pulling datahub-gms            ... done
Pulling datahub-frontend-react ... done
Pulling mysql-setup            ... done
Pulling zookeeper              ... done
Pulling broker                 ... done
Pulling schema-registry        ... done
Pulling kafka-setup            ... done

ERROR: for mysql  no matching manifest for linux/arm64/v8 in the manifest list entries
ERROR: no matching manifest for linux/arm64/v8 in the manifest list entries

zkan · 2021-07-18T14:15:12Z

Same here. I followed this quickstart guide and got an error when I run the command:

datahub docker quickstart

Logs:

Fetching docker-compose file from GitHub
No Datahub Neo4j volume found, starting with elasticsearch as graph service.
To use neo4j as a graph backend, run
`datahub docker quickstart --quickstart-compose-file ./docker/quickstart/docker-compose.quickstart.yml`
from the root of the datahub repo

[+] Running 0/10
 ⠿ elasticsearch Error                                                                                                                                                       5.3s
 ⠸ mysql Pulling                                                                                                                                                             5.3s
 ⠿ schema-registry Error                                                                                                                                                     5.3s
 ⠿ mysql-setup Error                                                                                                                                                         5.3s
 ⠿ elasticsearch-setup Error                                                                                                                                                 5.3s
 ⠿ datahub-frontend-react Error                                                                                                                                              5.3s
 ⠿ datahub-gms Error                                                                                                                                                         5.3s
 ⠿ kafka-setup Error                                                                                                                                                         5.3s
 ⠿ zookeeper Error                                                                                                                                                           5.3s
 ⠿ broker Error                                                                                                                                                              5.3s
no matching manifest for linux/arm64/v8 in the manifest list entries
[2021-07-18 21:08:22,001] ERROR    {datahub.entrypoints:106} - File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/datahub/entrypoints.py", line 98, in main
    95   def main(**kwargs):
    96       # This wrapper prevents click from suppressing errors.
    97       try:
--> 98           sys.exit(datahub(standalone_mode=False, **kwargs))
    99       except click.exceptions.Abort:
    ..................................................
     kwargs = {}
     datahub = <Group datahub>
     click.exceptions.Abort = <class 'click.exceptions.Abort'>
    ..................................................

File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    1135  def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:
 (...)
--> 1137      return self.main(*args, **kwargs)
    ..................................................
     self = <Group datahub>
     args = ()
     t.Any = typing.Any
     kwargs = {'standalone_mode': False}
    ..................................................

File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/datahub/cli/docker.py", line 195, in quickstart
    144  def quickstart(
    145      version: str,
    146      build_locally: bool,
    147      quickstart_compose_file: List[pathlib.Path],
    148      dump_logs_on_failure: bool,
    149      graph_service_impl: str,
    150  ) -> None:
 (...)
    191          "datahub",
    192      ]
    193
    194      # Pull and possibly build the latest containers.
--> 195      subprocess.run(
    196          [
    ..................................................
     quickstart = <Command quickstart>
     version = 'head'
     build_locally = False
     quickstart_compose_file = [PosixPath('/var/folders/tw/2ynjgp5j04sd45j_v09ykkqm0000gn/T/tmpmrtzoqyj.yml'), ]
     List = typing.List
     pathlib.Path = <class 'pathlib.Path'>
     dump_logs_on_failure = False
     graph_service_impl = None
     subprocess.run = <function 'run' subprocess.py:464>
    ..................................................

File "/Users/zkan/.pyenv/versions/3.9.6/lib/python3.9/subprocess.py", line 528, in run
    464  def run(*popenargs,
    465          input=None, capture_output=False, timeout=None, check=False, **kwargs):
 (...)
    524              # We don't call process.wait() as .__exit__ does that for us.
    525              raise
    526          retcode = process.poll()
    527          if check and retcode:
--> 528              raise CalledProcessError(retcode, process.args,
    529                                       output=stdout, stderr=stderr)
    ..................................................
     popenargs = (['docker-compose', '-f', '/var/folders/tw/2ynjgp5j04sd45j_v09ykkqm0000gn/T/tmpmrtzoqyj.yml', '-p', 'datahub', 'pull', ]
                  , )
     input = None
     capture_output = False
     timeout = None
     check = True
     kwargs = {}
     retcode = 18
     process.poll = <method 'Popen.poll' of <Popen: returncode: 18 args: ['docker-compose', '-f', '/var/folders/tw/2ynjg...> subprocess.py:1
                     158>
     CalledProcessError = <class 'subprocess.CalledProcessError'>
     process.args = ['docker-compose', '-f', '/var/folders/tw/2ynjgp5j04sd45j_v09ykkqm0000gn/T/tmpmrtzoqyj.yml', '-p', 'datahub', 'pull', ]
     stdout = None
     stderr = None
    ..................................................

---- (full traceback above) ----
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/datahub/entrypoints.py", line 98, in main
    sys.exit(datahub(standalone_mode=False, **kwargs))
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
File "/Users/zkan/Projects/zkan/hello-datahub/ENV/lib/python3.9/site-packages/datahub/cli/docker.py", line 195, in quickstart
    subprocess.run(
File "/Users/zkan/.pyenv/versions/3.9.6/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,

CalledProcessError: Command '['docker-compose', '-f', '/var/folders/tw/2ynjgp5j04sd45j_v09ykkqm0000gn/T/tmpmrtzoqyj.yml', '-p', 'datahub', 'pull']' returned non-zero exit status 18.

My environment:

Python 3.9.6
OS: MacOS 11.4

m-ketan · 2021-12-08T06:51:18Z

@zkan I'm having this exact same issue while running datahub docker quickstart on my M1 macbook. Were you able to figure it out?

dheinecke-bln · 2021-12-15T14:53:33Z

I have the same issue. Is there any work around?

zkan · 2021-12-16T00:59:48Z

@m-ketan No, not yet.

nhat2008 · 2022-01-10T10:03:07Z

in docker-compose file:
docker-compose-without-neo4j.quickstart.yml : under service mysql , add

image: mysql:5.7
platform: linux/x86_64

it will work for M1

seeholza · 2022-02-10T13:41:44Z

I had the same error, both for for neo4j and mysql;
Adding the platform tag for both entries

image: XXXX
platform: linux/x86_64

allowed me to start with neo4j via
datahub docker quickstart --quickstart-compose-file ./docker/quickstart/docker-compose.quickstart.yml
on a M1 macbook pro

cmpadden · 2022-04-20T16:39:12Z

To resolve this issue for all users on m1 macs, can we commit the change of adding the platform attribute? Or will this impact non-m1 users?

If it's fine, I'd happily create a PR - thanks!

pedro93 · 2022-04-20T16:54:03Z

Hello @cmpadden

Feel free to open a PR, I'll get someone with a non-m1 pc to validate. I'm on m1 myself too.
Thank you for finding a solution!!

shirshanka · 2022-04-20T22:46:27Z

@cmpadden : are you aware of the m1 specific quickstart file (without neo4j)?
https://github.com/datahub-project/datahub/blob/master/docker/quickstart/docker-compose-without-neo4j-m1.quickstart.yml

shirshanka · 2022-04-20T22:47:39Z

running datahub docker quickstart on Apple silicon now works.. so I'm closing this issue.
We can open a new one maybe if neo4j quickstart on m1 is the real issue we are trying to solve here.

kevinhu added the bug Bug report label Apr 19, 2021

cmpadden mentioned this issue Apr 20, 2022

Specify mysql platform in Docker Quickstart to support Apple Silicon #4709

Closed

5 tasks

shirshanka closed this as completed Apr 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issues when running quickstart on Apple silicon #2416

Issues when running quickstart on Apple silicon #2416

kevinhu commented Apr 19, 2021

kevinhu commented Jul 6, 2021

zkan commented Jul 18, 2021 •

edited

Loading

m-ketan commented Dec 8, 2021

dheinecke-bln commented Dec 15, 2021

zkan commented Dec 16, 2021

nhat2008 commented Jan 10, 2022

seeholza commented Feb 10, 2022 •

edited

Loading

cmpadden commented Apr 20, 2022

pedro93 commented Apr 20, 2022

shirshanka commented Apr 20, 2022

shirshanka commented Apr 20, 2022

Issues when running quickstart on Apple silicon #2416

Issues when running quickstart on Apple silicon #2416

Comments

kevinhu commented Apr 19, 2021

kevinhu commented Jul 6, 2021

zkan commented Jul 18, 2021 • edited Loading

m-ketan commented Dec 8, 2021

dheinecke-bln commented Dec 15, 2021

zkan commented Dec 16, 2021

nhat2008 commented Jan 10, 2022

seeholza commented Feb 10, 2022 • edited Loading

cmpadden commented Apr 20, 2022

pedro93 commented Apr 20, 2022

shirshanka commented Apr 20, 2022

shirshanka commented Apr 20, 2022

zkan commented Jul 18, 2021 •

edited

Loading

seeholza commented Feb 10, 2022 •

edited

Loading