Cross compiling on M1 Mac gives weird linker error

[korken89]

Checklist

• I've looked through the issues and pull requests for similar reports

Describe your issue

When building an app with the following cross config we get a linking error that was unexpected and googling it does not really give insight into the error.

Cross.toml

[target.x86_64-unknown-linux-gnu]
pre-build = ["apt-get update && apt-get install -y pkg-config gcc libssl-dev openssl libcrypto++8 mosquitto-dev libclang-dev "]
image = "ubuntu:22.04"

Build command:

DOCKER_BUILDKIT=1 \
cross build --release --target x86_64-unknown-linux-gnu

Error:

qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

Building this on a Linux (arch) computer works fine, but on an M1 Mac it fails.

Any hints on what might be wrong?

What target(s) are you cross-compiling for?

No response

Which operating system is the host (e.g computer cross is on) running?

• macOS
• Windows
• Linux / BSD
• other OS (specify in description)

What architecture is the host?

• x86_64 / AMD64
• arm32
• arm64 (including Mac M1)

What container engine is cross using?

• docker
• podman
• other container engine (specify in description)

cross version

cross 0.2.5

Example

Cross.toml

[target.x86_64-unknown-linux-gnu]
pre-build = ["apt-get update && apt-get install -y pkg-config gcc libssl-dev openssl libcrypto++8 mosquitto-dev libclang-dev "]
image = "ubuntu:22.04"

Build command:

DOCKER_BUILDKIT=1 \
cross build --release --target x86_64-unknown-linux-gnu

Additional information / notes

It builds on an Arch Linux computer

[Emilgardis]

this is a bug, solved by #817 (specifically this line)

the problem is similar to #1214 and has the same solution (and cause)

when cross 0.2.5 pulls the docker image, it doesn't specify --platform linux/amd64, instead docker assumes local which means the image will be linux/arm64 on an arm64 machine and thus ld for x86 will not be available by default.

So, either use cross from main which has #817 or run with CROSS_CONTAINER_OPTS="--platform linux/amd64"

Does that fix it?

[korken89]

Hi @Emilgardis, thank you for the help!
With that we get further, but still we have an issue.

It seems that when the docker should be saved it fails with an error that we can't find what's wrong with.
Do you know what is wrong here?

> precision-engine % make precision-engine

CROSS_CONFIG=../build/Cross-default.toml \
        CROSS_CONTAINER_OPTS="--platform linux/amd64" \
        cross build --release --target x86_64-unknown-linux-gnu

[+] Building 0.9s (6/6) FINISHED                                                                                
 => [internal] load build definition from Dockerfile.x86_64-unknown-linux-gnu-custom                       0.0s
 => => transferring dockerfile: 216B                                                                       0.0s
 => [internal] load .dockerignore                                                                          0.0s
 => => transferring context: 2B                                                                            0.0s
 => [internal] load metadata for docker.io/library/ubuntu:22.04                                            0.8s
 => [1/2] FROM docker.io/library/ubuntu:22.04@sha256:67211c14fa74f070d27cc59d69a7fa9aeff8e28ea118ef3babc2  0.0s
 => CACHED [2/2] RUN eval "apt-get update && apt-get install -y pkg-config gcc libssl-dev openssl libcryp  0.0s
 => exporting to image                                                                                     0.0s
 => => exporting layers                                                                                    0.0s
 => => writing image sha256:dd996ccb90303ec8bdeb2b866ebf0ae56f12efc009b517beecdcaf08a1bd75e5               0.0s
 => => naming to docker.io/library/cross-custom-precision-engine:x86_64-unknown-linux-gnu-ac233-pre-build  0.0s
Unable to find image 'cross-custom-precision-engine:x86_64-unknown-linux-gnu-ac233-pre-build' locally
docker: Error response from daemon: pull access denied for cross-custom-precision-engine, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
make: *** [precision-engine] Error 125

It does however work fine on my Linux computer, but not an M1 Mac.

[Emilgardis]

This is also a bug fixed in #817, solved by edf1e17 (#817 (comment))

So, add --output type=docker to CROSS_CONTAINER_OPTS

[korken89]

Hi @Emilgardis, thank you.
This did however generate a new error, that --output does not exist.

CROSS_CONFIG=../build/Cross-default.toml \
CROSS_CONTAINER_OPTS="--platform linux/amd64 --output type=docker" \
DOCKER_BUILDKIT=1 \
cross build \
	--release \
	--target x86_64-unknown-linux-gnu

[+] Building 0.2s (6/6) FINISHED
 => [internal] load .dockerignore                                                                0.1s
 => => transferring context: 2B                                                                  0.0s
 => [internal] load build definition from Dockerfile.x86_64-unknown-linux-gnu-custom             0.1s
 => => transferring dockerfile: 211B                                                             0.0s
 => [internal] load metadata for docker.io/library/ubuntu:22.04                                  0.0s
 => [1/2] FROM docker.io/library/ubuntu:22.04                                                    0.0s
 => CACHED [2/2] RUN eval "apt-get update && apt-get install -y pkg-config gcc libssl-dev opens  0.0s
 => exporting to image                                                                           0.0s
 => => exporting layers                                                                          0.0s
 => => writing image sha256:dda7f80bff5a244b649d8684673085b8720271dec02d87a02480a9fb60eaddb3     0.0s
 => => naming to docker.io/library/cross-custom-precision-engine:x86_64-unknown-linux-gnu-fc267  0.0s
unknown flag: --output
See 'docker run --help'.
make: *** [Makefile:37: precision-engine] Error 125

Any suggestions?

[Emilgardis]

As mentioned in that commit, it requires a newer docker api: 20.10 (API 1.40)

make sure you have that version or newer

[korken89]

Thanks for the insight @Emilgardis!
Hmm, weird checking docker version gives the following so it seems it should be more than enough new.

>>> docker version
Client:
 Version:           23.0.3
 API version:       1.42
 Go version:        go1.20.2
 Git commit:        3e7cbfdee1
 Built:             Wed Apr  5 13:17:36 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          23.0.3
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.20.2
  Git commit:       59118bff50
  Built:            Wed Apr  5 13:17:36 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.0
  GitCommit:        1fbd70374134b891f97ce19c70b6e50c7b9f4e0d.m
 runc:
  Version:          1.1.6
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[korken89]

Alright, I found one thing but I don't understand the internals of cross enough to understand the issue.
It seems docker run is used by cross and it does not have --output - but checking with docker build it has the --output flag.

Not sure if that helps or is relevant, but that's where the different in --output seems to be.

[Emilgardis]

If you add -v you'll see exactly what is happening, could you specify that and show what happens?

docker run is only used in the last step, and due to your usage of target.<target>.pre-build cross will first create a image that has executed whatever is in pre-build, using docker build and then use that image in the final docker run.

Also, you're now on 0.2.5 right?
Could you try using cargo install --git https://github.com/cross-rs/cross cross instead?

[korken89]

Hi @Emilgardis,

We went with installing cross from git as you recommended and then it JustWorked, thanks!

Edit: I spoke too soon, we're still seeing weird things - I'll come back after some debugging.

[danf0rth]

I was able to fix some weird linking issues by fully reset docker. Disabling rosetta2 virtualization in "Features in development" menu not worked, though i did not tried to disable containerd experimental feature.

Most likely the problem was in containerd feature.

[patrickelectric]

I had the problem using archlinux docker 25.0.2, cross 0.2.5.
Was able to fix using the following command line:

CROSS_BUILD_OPTS="--output=type=docker"  CROSS_CONFIG=Cross.toml cross build --target armv7-unknown-linux-gnueabihf --release