Tracking Issue for Aarch64 Images #975

Alexhuszagh · 2022-07-27T17:56:04Z

This is a tracking issue for building aarch64 images, meaning they run natively on linux/arm64 instead of linux/amd64, and what's confirmed to work, what needs to be fixed, and what's currently untested, so we can ensure the maintainers don't duplicate work, new contributors know where to start, and looking at what to patch for future builds. We can also likely add builds for PPC64LE, since almost all of the maintained packages that have install candidates for ARM64 also support PPC64LE. Currently, most of the linux GNU packages are/will be failing.

Note that the PPC64LE images are currently untested, but assumed to work.

Currently Succeeding

Builds of these images for aarch64 can be found here for debugging and testing purposes (these are not official images, not are they guaranteed to be stable).

aarch64-unknown-linux-gnu (x86_64, i686, arm64, ppc64le, depends on Add concept of a native Dockerfile. #982)
s390x-unknown-linux-gnu (x86_64, i686, arm64, pp64le)
arm-unknown-linux-gnueabi (x86_64, i686, arm64, pp64le)
armv5te-unknown-linux-gnueabi (x86_64, i686, arm64, pp64le)
armv7-unknown-linux-gnueabi (x86_64, i686, arm64, pp64le)
armv7-unknown-linux-gnueabihf (x86_64, i686, arm64, pp64le)
i586-unknown-linux-gnu (x86_64, arm64, pp64le)
i686-unknown-linux-gnu (x86_64, i686, arm64, pp64le)
powerpc64le-unknown-linux-gnu (x86_64, i686, arm64)
riscv64gc-unknown-linux-gnu (x86_64, i686, arm64, ppc64le)
x86_64-unknown-linux-gnu (x86_64, i686, arm64, ppc64le, depends on Add concept of a native Dockerfile. #982)
aarch64-unknown-linux-musl (likely all architectures)
arm-unknown-linux-musleabi (likely all architectures)
arm-unknown-linux-musleabihf (likely all architectures)
armv5te-unknown-linux-musleabi (likely all architectures)
armv7-unknown-linux-musleabi (likely all architectures)
armv7-unknown-linux-musleabihf (likely all architectures)
i586-unknown-linux-musl (likely all architectures)
i686-unknown-freebsd (likely all architectures)
mips-unknown-linux-musl (likely all architectures)
mips64-unknown-linux-muslabi64 (likely all architectures)
mips64el-unknown-linux-muslabi64 (likely all architectures)
mipsel-unknown-linux-musl (likely all architectures)
sparcv9-sun-solaris (likely all architectures)
thumbv6m-none-eabi (x86_64, armhf, arm64, pp64le, s390x)
thumbv7em-none-eabi (x86_64, armhf, arm64, pp64le, s390x)
thumbv7em-none-eabihf (x86_64, armhf, arm64, pp64le, s390x)
thumbv7m-none-eabi (x86_64, armhf, arm64, pp64le, s390x)
thumbv7neon-unknown-linux-gnueabihf (x86_64, armhf, arm64, pp64le, s390x)
x86_64-unknown-freebsd (likely all architectures)

Currently Failing

arm-unknown-linux-gnueabihf (Unknown error in crosstool-ng build process)
sparc64-unknown-linux-gnu (Unable to locate package g++-sparc64-linux-gnu, not available for ARM64, will need a full compile from source)
mips-unknown-linux-gnu (Package for ARM64 and PPC64LE only in 22.04+, not in 20.04)
mips64-unknown-linux-gnuabi64 (Unable to locate package g++-mips64-linux-gnuabi64, not available for ARM64, will need a full compile from source)
mips64el-unknown-linux-gnuabi64 (Unable to locate package g++-mips64el-linux-gnuabi64, not available for ARM64, will need a full compile from source)
mipsel-unknown-linux-gnu (Unable to locate package g++-mipsel-linux-gnu, not available for ARM64, will need a full compile from source)
powerpc-unknown-linux-gnu (Unable to locate package g++-powerpc-linux-gnu, not available for ARM64, only x86_64 and PPC64LE, will need a full compile from source)
powerpc64-unknown-linux-gnu (Unable to locate package g++-powerpc64-linux-gnu, not available for ARM64, will need a full compile from source)
i686-pc-windows-gnu (unlikely to work without building WINE from source)
x86_64-pc-windows-gnu (unlikely to work without building WINE from source)
wasm32-unknown-emscripten (no linux/arm64 image for emsdk)
x86_64-sun-solaris (fails while building GCC).

Failing, Cannot Fix

aarch64-linux-android
arm-linux-androideabi
armv7-linux-androideabi
i686-linux-android
thumbv7neon-linux-androideabi
x86_64-linux-android

The Android images have the following requirements, which means it will be impossible to build a cross-compiler for them without significant effort from a non-x86 host:

They require gcc-multilib and g++-multilib, which are not available on aarch64 and any host other than i686, x86_64, and s390x.
They use prebuilts for clang, gcc and go that assume an x86 host.
The NDKs assume an x86_64 host.

The only solution therefore would be to rebuild the entire NDK from source, get Android system, replace the prebuilts with native toolchains for a non-x86 host, update code so it uses the correct non-x86 prebuilts, etc. If this sounds like an issue, it is.

Currently Untested

x86_64-unknown-dragonfly
x86_64-unknown-illumos
x86_64-unknown-linux-gnu:centos
x86_64-unknown-linux-musl
x86_64-unknown-netbsd

Contributing

If you'd like to contribute your own tests, successes, etc., feel free to do so with the following examples. The runners, std, C++, and dynamic library support can be found in the Github actions ci.yml.

# change this for specific targets
$ export TARGET=s390x-unknown-linux-gnu
$ cargo build-docker-image --platform linux/arm64/v8=aarch64-unknown-linux-gnu $TARGET
$ export CROSS_TARGET_S390X_UNKNOWN_LINUX_GNU_IMAGE_TOOLCHAIN="aarch64-unknown-linux-gnu"
$ export CROSS_TARGET_S390X_UNKNOWN_LINUX_GNU_IMAGE="ghcr.io/cross-rs/$TARGET:local"
$ export STD=1
$ export CPP=1
$ export DYLIB=1
$ export RUN=1
$ export RUNNERS="qemu-user qemu-system"
$ ci/test.sh

The text was updated successfully, but these errors were encountered:

Alexhuszagh · 2022-07-28T14:59:50Z

If we install packages from Debian for some of these toolchains, we can support a lot more architectures. Potential solutions include:

mips64-unknown-linux-gnuabi64: (amd64 arm64 i386 mips64el mipsel ppc64el)
mips64el-unknown-linux-gnuabi64: (amd64 arm64 i386 mipsel ppc64el)
mipsel-unknown-linux-gnu: (amd64 arm64 i386 mips64el ppc64el)

This only seems to affect MIPS however.

DoumanAsh · 2022-07-29T03:27:58Z

Just observation I encountered issue with latest libc crate (0.2.126) requiring newer glib version

  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)

It would be good to consider bumping ubuntu images as they have relatively old glib for cross compilation

Alexhuszagh · 2022-07-29T04:16:28Z

Just observation I encountered issue with latest libc crate (0.2.126) requiring newer glib version

  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)

It would be good to consider bumping ubuntu images as they have relatively old glib for cross compilation

That's unrelated (and an upstream bug): see #724. Try running cargo clean and then rebuilding. This can occur in any repository with a build script. It can happen fairly regularly if switching between toolchains. This is a cargo bug, but is exacerbated by how cross functions (swapping toolchains with the same target directory, which causes the bug to manifest).

abcd-ca · 2022-11-05T14:49:41Z

I downloaded an aarch64 image to my host (my M1 mac). Would someone tell me how to use it? I've got a knowledge gap between having the native (M1, aarch64) image and this command:
CROSS_REMOTE=1 DOCKER_HOST=tcp://docker:2375/ cross build --release --target=armv7-unknown-linux-musleabihf.

I think for me the DOCKER_HOST part is wrong and when I try to run the image (start a container based on it) I see no port has been exposed.

Thank you

Alexhuszagh · 2022-11-05T15:35:01Z

@abcd-ca

Do you need CROSS_REMOTE? You should be able to run it natively on your Mac M1, and therefore don't need remote docker, unless you are connecting to the client remotely. You do have to provide the image name itself, however: these aren't named the same so you have to override the image name as follows:

CROSS_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_IMAGE=ahuszagh/aarch64-cross:armv7-unknown-linux-musleabihf \
    cross build --release --target=armv7-unknown-linux-musleabihf

Does that help? All the image names and tags can be found here.

abcd-ca · 2022-11-05T16:33:26Z

@Alexhuszagh Thanks, that does look more like what I need however with that I get,

docker: Error response from daemon: image with reference ahuszagh/aarch64-cross:armv7-unknown-linux-musleabihf was found but does not match the specified platform: wanted linux/amd64, actual: linux/arm64.

Alexhuszagh · 2022-11-05T16:36:23Z

Is your local machine the M1? Or are you connecting remotely to it? If so, then you do need CROSS_REMOTE along with the correct connection parameters (SSH connection? TCP?).

abcd-ca · 2022-11-05T16:37:59Z

Local is M1. No physically remote build computer yet. Hoping to avoid that

Alexhuszagh · 2022-11-05T19:44:58Z

Local is M1. No physically remote build computer yet. Hoping to avoid that

Ah I think I know what the issue is. Sorry for the repeated mentions. You need to do the instructions under target.TARGET.image, or provide them as environment variables.

So the full invocation is:

CROSS_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_IMAGE_TOOLCHAIN="aarch64-unknown-linux-gnu" \
    CROSS_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_IMAGE=ahuszagh/aarch64-cross:armv7-unknown-linux-musleabihf \
    cross build --release --target=armv7-unknown-linux-musleabihf

If we deploy native ARM64 images, this should be streamlined a lot later on (we need custom runners which aren't free), but right now it's a bit janky. Does that fix the issue?

abcd-ca · 2022-11-05T21:06:13Z

Sorry for the repeated mentions

Are you kidding? Thanks for being so generous with your time and expertise!

Success! 🎉

Here are some test results (docker images already automatically downloaded so time to download those not included):

Build time with default build image
cross clean
time ./build.sh

Finished release [optimized] target(s) in 8m 45s
./build.sh 0.46s user 0.26s system 0% cpu 8:48.76 total

Build time with native aarch64 (M1) build image:
cross clean
time ./build.sh (modified to use your latest env vars)

Finished release [optimized] target(s) in 2m 23s
./build.sh 0.65s user 0.64s system 0% cpu 2:27.52 total

Summary:

About 6.5 minutes (73%) faster!

Cross's android images are causing CI to fail. Maybe related: cross-rs/cross#975

jaysonsantos · 2023-03-31T14:36:51Z

Hey there folks, are these images being pushed somewhere? In this repo, I can only see linux/amd64 and unknown/unknown

Emilgardis · 2023-03-31T16:13:21Z

@Alexhuszagh has pushed initial images here as mentioned in this issue.

We have not started implementing this in-org yet, and thus they are not on ghcr.io/cross-rs. You can also build these images yourself

YOU54F · 2023-05-30T13:32:55Z

Hey @Alexhuszagh

Kudos on the awesome tracking list, I have this saved as a bookmark, was SUPER useful for me, in my current ARMaggedon flex on our open source project.

Linked PR above ☝🏾 to attempt to cover off the case for aarch64-unknown-freebsd

1271: feat: add aarch64-unknown-freebsd image r=Emilgardis a=YOU54F 👋🏾 Thanks for Cross, we've been using it at the pact-foundation to build cross platform targets, and I've gone through the target list and attempted to build for everything I can. See [PR here](pact-foundation/pact-reference#276) I've managed to build for freebsd in a parallels vm on a mac m1 pro, but makes sense to try and apply that to a Docker image. Relates to #975 Still requires testing but thought I would drop it here for others to have a look at / test / try out Co-authored-by: Yousaf Nabi <[email protected]>

mcandre · 2024-01-04T07:26:50Z

Configuring a DOCKER_DEFAULT_PLATFORM environment variable with value linux/amd64 can smooth out rough edges while awaiting publication of multi-architecture images. You may need to explicitly initialize the Docker buildx subsystem, something that the tug third party utility can do for ya.

https://github.com/mcandre/tug

Recommend to house this configuration in direnv / psenv, in order to reduce conflicts between different software projects you may be working on.

Coruscant11 · 2024-07-30T08:30:34Z

Hi @Alexhuszagh !
I was wondering if providing a cross-rs/zig:aarch64 image would make it easier ?

molenick · 2024-08-06T02:59:12Z

I'm trying to install libswupdate-dev:arm64 against the default image and have not been able to figure out how using package management.

My cross file looks like this:

[target.aarch64-unknown-linux-gnu]
pre-build = [
    "dpkg --add-architecture $CROSS_DEB_ARCH",
    "apt-get update",
    "apt-get --assume-yes install libswupdate-dev:$CROSS_DEB_ARCH",
]

But when I run:

cross build --target aarch64-unknown-linux-gnu --verbose

I get the following error:

E: Unable to locate package libswupdate-dev:arm64

I've tried a bunch of stuff in the pkg-mngt wrangling but no success. At one point I got it to be located but it wouldn't install because of issues with libc toolchain not being available. I was able to build successfully by directly installing the .deb packages, but my preference is to use apt.

I've read the docs linked from the README a few times but I feel like I'm missing something here about the problem I'm facing.

edit: Is it possible to get command output from the docker setup? I can see the command thats running with export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true but I can't seem to get command output; that make it easier for me to debug.

Emilgardis · 2024-08-06T07:20:24Z

edit: Is it possible to get command output from the docker setup? I can see the command thats running with export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true but I can't seem to get command output; that make it easier for me to debug.

Yes, simply add -v @molenick

Alexhuszagh added container-images A-aarch64-host Area: ARMv8 hosts tracking-issue Issue to track progress for a certain feature/enhancement.. labels Jul 27, 2022

Alexhuszagh mentioned this issue Jul 28, 2022

Strange behavior on aarch64 linux #970

Closed

Emilgardis mentioned this issue Jul 28, 2022

Support running natively on aarch64 / arm64 hosts / Mac M1/M2 #751

Open

2 tasks

krobelus mentioned this issue Aug 11, 2022

macOS Homebrew/binary release does not support arm64 kakoune-lsp/kakoune-lsp#638

Closed

This comment was marked as off-topic.

Sign in to view

djkoloski added a commit to rkyv/rkyv that referenced this issue Mar 9, 2023

Disable android test targets

1688d9c

Cross's android images are causing CI to fail. Maybe related: cross-rs/cross#975

djkoloski mentioned this issue Mar 9, 2023

Disable android test targets rkyv/rkyv#365

Merged

djkoloski added a commit to rkyv/rkyv that referenced this issue Mar 9, 2023

Disable android test targets (#365)

bae3f4d

Cross's android images are causing CI to fail. Maybe related: cross-rs/cross#975

YOU54F mentioned this issue May 30, 2023

feat: add aarch64-unknown-freebsd image #1271

Merged

This comment was marked as off-topic.

Sign in to view

Emilgardis mentioned this issue Dec 29, 2023

"no matching manifest" on cross v0.2.5 #1214

Open

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tracking Issue for Aarch64 Images #975

Tracking Issue for Aarch64 Images #975

Alexhuszagh commented Jul 27, 2022 •

edited by Emilgardis

Loading

Alexhuszagh commented Jul 28, 2022

DoumanAsh commented Jul 29, 2022 •

edited

Loading

Alexhuszagh commented Jul 29, 2022 •

edited

Loading

This comment was marked as off-topic.

abcd-ca commented Nov 5, 2022

Alexhuszagh commented Nov 5, 2022 •

edited

Loading

abcd-ca commented Nov 5, 2022

Alexhuszagh commented Nov 5, 2022

abcd-ca commented Nov 5, 2022

Alexhuszagh commented Nov 5, 2022 •

edited

Loading

abcd-ca commented Nov 5, 2022 •

edited

Loading

jaysonsantos commented Mar 31, 2023

Emilgardis commented Mar 31, 2023 •

edited

Loading

YOU54F commented May 30, 2023

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

mcandre commented Jan 4, 2024 •

edited

Loading

Coruscant11 commented Jul 30, 2024 •

edited

Loading

molenick commented Aug 6, 2024 •

edited

Loading

Emilgardis commented Aug 6, 2024

Tracking Issue for Aarch64 Images #975

Tracking Issue for Aarch64 Images #975

Comments

Alexhuszagh commented Jul 27, 2022 • edited by Emilgardis Loading

Currently Succeeding

Currently Failing

Failing, Cannot Fix

Currently Untested

Contributing

Alexhuszagh commented Jul 28, 2022

DoumanAsh commented Jul 29, 2022 • edited Loading

Alexhuszagh commented Jul 29, 2022 • edited Loading

This comment was marked as off-topic.

abcd-ca commented Nov 5, 2022

Alexhuszagh commented Nov 5, 2022 • edited Loading

abcd-ca commented Nov 5, 2022

Alexhuszagh commented Nov 5, 2022

abcd-ca commented Nov 5, 2022

Alexhuszagh commented Nov 5, 2022 • edited Loading

abcd-ca commented Nov 5, 2022 • edited Loading

jaysonsantos commented Mar 31, 2023

Emilgardis commented Mar 31, 2023 • edited Loading

YOU54F commented May 30, 2023

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

mcandre commented Jan 4, 2024 • edited Loading

Coruscant11 commented Jul 30, 2024 • edited Loading

molenick commented Aug 6, 2024 • edited Loading

Emilgardis commented Aug 6, 2024

Alexhuszagh commented Jul 27, 2022 •

edited by Emilgardis

Loading

DoumanAsh commented Jul 29, 2022 •

edited

Loading

Alexhuszagh commented Jul 29, 2022 •

edited

Loading

Alexhuszagh commented Nov 5, 2022 •

edited

Loading

Alexhuszagh commented Nov 5, 2022 •

edited

Loading

abcd-ca commented Nov 5, 2022 •

edited

Loading

Emilgardis commented Mar 31, 2023 •

edited

Loading

mcandre commented Jan 4, 2024 •

edited

Loading

Coruscant11 commented Jul 30, 2024 •

edited

Loading

molenick commented Aug 6, 2024 •

edited

Loading