Add support of cross-building the engine for ARM64 Linux Host #390

HidenoriMatsubayashi · 2020-08-05T04:46:49Z

Description

Added cross-building Flutter Engine for ARM64 Linux platforms. This PR is part of ARM64 Linux support in the Flutter SDK.

Related PRs

Related Issues

Tests

1. Build Test

First, I confirmed that Flutter Engine can be built for x64 and arm64. The procedure is as follows. All are operations on x64 Linux Host.

Get source code and setup build enviroment.

create .gclient file.
$ gclient sync
$ cd src

Cross-build for ARM64 Linux

$ ./flutter/tools/gn --target-os linux --linux-cpu arm64 --runtime-mode release
$ ninja -C out/linux_release_arm64
-> Passed!

$ ./flutter/tools/gn --target-os linux --linux-cpu arm64 --runtime-mode debug
$ ninja -C out/linux_debug_arm64
-> Passed!

Build for x64 Linux and Regression test.

$ ./flutter/tools/gn --unoptimized
$ ninja -C out/host_debug_unopt
-> Passed!

$ ./flutter/testing/run_tests.sh
-> All Tests Passed!

2. Run Flutter SDK and app on ARM64 Linux (Jetson Nano)

Second, using the artifacts for ARM64 built by the above procedure, I have confirmed that the Flutter SDK and Flutter app can work fine on Jetson Nano (ARM64 CPU).

All are operations on ARM64 Linux Host (Jetson Nano).

$ git clone https://github.com/flutter/flutter.git
$ export PATH=$PATH:{your working dir path}/flutter/bin
$ flutter doctor
   ... Error happened.

Copy dart-sdk which was locally built to ${your installpath}/bin/cache/.

$ flutter doctor
$ flutter config --enable-linux-desktop
$ flutter create sample
$ cd sample
$ flutter build linux
   ... Error happened.

Copy the Flutter SDK artifacts (build data) to ${your installpath}/bin/cache/artifacts/engine directory manually.

out/linux_debug_arm64/flutter_patched_sdk to ${your installpath}/bin/cache/artifacts/engine/common/flutter_patched_sdk
out/linux_debug_arm64/flutter_linux, gen/const_finder.dart.snapshot, flutter_tester, font-subset, gen/frontend_server.dart.snapshot, icudtl.dat, libflutter_linux_gtk.so, gen/flutter/lib/snapshot/isolate_snapshot.bin, gen/flutter/lib/snapshot/vm_isolate_snapshot.bin to ${your installpath}/bin/cache/artifacts/engine/artifacts/engine/linux-x64
out/linux_release_arm64/flutter_patched_sdk to ${your installpath}/bin/cache/artifacts/engine/common/flutter_patched_sdk_product
out/linux_release_arm64/flutter_linux, libflutter_linux_gtk.so to ${your installpath}/bin/cache/artifacts/engine/artifacts/engine/linux-x64-release
out/linux_release_arm64/clang_x64/gen_snapshot to ${your installpath}/bin/cache/artifacts/engine/artifacts/engine/linux-x64-release

For simple operation check, ${your installpath}/bin/cache/artifacts/engine/artifacts/engine/linux-x64-profile is omitted.

( $ flutter build linux --verbose )
$ flutter run -d linux

…-aarch64-cross-build

build/config/sysroot.gni

stuartmorgan-g · 2020-09-01T13:06:05Z

build/toolchain/linux/BUILD.gn

  ar = "ar"
  ld = cxx
-  strip = "strip"
+  strip = "${compiler_prefix}$prefix/llvm-strip"


Why is this needed? Also, what are the implications for x64 for changing from strip to llvm-strip?

Thank you for your response.

Why is this needed?

This is to fix the following error. We have to use the binaries under buildtools/linux-x64/clang/bin/. I think that originally wrong. dart-sdk is helpful. I guess it's better to modify readelf, nm, and ar as well...

[4439/6380] LINK ./font-subset FAILED: font-subset exe.unstripped/font-subset ../../buildtools/linux-x64/clang/bin/clang++ -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -pthread --target=aarch64-linux-gnu --sysroot=/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot -L/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/usr/local/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/usr/local/lib/aarch64-linux-gnu\ -L/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/lib/aarch64-linux-gnu\ -L/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/usr/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/usr/lib/aarch64-linux-gnu -Wl,-O2 -Wl,--gc-sections -Wl,--as-needed -Wl,-rpath=\$ORIGIN/ -Wl,-rpath-link= -Wl,--disable-new-dtags -o ./exe.unstripped/font-subset -Wl,--build-id=sha1 -Wl,--start-group @./font-subset.rsp -Wl,--end-group -ldl && strip --strip-unneeded -o ./font-subset ./exe.unstripped/font-subset strip: Unable to recognise the format of the input file `./exe.unstripped/font-subset' [4444/6380] LINK ./dart_precompiled_runtime_product FAILED: dart_precompiled_runtime_product exe.unstripped/dart_precompiled_runtime_product ../../buildtools/linux-x64/clang/bin/clang++ -rdynamic -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -pthread --target=aarch64-linux-gnu --sysroot=/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot -L/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/usr/local/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/usr/local/lib/aarch64-linux-gnu\ -L/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/lib/aarch64-linux-gnu\ -L/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/usr/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/usr/lib/aarch64-linux-gnu -Wl,-O2 -Wl,--gc-sections -Wl,--as-needed -Wl,-rpath=\$ORIGIN/ -Wl,-rpath-link= -Wl,--disable-new-dtags -o ./exe.unstripped/dart_precompiled_runtime_product -Wl,--build-id=sha1 -Wl,--start-group @./dart_precompiled_runtime_product.rsp -Wl,--end-group -ldl -lpthread && strip --strip-unneeded -o ./dart_precompiled_runtime_product ./exe.unstripped/dart_precompiled_runtime_product strip: Unable to recognise the format of the input file `./exe.unstripped/dart_precompiled_runtime_product' [4446/6380] LINK ./gen_snapshot_product FAILED: gen_snapshot_product exe.unstripped/gen_snapshot_product ../../buildtools/linux-x64/clang/bin/clang++ -rdynamic -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs -pthread --target=aarch64-linux-gnu --sysroot=/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot -L/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/usr/local/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/usr/local/lib/aarch64-linux-gnu\ -L/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/lib/aarch64-linux-gnu\ -L/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/usr/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/root/flutter_work/build_engine/src/build/linux/debian_sid_arm64-sysroot/usr/lib/aarch64-linux-gnu -Wl,-O2 -Wl,--gc-sections -Wl,--as-needed -Wl,-rpath=\$ORIGIN/ -Wl,-rpath-link= -Wl,--disable-new-dtags -o ./exe.unstripped/gen_snapshot_product -Wl,--build-id=sha1 -Wl,--start-group @./gen_snapshot_product.rsp -Wl,--end-group -ldl -lpthread && strip --strip-unneeded -o ./gen_snapshot_product ./exe.unstripped/gen_snapshot_product strip: Unable to recognise the format of the input file `./exe.unstripped/gen_snapshot_product' [4452/6380] CXX clang_x64/obj/third_party/dart/runtime/lib/libdart_lib_jit.ffi_dynamic_library.o ninja: build stopped: subcommand failed.

Also, what are the implications for x64 for changing from string to llvm-strip?

No implications for x64 because clang_x64 is used for self-building for x64, and the commands installed on a Host are used.

cc_toolchain("clang_x64") { if (use_clang_type_profiler) { prefix = rebase_path("//third_party/llvm-allocated-type/Linux_x64/bin", root_build_dir) } else { prefix = rebase_path("//buildtools/linux-x64/clang/bin", root_build_dir) } cc = "${compiler_prefix}$prefix/clang" cxx = "${compiler_prefix}$prefix/clang++" readelf = "readelf" nm = "nm" ar = "ar" ld = cxx strip = "strip" toolchain_cpu = "x64" toolchain_os = "linux" is_clang = true }

No implications for x64 because clang_x64 is used for self-building for x64

Sorry, I didn't expand the context to see that it was in an arm-specific path.

dart-sdk is helpful

Interestingly they don't seem to use llvm-strip though; they just do prefixing in that file. Do you know why it's different?

@stuartmorgan

Thank you so much.

First of all, strip command is for x86_64. So, we should use another command for ARM64. dart-sdk uses llvm-objcopy. (source code)

I should have used llvm_objcopy. In addition, "build / toolchain / gcc_toolchain.gni" needed to be modified.

I modified it to use llvm_objcopy for now.

@stuartmorgan

I fixed. Could you review the source code when you have time?

HidenoriMatsubayashi · 2020-09-25T06:29:16Z

When I change from nm to llvm-nm command, the error happened.

  #nm = "nm"
  nm = "${prefix}/llvm-nm"

The error message

$ ninja -C out/linux_release_arm64 -v
ninja: Entering directory `out/linux_release_arm64'
[4282/6029] ../../buildtools/linux-x64/clang/bin/clang++ -shared -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now \
-Wl,-z,relro -Wl,-z,defs -pthread --target=aarch64-linux-gnu --sysroot=/home/0000131324/workspace/flutter/engine_work/src/\
build/linux/debian_sid_arm64-sysroot -L/home/0000131324/workspace/flutter/engine_work/src/build/linux/debian_sid_arm64-sys\
root/usr/local/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/home/0000131324/workspace/flutter/engine_work/src/build/linux/debia\
n_sid_arm64-sysroot/usr/local/lib/aarch64-linux-gnu\ -L/home/0000131324/workspace/flutter/engine_work/src/build/linux/debi\
an_sid_arm64-sysroot/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/home/0000131324/workspace/flutter/engine_work/src/build/linux\
/debian_sid_arm64-sysroot/lib/aarch64-linux-gnu\ -L/home/0000131324/workspace/flutter/engine_work/src/build/linux/debian_s\
id_arm64-sysroot/usr/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/home/0000131324/workspace/flutter/engine_work/src/build/linux\
/debian_sid_arm64-sysroot/usr/lib/aarch64-linux-gnu -Wl,-O2 -Wl,--gc-sections -Wl,--as-needed  -o ./libflutter_engine.so -\
Wl,--build-id=sha1 -Wl,-soname=libflutter_engine.so @./libflutter_engine.so.rsp && { readelf -d ./libflutter_engine.so | g\
rep SONAME ; ../../buildtools/linux-x64/clang/bin/llvm-nm -gD -f p ./libflutter_engine.so | cut -f1-2 -d' '; } > ./libflut\
ter_engine.so.tmp && if ! cmp -s ./libflutter_engine.so.tmp ./libflutter_engine.so.TOC; then mv ./libflutter_engine.so.tmp\
 ./libflutter_engine.so.TOC; fi
llvm-nm: for the --format option: Cannot find option named 'p'!
[4283/6029] touch obj/flutter/shell/platform/embedder/flutter_engine.stamp
[4284/6029] ../../buildtools/linux-x64/clang/bin/clang++ -shared -Wl,--fatal-warnings -fPIC -Wl,-z,noexecstack -Wl,-z,now \
-Wl,-z,relro -Wl,-z,defs -pthread --target=aarch64-linux-gnu --sysroot=/home/0000131324/workspace/flutter/engine_work/src/\
build/linux/debian_sid_arm64-sysroot -L/home/0000131324/workspace/flutter/engine_work/src/build/linux/debian_sid_arm64-sys\
root/usr/local/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/home/0000131324/workspace/flutter/engine_work/src/build/linux/debia\
n_sid_arm64-sysroot/usr/local/lib/aarch64-linux-gnu\ -L/home/0000131324/workspace/flutter/engine_work/src/build/linux/debi\
an_sid_arm64-sysroot/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/home/0000131324/workspace/flutter/engine_work/src/build/linux\
/debian_sid_arm64-sysroot/lib/aarch64-linux-gnu\ -L/home/0000131324/workspace/flutter/engine_work/src/build/linux/debian_s\
id_arm64-sysroot/usr/lib/aarch64-linux-gnu\ -Wl,-rpath-link=/home/0000131324/workspace/flutter/engine_work/src/build/linux\
/debian_sid_arm64-sysroot/usr/lib/aarch64-linux-gnu -Wl,-O2 -Wl,--gc-sections -Wl,--as-needed -L../../build/linux/debian_s\
id_arm64-sysroot/usr/lib/aarch64-linux-gnu  -o ./libflutter_linux_gtk.so -Wl,--build-id=sha1 -Wl,-soname=libflutter_linux_\
gtk.so @./libflutter_linux_gtk.so.rsp && { readelf -d ./libflutter_linux_gtk.so | grep SONAME ; ../../buildtools/linux-x64\
/clang/bin/llvm-nm -gD -f p ./libflutter_linux_gtk.so | cut -f1-2 -d' '; } > ./libflutter_linux_gtk.so.tmp && if ! cmp -s \
./libflutter_linux_gtk.so.tmp ./libflutter_linux_gtk.so.TOC; then mv ./libflutter_linux_gtk.so.tmp ./libflutter_linux_gtk.\
so.TOC; fi
llvm-nm: for the --format option: Cannot find option named 'p'!

HidenoriMatsubayashi · 2020-09-25T07:30:55Z

~~-f p is incorrect. This is the bug. -f posix is correct.~~
Considering LLVM, -f posix is better than -f p.

build/toolchain/gcc_toolchain.gni

toc_command = "{ $readelf -d $sofile | grep SONAME ; $nm -gD -f p $sofile | cut -f1-2 -d' '; } > $temporary_tocname"

$ man nm

      -f format
       --format=format
           Use the output format format, which can be "bsd", "sysv", or "posix".  The default is "bsd".  Only the first
           character of format is significant; it can be either upper or lower case.

stuartmorgan-g

Sorry, I forgot this part of the engine work was still outstanding. LGTM.

It looks like Dart is using llvm-objcopy for clang_x64 as well, so we should investigate whether to unify that at some point on our side, but limiting the changes to just arm64 here is the safer option.

stuartmorgan-g · 2020-11-03T18:38:21Z

Do you have a test PR in flutter/engine that uses this commit for the buildroot DEPS that is green, so we can be sure it's safe to land?

HidenoriMatsubayashi · 2020-11-04T01:34:45Z

It looks like Dart is using llvm-objcopy for clang_x64 as well, so we should investigate whether to unify that at some point on our side

I agree with you. I think it is better to use LLVM commands as well Dart-sdk if you have no specific reasons. The Linux desktop version is still the alpha version, and if you want to change it, I think it's better before the beta version. So, I'll investigate about clang_x64 after this work.

Do you have a test PR in flutter/engine that uses this commit for the buildroot DEPS that is green

I created the PR (flutter/engine#22284) just now.

stuartmorgan-g · 2020-11-04T17:04:05Z

I created the PR (flutter/engine#22284) just now.

Thanks! Everything is green there, so I'll go ahead and land this.

HidenoriMatsubayashi · 2020-11-04T22:59:50Z

Thanks.

flutter/buildroot#390

…r#390) Added cross-building Flutter Engine for ARM64 Linux platforms. This PR is part of ARM64 Linux support in the Flutter SDK.

Hidenori Matsubayashi added 4 commits August 3, 2020 11:27

support cross-build for linux aarch64

bacb214

support cross-build for linux aarch64

5658a5f

Merge remote-tracking branch 'origin/master' into topic-support-linux…

a29ffcf

…-aarch64-cross-build

support cross-build for linux arm64 host

187922a

This was referenced Aug 5, 2020

Add support of cross-building the engine for ARM64 Linux Host flutter/engine#20254

Merged

[flutter_tools] Add ARM64 Linux host and cross-building option support flutter/flutter#61221

Merged

stuartmorgan-g reviewed Aug 6, 2020

View reviewed changes

build/config/sysroot.gni Show resolved Hide resolved

HidenoriMatsubayashi changed the title ~~Add Linux Host ARM64 Support~~ Add support of cross-build the engine for ARM64 Linux Host Aug 6, 2020

HidenoriMatsubayashi changed the title ~~Add support of cross-build the engine for ARM64 Linux Host~~ Add support of cross-building the engine for ARM64 Linux Host Aug 6, 2020

Hidenori Matsubayashi added 2 commits August 17, 2020 11:04

Merge branch 'master' into topic-support-linux-aarch64-cross-build

3c1c4bd

Merge branch 'master' into topic-support-linux-aarch64-cross-build

5b7005c

stuartmorgan-g reviewed Sep 1, 2020

View reviewed changes

Hidenori Matsubayashi added 2 commits September 11, 2020 12:24

Merge branch 'master' into topic-support-linux-aarch64-cross-build

b2da7ca

Modified to match dart-sdk

22e8ed9

This was referenced Sep 13, 2020

Flutter sdk for Termux termux/termux-packages#5798

Closed

Building Flutter crashes Termux termux/termux-app#1247

Closed

Hidenori Matsubayashi added 2 commits September 17, 2020 09:22

Merge branch 'master' into topic-support-linux-aarch64-cross-build

fed5bc5

Changed to use llvm-ar command

834cf5e

Hidenori Matsubayashi added 4 commits September 25, 2020 16:39

Fix nm command option bug

1e69303

Changed to use llvm-nm command

23f5863

Merge branch 'master' into topic-support-linux-aarch64-cross-build

96528f1

Merge branch 'master' into topic-support-linux-aarch64-cross-build

157b4c7

stuartmorgan-g approved these changes Nov 3, 2020

View reviewed changes

HidenoriMatsubayashi mentioned this pull request Nov 4, 2020

[linux] Change the buildroot DEPS for Linux Arm64 support flutter/engine#22284

Merged

9 tasks

stuartmorgan-g merged commit 89b49b1 into flutter:master Nov 4, 2020

HidenoriMatsubayashi pushed a commit to HidenoriMatsubayashi/engine that referenced this pull request Nov 4, 2020

Add cross-building for Linux Arm64.

4c9ba39

flutter/buildroot#390

This was referenced Nov 13, 2020

[linux] Update buildroot DEPS flutter/engine#22484

Closed

Fix toolchain for linux x86_64 #412

Merged

Add support of cross-building the engine for ARM64 Linux Host #390

Add support of cross-building the engine for ARM64 Linux Host #390

Uh oh!

Conversation

HidenoriMatsubayashi commented Aug 5, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related PRs

Related Issues

Tests

1. Build Test

2. Run Flutter SDK and app on ARM64 Linux (Jetson Nano)

Uh oh!

Uh oh!

stuartmorgan-g Sep 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HidenoriMatsubayashi Sep 3, 2020

Choose a reason for hiding this comment

Uh oh!

stuartmorgan-g Sep 8, 2020

Choose a reason for hiding this comment

Uh oh!

stuartmorgan-g Sep 8, 2020

Choose a reason for hiding this comment

Uh oh!

HidenoriMatsubayashi Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HidenoriMatsubayashi Sep 11, 2020

Choose a reason for hiding this comment

Uh oh!

HidenoriMatsubayashi Sep 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HidenoriMatsubayashi commented Sep 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The error message

Uh oh!

HidenoriMatsubayashi commented Sep 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

build/toolchain/gcc_toolchain.gni

$ man nm

Uh oh!

stuartmorgan-g left a comment

Choose a reason for hiding this comment

Uh oh!

stuartmorgan-g commented Nov 3, 2020

Uh oh!

HidenoriMatsubayashi commented Nov 4, 2020

Uh oh!

stuartmorgan-g commented Nov 4, 2020

Uh oh!

HidenoriMatsubayashi commented Nov 4, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HidenoriMatsubayashi commented Aug 5, 2020 •

edited

Loading

stuartmorgan-g Sep 1, 2020 •

edited

Loading

HidenoriMatsubayashi Sep 11, 2020 •

edited

Loading

HidenoriMatsubayashi Sep 25, 2020 •

edited

Loading

HidenoriMatsubayashi commented Sep 25, 2020 •

edited

Loading

HidenoriMatsubayashi commented Sep 25, 2020 •

edited

Loading