How to cross compile release binaries?

[hackfisher]

No description provided.

[freehere107]

https://github.com/japaric/rust-cross
compiler-explorer/compiler-explorer#496

rustup target list
rustup target add arm-unknown-linux-gnueabihf
cargo build --target=arm-unknown-linux-gnueabihf

[hackfisher]

cargo build --target x86_64-unknown-linux-gnu

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.0.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.1.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.10.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.11.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.12.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.13.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.14.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.15.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.2.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.3.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.4.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.5.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.6.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.7.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.8.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.9.rcgu.o" "-o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libsysinfo-7d0ee781c0b8d28e.so" "-Wl,--version-script=/var/folders/2n/b25hfjqn227bg1cgqjr08kj80000gn/T/rustcKZT0gC/list" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.4ep3dw2c68gm0jq8.rcgu.o" "-Wl,--gc-sections" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps" "-L" "/Users/denny/github.com/darwinia-network/darwinia/target/debug/deps" "-L" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libdoc_comment-aa01f56b96fcebba.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/librayon-19f3ef13bb6a7649.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/librayon_core-421b9fa8e4f5e77a.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libnum_cpus-257b1d90c65c1eed.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libcrossbeam_queue-601f56069536aeb7.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libeither-2f7041046007e0b6.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libcrossbeam_deque-d4efff8313c4bd1f.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libcrossbeam_epoch-b44e45285075a88e.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libscopeguard-c32e5bb4f5a6f264.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libmemoffset-b10fd94f22f1a52e.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libcrossbeam_utils-72aaeb9eafcb3c0c.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/liblazy_static-a00d277a0feb98ab.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libspin-fbcbce6cad2e3a9c.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libarrayvec-574e931460d9c48d.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libnodrop-87bad162b081c3c7.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/liblibc-191c2dc454fc2fab.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libcfg_if-55e3df711ce7a053.rlib" "-Wl,--start-group" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-9d237d17ebcf33da.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-c7aeb564d27a8408.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-8dcce133820ce36b.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-550595cd0e8605f6.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-616d93738996b317.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-0bdc7ca6876dfe08.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-5f6229b11bb8dfe3.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-74acf7fdf307aa94.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-08e0675cab0aedce.rlib" "-Wl,--end-group" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-ccce53ba085ea72e.rlib" "-Wl,-Bdynamic" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil" "-shared"
  = note: ld: unknown option: --as-needed
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          

error: aborting due to previous error

error: Could not compile `sysinfo`.
warning: build failed, waiting for other jobs to finish...
error: build failed

[hackfisher]

https://blog.rust-lang.org/2016/05/13/rustup.html

[freehere107]

It need Linux-toolchain, https://github.com/crosstool-ng/crosstool-ng

https://github.com/japaric/rust-cross#i-want-to-build-binaries-for-linux-mac-and-windows-how-do-i-cross-compile-from-linux-to-mac

[AurevoirXavier]

There is a lot to think about. Some of them are mentioned in the rust-cross.

The problem ld: unknown option: --as-needed is caused by linker not found, see:

• rustc on Mac OS X: "unknown option --as-needed" rust-lang/rust#34282
• https://stackoverflow.com/questions/54627521/cross-compile-for-mips-mt7688-openwrt-on-osx-mipsel-unknown-linux-musl
• https://users.rust-lang.org/t/re-solved-cross-compilation-from-mac-os-x-using-steed/9709

Furthermore, I would like to point out that (different) compiler do some optimization for (different) CPU architecture. For the best performance, just build it on the target machine. Also you can specify RUSTFLAGS="-Ctarget-cpu=<target-cpu>" (it might lose a little performance?).

[hackfisher]

https://timryan.org/2018/07/27/cross-compiling-linux-binaries-from-macos.html

[hackfisher]

I encountered some errors under target "x86_64-unknown-linux-gnu"
Maybe use docker for cross compile is a more practical and general solution:

https://github.com/dockcross/dockcross
https://www.ev3dev.org/docs/tutorials/using-docker-to-cross-compile/

In function `core::ptr::real_drop_in_place':
          ed25519_dalek.3c2jd33s-cgu.2:(.text._ZN4core3ptr18real_drop_in_place17h5a28c7b3b94ddc9bE+0x17): undefined reference to `clear_on_drop_hide'
          /Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/release/deps/libed25519_dalek-6c7900b42b0dfebc.rlib(ed25519_dalek-6c7900b42b0dfebc.ed25519_dalek.3c2jd33s-cgu.2.rcgu.o): In function `ed25519_dalek::ed25519::Keypair::sign':
          ed25519_dalek.3c2jd33s-cgu.2:(.text._ZN13ed25519_dalek7ed255197Keypair4sign17hf67a89f2782be6f2E+0x135): undefined reference to `clear_on_drop_hide'
          /Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/release/deps/libed25519_dalek-6c7900b42b0dfebc.rlib(ed25519_dalek-6c7900b42b0dfebc.ed25519_dalek.3c2jd33s-cgu.7.rcgu.o):ed25519_dalek.3c2jd33s-cgu.7:(.text._ZN74_$LT$ed25519_dalek..secret..SecretKey$u20$as$u20$core..ops..drop..Drop$GT$4drop17hfa6f9c67876178a9E+0x10): more undefined references to `clear_on_drop_hide' follow
          collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

[AurevoirXavier]

undefined references means that the ld can't find the symbols which was generated by the compiler.

In this case, your TARGET_CC is x86_64-unknown-linux-gnu-gcc. And the symbols come from x86_64-apple-darwin.

You need to:

1. compile all dependencies (openssl, rocksdb, zlib, zstd ...) to dylib(.so) on the target-machine
2. copy them to the build-machine
3. specify the LD_LIBRARY_PATH, DYLD_LIBRARY_PATH which contains the foreign system's lib:

   export LD_LIBRARY_PATH=/path/to/x86_64-unknown-linux-gnu/lib
   export DYLD_LIBRARY_PATH=/path/to/x86_64-unknown-linux-gnu/lib

4. specify the openssl dir manually:

   export X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR=/path/to/openssl
   export X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR=/path/to/openssl/lib
   export X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR=/path/to/openssl/include

5. specify the linker:

   export TARGET=x86_64-unknown-linux-gnu
   export TARGET_CC=${TARGET}-gcc
   export TARGET_CXX=${TARGET}-g++
   export TARGET_LD=${TARGET}-ld
   export TARGET_AS=${TARGET}-as
   export TARGET_AR=${TARGET}-ar

tips:
The ld require the same libc.so, libc++.so which use for compiling the rocksdb lib to find the symbols to link.

---

It's much easier to build with the x86_64-unknown-linux-musl, but MUST mark the openssl as vendored. sfackler/rust-openssl#980

---

x86_64-unknown-linux-gnu from:

brew tap SergioBenitez/osxct
brew install x86_64-unknown-linux-gnu

x86_64-unknown-linux-musl from:

brew install FiloSottile/musl-cross/musl-cross

~/.cargo/config:

[target.x86_64-unknown-linux-musl]
linker = "x86_64-linux-musl-gcc"

[target.x86_64-unknown-linux-gnu]
linker = "x86_64-unknown-linux-gnu-gcc"

[AurevoirXavier]

We can compile openssl to static library with x86_64-unknown-linux-musl. So we solve the previous problem!

Unfortunately, the compiling will fail at rocksdb.

After I read the gcc book. It's almost impossible to compile with x86_64-unknown-linux-musl. Cause the rocksdb heavily dependent on c++!!

So I focus on x86_64-unknown-linux-gnu.

1. brew install x86_64-unknown-linux-gnu give us a gcc-7.2. To keep the same ENV with target machine, I build my own cross-compile-toolchain gcc-8.3
2. to solve the undefine symbols, we need to copy some foreign system's lib to our $LD_LIBRARY_PATH
3. build the openssl shared (NOT static) lib in target machine or build it with our cross-compile-toolchain(gcc-8.3)
4. build the rocksdb shared (NOT static) lib in target machine, because it need some linux kernel headers and (...). It's hard to compile on host machine as I said before
5. export ENV VARS:

   export TARGET=x86_64-unknown-linux-gnu
   export TARGET_CC=${TARGET}-gcc
   export OPENSSL_INCLUDE_DIR=/Users/xavier/Documents/program/darwinia/linux-x86_64/include
   export OPENSSL_LIB_DIR=/Users/xavier/Documents/program/darwinia/linux-x86_64/lib/openssl
   export ROCKSDB_LIB_DIR=/Users/xavier/Documents/program/darwinia/linux-x86_64/lib/rocksdb

6. in .cargo/config:

   [target.x86_64-unknown-linux-gnu]
   linker = "x86_64-unknown-linux-gnu-gcc"

7. cargo build --release --target=x86_64-unknown-linux-gnu It just take 4 mins!
8. simply copy it to target machine and run it!!

Q: Why export?
A: Because we use x86_64-unknown-linux-gnu that it won't embed the library in binary. You also need other libs. More detail check the Dependencies.

Q: Why export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/root/rocksdb" on target machine?
A: Because I build the rocksdb myself, it was not in my $LD_LIBRARY_PATH.

[sekisamu]

👍