Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to cross compile release binaries? #21

Closed
hackfisher opened this issue Jul 2, 2019 · 10 comments
Closed

How to cross compile release binaries? #21

hackfisher opened this issue Jul 2, 2019 · 10 comments
Labels
U-Question [Uncategorized] Further information/discussion is requested

Comments

@hackfisher
Copy link
Contributor

No description provided.

@hackfisher hackfisher added the U-Question [Uncategorized] Further information/discussion is requested label Jul 2, 2019
@freehere107
Copy link
Contributor

https://github.com/japaric/rust-cross
compiler-explorer/compiler-explorer#496

rustup target list
rustup target add arm-unknown-linux-gnueabihf
cargo build --target=arm-unknown-linux-gnueabihf

@hackfisher
Copy link
Contributor Author

cargo build --target x86_64-unknown-linux-gnu
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.0.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.1.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.10.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.11.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.12.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.13.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.14.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.15.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.2.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.3.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.4.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.5.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.6.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.7.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.8.rcgu.o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.sysinfo.26bgz1f1-cgu.9.rcgu.o" "-o" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libsysinfo-7d0ee781c0b8d28e.so" "-Wl,--version-script=/var/folders/2n/b25hfjqn227bg1cgqjr08kj80000gn/T/rustcKZT0gC/list" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/sysinfo-7d0ee781c0b8d28e.4ep3dw2c68gm0jq8.rcgu.o" "-Wl,--gc-sections" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps" "-L" "/Users/denny/github.com/darwinia-network/darwinia/target/debug/deps" "-L" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libdoc_comment-aa01f56b96fcebba.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/librayon-19f3ef13bb6a7649.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/librayon_core-421b9fa8e4f5e77a.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libnum_cpus-257b1d90c65c1eed.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libcrossbeam_queue-601f56069536aeb7.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libeither-2f7041046007e0b6.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libcrossbeam_deque-d4efff8313c4bd1f.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libcrossbeam_epoch-b44e45285075a88e.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libscopeguard-c32e5bb4f5a6f264.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libmemoffset-b10fd94f22f1a52e.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libcrossbeam_utils-72aaeb9eafcb3c0c.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/liblazy_static-a00d277a0feb98ab.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libspin-fbcbce6cad2e3a9c.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libarrayvec-574e931460d9c48d.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libnodrop-87bad162b081c3c7.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/liblibc-191c2dc454fc2fab.rlib" "/Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/debug/deps/libcfg_if-55e3df711ce7a053.rlib" "-Wl,--start-group" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-9d237d17ebcf33da.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-c7aeb564d27a8408.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libbacktrace_sys-8dcce133820ce36b.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-550595cd0e8605f6.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-616d93738996b317.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-0bdc7ca6876dfe08.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-5f6229b11bb8dfe3.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-74acf7fdf307aa94.rlib" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-08e0675cab0aedce.rlib" "-Wl,--end-group" "/Users/denny/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-ccce53ba085ea72e.rlib" "-Wl,-Bdynamic" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil" "-shared"
  = note: ld: unknown option: --as-needed
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
          

error: aborting due to previous error

error: Could not compile `sysinfo`.
warning: build failed, waiting for other jobs to finish...
error: build failed

@hackfisher
Copy link
Contributor Author

@AurevoirXavier
Copy link
Member

AurevoirXavier commented Aug 13, 2019

There is a lot to think about. Some of them are mentioned in the rust-cross.

The problem ld: unknown option: --as-needed is caused by linker not found, see:

Furthermore, I would like to point out that (different) compiler do some optimization for (different) CPU architecture. For the best performance, just build it on the target machine. Also you can specify RUSTFLAGS="-Ctarget-cpu=<target-cpu>" (it might lose a little performance?).

@hackfisher
Copy link
Contributor Author

@hackfisher
Copy link
Contributor Author

I encountered some errors under target "x86_64-unknown-linux-gnu"
Maybe use docker for cross compile is a more practical and general solution:

https://github.com/dockcross/dockcross
https://www.ev3dev.org/docs/tutorials/using-docker-to-cross-compile/

In function `core::ptr::real_drop_in_place':
          ed25519_dalek.3c2jd33s-cgu.2:(.text._ZN4core3ptr18real_drop_in_place17h5a28c7b3b94ddc9bE+0x17): undefined reference to `clear_on_drop_hide'
          /Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/release/deps/libed25519_dalek-6c7900b42b0dfebc.rlib(ed25519_dalek-6c7900b42b0dfebc.ed25519_dalek.3c2jd33s-cgu.2.rcgu.o): In function `ed25519_dalek::ed25519::Keypair::sign':
          ed25519_dalek.3c2jd33s-cgu.2:(.text._ZN13ed25519_dalek7ed255197Keypair4sign17hf67a89f2782be6f2E+0x135): undefined reference to `clear_on_drop_hide'
          /Users/denny/github.com/darwinia-network/darwinia/target/x86_64-unknown-linux-gnu/release/deps/libed25519_dalek-6c7900b42b0dfebc.rlib(ed25519_dalek-6c7900b42b0dfebc.ed25519_dalek.3c2jd33s-cgu.7.rcgu.o):ed25519_dalek.3c2jd33s-cgu.7:(.text._ZN74_$LT$ed25519_dalek..secret..SecretKey$u20$as$u20$core..ops..drop..Drop$GT$4drop17hfa6f9c67876178a9E+0x10): more undefined references to `clear_on_drop_hide' follow
          collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

@AurevoirXavier
Copy link
Member

AurevoirXavier commented Aug 25, 2019

undefined references means that the ld can't find the symbols which was generated by the compiler.

In this case, your TARGET_CC is x86_64-unknown-linux-gnu-gcc. And the symbols come from x86_64-apple-darwin.

You need to:

  1. compile all dependencies (openssl, rocksdb, zlib, zstd ...) to dylib(.so) on the target-machine
  2. copy them to the build-machine
  3. specify the LD_LIBRARY_PATH, DYLD_LIBRARY_PATH which contains the foreign system's lib:
    export LD_LIBRARY_PATH=/path/to/x86_64-unknown-linux-gnu/lib
    export DYLD_LIBRARY_PATH=/path/to/x86_64-unknown-linux-gnu/lib
  4. specify the openssl dir manually:
    export X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR=/path/to/openssl
    export X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR=/path/to/openssl/lib
    export X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR=/path/to/openssl/include
  5. specify the linker:
    export TARGET=x86_64-unknown-linux-gnu
    export TARGET_CC=${TARGET}-gcc
    export TARGET_CXX=${TARGET}-g++
    export TARGET_LD=${TARGET}-ld
    export TARGET_AS=${TARGET}-as
    export TARGET_AR=${TARGET}-ar

tips:
The ld require the same libc.so, libc++.so which use for compiling the rocksdb lib to find the symbols to link.


It's much easier to build with the x86_64-unknown-linux-musl, but MUST mark the openssl as vendored. sfackler/rust-openssl#980


x86_64-unknown-linux-gnu from:

brew tap SergioBenitez/osxct
brew install x86_64-unknown-linux-gnu

x86_64-unknown-linux-musl from:

brew install FiloSottile/musl-cross/musl-cross

~/.cargo/config:

[target.x86_64-unknown-linux-musl]
linker = "x86_64-linux-musl-gcc"

[target.x86_64-unknown-linux-gnu]
linker = "x86_64-unknown-linux-gnu-gcc"

@AurevoirXavier
Copy link
Member

We can compile openssl to static library with x86_64-unknown-linux-musl. So we solve the previous problem!

Unfortunately, the compiling will fail at rocksdb.

After I read the gcc book. It's almost impossible to compile with x86_64-unknown-linux-musl. Cause the rocksdb heavily dependent on c++!!

So I focus on x86_64-unknown-linux-gnu.

  1. brew install x86_64-unknown-linux-gnu give us a gcc-7.2. To keep the same ENV with target machine, I build my own cross-compile-toolchain gcc-8.3
  2. to solve the undefine symbols, we need to copy some foreign system's lib to our $LD_LIBRARY_PATH
  3. build the openssl shared (NOT static) lib in target machine or build it with our cross-compile-toolchain(gcc-8.3)
  4. build the rocksdb shared (NOT static) lib in target machine, because it need some linux kernel headers and (...). It's hard to compile on host machine as I said before
  5. export ENV VARS:
    export TARGET=x86_64-unknown-linux-gnu
    export TARGET_CC=${TARGET}-gcc
    export OPENSSL_INCLUDE_DIR=/Users/xavier/Documents/program/darwinia/linux-x86_64/include
    export OPENSSL_LIB_DIR=/Users/xavier/Documents/program/darwinia/linux-x86_64/lib/openssl
    export ROCKSDB_LIB_DIR=/Users/xavier/Documents/program/darwinia/linux-x86_64/lib/rocksdb
  6. in .cargo/config:
    [target.x86_64-unknown-linux-gnu]
    linker = "x86_64-unknown-linux-gnu-gcc"
  7. cargo build --release --target=x86_64-unknown-linux-gnu It just take 4 mins!
  8. simply copy it to target machine and run it!!

Screen Shot 2019-09-06 at 13 32 31

Screen Shot 2019-09-06 at 14 18 29

Q: Why export?
A: Because we use x86_64-unknown-linux-gnu that it won't embed the library in binary. You also need other libs. More detail check the Dependencies.

Q: Why export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/root/rocksdb" on target machine?
A: Because I build the rocksdb myself, it was not in my $LD_LIBRARY_PATH.

@sekisamu
Copy link
Contributor

sekisamu commented Sep 6, 2019

👍

@hackfisher hackfisher removed this from the Darwinia Canary Net milestone Sep 6, 2019
hackfisher added a commit that referenced this issue Sep 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
U-Question [Uncategorized] Further information/discussion is requested
Projects
None yet
Development

No branches or pull requests

4 participants