[WIP] Rust cross compilation by Mic92 · Pull Request #50866 · NixOS/nixpkgs

Mic92 · 2018-11-21T01:58:21Z

Status

rustc
fix non-cross rustc
buildRustPackage
some packages need fixes & testing

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Determined the impact on package closure size (by running nix path-info -S before and after)
Fits CONTRIBUTING.md.

Mic92 · 2018-11-21T01:59:01Z

Still compiling. Cross-compiled LLVM is also still broken.

GrahamcOfBorg · 2018-11-21T02:17:03Z

Success on x86_64-linux (full log)

Attempted: libpfm, openssl

Partial log (click to expand)

gzipping man pages under /nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man/share/man/
patching script interpreter paths in /nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man
checking for references to /build/ in /nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/q66rhjlqjkn194c6zji5sjmr3p6z1z6y-openssl-1.0.2p-debug
patching script interpreter paths in /nix/store/q66rhjlqjkn194c6zji5sjmr3p6z1z6y-openssl-1.0.2p-debug
checking for references to /build/ in /nix/store/q66rhjlqjkn194c6zji5sjmr3p6z1z6y-openssl-1.0.2p-debug...
cannot link '/nix/store/.links/0jv8zns4raaa2vc5gjwxqm9n5wx9qi6lbv8556dg6zg80aw3z0yr' to '/nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man/share/man/man3/EVP_PKEY_derive_set_peer.3.gz': No space left on device
cannot link '/nix/store/.links/0jv8zns4raaa2vc5gjwxqm9n5wx9qi6lbv8556dg6zg80aw3z0yr' to '/nix/store/m9izkl3p2hr6xngm529jl1gy0ql3gj42-openssl-1.0.2p-man/share/man/man3/EVP_PKEY_derive_init.3.gz': No space left on device
/nix/store/k4dhk7hqqy3kzmgdvs0j5pi88gfjr4pp-libpfm-4.10.1
/nix/store/pl3km7jksa9r6zrvrwsh5djaxgk004s0-openssl-1.0.2p-bin

GrahamcOfBorg · 2018-11-21T02:19:06Z

Success on aarch64-linux (full log)

Attempted: libpfm, openssl

Partial log (click to expand)

checking for references to /build/ in /nix/store/818azw7kbva6jznsj438h05lh33r83wv-openssl-1.0.2p...
shrinking RPATHs of ELF executables and libraries in /nix/store/995mh5wj5dayr041hjldyw2a9wp94qr5-openssl-1.0.2p-man
gzipping man pages under /nix/store/995mh5wj5dayr041hjldyw2a9wp94qr5-openssl-1.0.2p-man/share/man/
patching script interpreter paths in /nix/store/995mh5wj5dayr041hjldyw2a9wp94qr5-openssl-1.0.2p-man
checking for references to /build/ in /nix/store/995mh5wj5dayr041hjldyw2a9wp94qr5-openssl-1.0.2p-man...
shrinking RPATHs of ELF executables and libraries in /nix/store/8arzdc863bwrqx50jv441vqfxfbxpjgg-openssl-1.0.2p-debug
patching script interpreter paths in /nix/store/8arzdc863bwrqx50jv441vqfxfbxpjgg-openssl-1.0.2p-debug
checking for references to /build/ in /nix/store/8arzdc863bwrqx50jv441vqfxfbxpjgg-openssl-1.0.2p-debug...
/nix/store/d8g0sv493ar86xvz97bc6ndlv6h3l5g4-libpfm-4.10.1
/nix/store/7y00xv5kwkjvy3pq4npq6finarpyxwc5-openssl-1.0.2p-bin

matthewbauer · 2018-11-21T02:43:35Z

pkgs/data/sgml+xml/schemas/xml-dtd/docbook/generic.nix

This is a actually a bug I think. "propagatedNativeBuildInputs" should be correct. Will have a fix soon hopefully.

I guess I can wait a bit, I just get the things working properly and not blocking on nitpicks.

findXMLCatalogs does sound like a tool that should be propagatedNativeBuildInputs.

pkgs/development/libraries/libpfm/default.nix

matthewbauer · 2018-11-21T02:55:16Z

pkgs/development/libraries/openssl/default.nix

I didn't think this would be a problem now that #49608 is merged. Do you know where this comes from? It also looks like openssl supports passing in HASHBANGPERL to do this explicity.

HASHBANGPERL is only supported in openssl 1.1, but we still use openssl 1.0.2 in most places.

Also #49608 was reverted again in staging by @LnL7

Yeah, the stdenv was broken. Since this is the second time I think that change has a bit too much impact to merge into staging directly, we should run it through a separate jobset first to get an overview of the breakages.

I probably don't need this change to make cross-compiling rust packages work. So I will split it of probably later.

GrahamcOfBorg · 2018-11-21T03:18:10Z

Timed out, unknown build status on x86_64-darwin (full log)

Attempted: openssl

The following builds were skipped because they don't evaluate on x86_64-darwin: libpfm

Partial log (click to expand)

ld: warning: directory not found for option '-L../bootstrap/x86_64-apple-darwin/usr/lib'
ld: warning: undefined base symbol '__TMC15SwiftFoundation19_NSCFConstantString' for alias '___CFConstantStringClassReference'
installing
post-installation fixup
strip is /nix/store/2ws9cmamvr7xyvdg4d2nnd1bmr1zjrrq-bootstrap-tools/bin/strip
patching script interpreter paths in /nix/store/awcr3pzcd5vpqwh3lhb0ijxbv72vf9l7-swift-corefoundation
cannot build derivation '/nix/store/4xqfqm5lbdqpb9qds1xxrzmyss3818h3-stdenv-darwin.drv': 7 dependencies couldn't be built
cannot build derivation '/nix/store/mv8zixg5sy9lqnqnn15ca4wl2vxipii6-perl-5.28.0.drv': 3 dependencies couldn't be built
cannot build derivation '/nix/store/7lipgdn8jvyy7k45idq1sh17k7bpj9z3-openssl-1.0.2p.drv': 2 dependencies couldn't be built
error: build of '/nix/store/7lipgdn8jvyy7k45idq1sh17k7bpj9z3-openssl-1.0.2p.drv' failed

Mic92 · 2018-11-21T12:00:18Z

Mhm, I currently stuck at this fix-point stuff that is used to create buildRustPackage. It somehow puts a non-native rustc/cargo into nativeBuildInputs:

nix-repl> pkgsCross.aarch64-multiplatform.fd.nativeBuildInputs
[ «derivation /nix/store/d9abfxjb0icbjg7k8j62pasgdj7s8zwl-git-2.19.1.drv» «derivation /nix/store/gnvkwv8mls6a2n4bpw3xdpp6fh1d1lyc-cargo-1.30.0-aarch64-unknown-linux-gnu.drv» «derivation /nix/store/hibvax8p73n9imhi9ix7z55ka5hl47ai-rustc-1.30.1-aarch64-unknown-linux-gnu.drv» ]

Mic92 · 2018-11-21T14:11:40Z

Build inputs are clean now.

$ nix-repl> pkgsCross.aarch64-multiplatform.fd.nativeBuildInputs
[ «derivation /nix/store/zp9aqlm1gg9k2hz990yvvwfgja3ybyiy-cargo-1.30.0.drv» «derivation /nix/store/4sjcxw65jn8l90ql2ivn2znlpw24fc09-rustc-1.30.1.drv» «derivation /nix/store/d9abfxjb0icbjg7k8j62pasgdj7s8zwl-git-2.19.1.drv» «derivation /nix/store/8ns0q8dc8bcxyc178ck6zcmhp39rvhwa-nss-cacert-3.40.drv» ]

I think most cross-compile fixes from this pr are actually not needed.
I will separate them later into a new pull request.

Mic92 · 2018-11-25T21:46:36Z

Compilation works now!

$ rustc --target=aarch64-unknown-linux-gnu foo.rs
$ file foo
foo: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/1r1pdrdwgj6k3wy3bhrxjiqjdxf9z2f5-glibc-2.27-aarch64-unknown-linux-gnu/lib/ld-linux-aarch64.so.1, for GNU/Linux 2.6.32, with debug_info, not stripped

Mic92 · 2018-11-28T09:20:11Z

The first working buildRustPackage!

$ file result/bin/fd
result/bin/fd: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /nix/store/1r1pdrdwgj6k3wy3bhrxjiqjdxf9z2f5-glibc-2.27-aarch64-unknown-linux-gnu/lib/ld-linux-aarch64.so.1, for GNU/Linux 2.6.32, not stripped

pkgs/build-support/rust/default.nix

Ericson2314 · 2018-11-28T23:28:07Z

pkgs/development/compilers/llvm/7/llvm.nix

Can we do this for the other LLVMs?

I will split this off into a different pull request: #51226

Mic92 · 2018-12-18T12:46:25Z

pkgs/development/compilers/rust/rustc.nix

+
+    "${setBuild}.jemalloc=${hostJemalloc}/lib/libjemalloc_pic.a"
+    "${setHost}.jemalloc=${hostJemalloc}/lib/libjemalloc_pic.a"
+    "${setTarget}.jemalloc=${targetJemalloc}/lib/libjemalloc_pic.a"


@Ericson2314 this one seems actually undefined, when I try to build nix-build default.nix -A pkgsCross.aarch64-multiplatform.rustc

Yeah that's OK. For now. I need to change something in booter.nix to avoid these issues. Need to throw in a buildPackages to get the cross compiler vs cross compiler compiler, which is the immediate need anyways.

Ericson2314 · 2019-01-02T17:13:47Z

Is this just blocked on ofborg?

Mic92 · 2019-01-02T19:01:31Z

It also requires to fix a few cargo packages that depend on ./target/release to be present, as now cross-compiled packages are put into a architecture subdirectory that can be accessed by using $releaseDir

Ericson2314 · 2019-01-02T19:10:42Z

If you want to list those, I'm happy to divide + conquer on fixing that + eval failures.

Mic92 · 2019-01-02T19:50:19Z

I will not have a look at this the next few days, so you could just fix the
few packages right away as it is only string-replacement in 11 packages. (target/release -> $releaseDir)

$ ag -l 'target/release'
pkgs/development/tools/rust/racerd/default.nix
pkgs/development/tools/misc/ycmd/default.nix
pkgs/build-support/rust/default.nix
pkgs/tools/text/ripgrep/default.nix
pkgs/tools/misc/vdirsyncer/default.nix
pkgs/tools/misc/fd/default.nix
pkgs/tools/misc/bat/default.nix
pkgs/applications/altcoins/zcash/librustzcash/default.nix
pkgs/applications/networking/dyndns/cfdyndns/default.nix
pkgs/applications/misc/alacritty/default.nix
pkgs/applications/window-managers/wtftw/default.nix

This should probably also mentioned in the changelog since this also affects out-of-tree rust packages.

Ericson2314 · 2019-01-18T15:44:20Z

We talked briefly about this on IRC. I think it should action be changed back to target/release since we only install for a single platform at once. This could be done with some fixup phase thing.

illegalprime · 2019-02-19T16:31:40Z

should armv7l-unknown-linux-gnueabihf be mapped to armv7-unknown-linux-gnueabihf (since that's the triple rust supports)?
also re: target/release, is it just a matter of adding:

"mv ${releaseDir} target/"

after cargo build?
this looks great BTW, can't wait!

matthewbauer · 2019-02-21T01:41:02Z

pkgs/build-support/rust/default.nix

      cargoDepsCopy="$sourceRoot/${cargoVendorDir}"
    '';

+  ccForBuild="${buildPackages.stdenv.cc}/bin/${buildPackages.stdenv.cc.targetPrefix}cc";


The build cc is almost never prefixed. I suppose there could be some weird case where it needs to be though.

illegalprime · 2019-02-22T06:23:27Z

I rebased this on the latest nixos-unstable and did the renaming for target/release as a postBuild hook, maybe it'll help:
fa82ebc...illegalprime:rust-cross
still testing but fd seems to build, ripgrep will build after #54510 i believe

Mic92 · 2019-02-22T08:17:59Z

@illegalprime can you just open a new pull request?
Then we can continue from there.

Mic92 · 2019-02-22T08:19:01Z

We can get rid of $releaseDir in this case I think:
fa82ebc...illegalprime:rust-cross#diff-2c7a7586fcf2d0d0d778bdddcf893a1bR124

Mic92 · 2019-02-22T08:20:23Z

and because people might have overwritten postBuild this could be moved to buildPhase itself.

illegalprime · 2019-02-22T20:02:52Z

will do! I believe pkgs.pkgsCross.aarch64-multiplatform.glibc is broken on staging so I'm trying to bisect. i'll post a PR when i have some concrete information on what works or not

Mic92 · 2019-02-22T22:10:37Z

You don't need to bisect. This was a commit where I introduced C.UTF-8: #56143 (comment)
The build process of this locale puts a non-native libc into the library path unfortunately.

vcunat · 2019-02-22T22:29:22Z

We could temporarily drop the new locale when cross-compiling or something like that, right? EDIT: I mean, we worked without it until recently, so it shouldn't be too bad...

Mic92 · 2019-02-22T22:31:15Z

@vcunat that might work also it would break other packages that assumes we have it, which might be python packages rather soon.

Mic92 · 2019-02-22T22:32:48Z

the problem is $(rtld-prefix) in the glibc makefile, which contains the host ld binary and is prefixed in the locale generation.

Mic92 · 2019-02-22T22:37:08Z

It is not too complicated to call localedef manually:
http://www.linuxfromscratch.org/clfs/view/clfs-2.0/arm/cross-tools/glibc.html
They seems to differentiate between endianness though by a flag, which we can query from our stdenv too.

illegalprime · 2019-02-22T22:38:39Z

thanks for linking that @Mic92!

bisecting is weird, maybe I don't understand it enough or all of the merge commits are making it complicated but this didn't find the right commit:

git bisect start nixpkgs/staging nixpkgs-channels/nixos-unstable
git bisect run nix build -f . pkgs.pkgsCross.aarch64-multiplatform.glibc

however this did:

git bisect start nixpkgs/staging HEAD~600
git bisect run nix build -f . pkgs.pkgsCross.aarch64-multiplatform.glibc

picking 600 commits in the past was just arbitrary and I'm lucky it found the commit , but how should one do this bisect in the future? do i have to flatten out the history to do it right or is my understanding just poor?

vcunat · 2019-02-23T08:21:05Z

Are you sure that nixpkgs-channels/nixos-unstable was a good commit (at that time)? I had bisected across merges many times and haven't noticed any issue with the algorithm, except that merges complicate the whole concept.

vcunat · 2019-02-23T08:24:47Z

it would break other packages that assumes we have it

Well, better than breaking glibc build already :-) It's why I meant it as a temporary workaround. The "assumes we have it" part won't be portable to non-glibc anyway, but perhaps those don't need it or something...

Mic92 · 2019-02-23T09:27:23Z

@vcunat this should be the proper fix instead: #56235

Mic92 · 2019-03-13T11:32:33Z

Superseded by #56540

Mic92 requested a review from LnL7 as a code owner November 21, 2018 01:58

Mic92 changed the title ~~Rust cross compilation~~ [WIP] Rust cross compilation Nov 21, 2018

matthewbauer reviewed Nov 21, 2018

View reviewed changes

pkgs/development/libraries/libpfm/default.nix Outdated Show resolved Hide resolved

matthewbauer reviewed Nov 21, 2018

View reviewed changes

Mic92 force-pushed the rust-cross branch from 45d9f1d to 9723ba9 Compare November 21, 2018 14:09

Mic92 force-pushed the rust-cross branch from 9723ba9 to c37efea Compare November 25, 2018 00:52

GrahamcOfBorg removed the 10.rebuild-darwin-stdenv This PR causes stdenv to rebuild on Darwin and must target a staging branch. label Nov 25, 2018

Mic92 force-pushed the rust-cross branch from 00f0bc7 to 97d2e9b Compare November 25, 2018 21:45

Mic92 force-pushed the rust-cross branch from 97d2e9b to 4116129 Compare November 25, 2018 21:47

Mic92 mentioned this pull request Nov 25, 2018

Call for Content: 2018/13 NixOS/nixos-weekly#71

Merged

Mic92 force-pushed the rust-cross branch from 4116129 to 7a7313e Compare November 26, 2018 08:26

GrahamcOfBorg added the 8.has: package (new) This PR adds a new package label Nov 26, 2018

Mic92 force-pushed the rust-cross branch from af33ed7 to debf6f3 Compare November 28, 2018 09:19

Mic92 mentioned this pull request Nov 28, 2018

[WIP] Port switch-to-configuration script to Python #48378

Closed

9 tasks

Ericson2314 reviewed Nov 28, 2018

View reviewed changes

pkgs/build-support/rust/default.nix Outdated Show resolved Hide resolved

Ericson2314 reviewed Nov 28, 2018

View reviewed changes

Mic92 force-pushed the rust-cross branch from 41c0e0e to 7769066 Compare December 18, 2018 10:51

rust: fix cross-compilation

bceda58

Mic92 force-pushed the rust-cross branch from 7769066 to bceda58 Compare December 18, 2018 12:44

Mic92 commented Dec 18, 2018

View reviewed changes

matthewbauer reviewed Feb 21, 2019

View reviewed changes

ryantm added the 2.status: work-in-progress label Feb 24, 2019

illegalprime mentioned this pull request Feb 28, 2019

Rust Cross Compilation (1.32.0) #56540

Merged

24 tasks

Mic92 closed this Mar 13, 2019

Mic92 deleted the rust-cross branch March 13, 2019 11:32

Uh oh!

Conversation

Mic92 commented Nov 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Status

Things done

Uh oh!

Mic92 commented Nov 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GrahamcOfBorg commented Nov 21, 2018

Uh oh!

GrahamcOfBorg commented Nov 21, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GrahamcOfBorg commented Nov 21, 2018

Uh oh!

Mic92 commented Nov 21, 2018

Uh oh!

Mic92 commented Nov 21, 2018

Uh oh!

Mic92 commented Nov 25, 2018

Uh oh!

Mic92 commented Nov 28, 2018

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ericson2314 commented Jan 2, 2019

Uh oh!

Mic92 commented Jan 2, 2019

Uh oh!

Ericson2314 commented Jan 2, 2019

Uh oh!

Mic92 commented Jan 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ericson2314 commented Jan 18, 2019

Uh oh!

illegalprime commented Feb 19, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

illegalprime commented Feb 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Mic92 commented Feb 22, 2019

Uh oh!

Mic92 commented Feb 22, 2019

Uh oh!

Mic92 commented Feb 22, 2019

Uh oh!

illegalprime commented Feb 22, 2019

Uh oh!

Mic92 commented Feb 22, 2019

Uh oh!

vcunat commented Feb 22, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Mic92 commented Nov 21, 2018 •

edited

Loading

Mic92 commented Nov 21, 2018 •

edited

Loading

Mic92 commented Jan 2, 2019 •

edited

Loading

illegalprime commented Feb 22, 2019 •

edited

Loading

vcunat commented Feb 22, 2019 •

edited

Loading