pkgsMusl.cargo: fix build

[alyssais]

Description of changes

At long last, this finally fixes the build of even quite complex Rust packages with pkgsMusl.

Upstream rustc still assumes that musl = static. The fix for this is to disable crt-static by default for musl targets.

For every package apart from Cargo, we can fix this by just patching rustc to not have crt-static by default. But Cargo is built with the upstream bootstrap binary for rustc, which we can't easily patch. This means we need to find another way to make sure crt-static is not used during the build of pkgsMusl.cargo.

By default, Cargo doesn't apply Rust flags when building build.rs if --target is passed. The only good fix for this is to use the unstable -Zhost-config Cargo feature, which allows us to specify flags that should be passed to rustc when building for the (in Nixpkgs parlance) build system (e.g. for build.rs scripts).

Unfortunately, Cargo does not support e.g. a [host.'cfg(target_env = "musl")'] key (it only supports it for "target"), so we have to patch this conditionally to avoid it messing with non-Musl targets.

I haven't taken most of @yu-re-ka's patches from #186466, just because they weren't strictly necessary to get them to work. They're probably still worth looking at.

I've tested fd, pkgsMusl.fd, and pkgsStatic.fd. I checked that the static one is still static, and that the pkgsMusl one is dynamically linked.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[zowoq]

Can you add code comments for these please?

[yu-re-ka]

Could make this conditional to stdenv.hostPlatform.isMusl too, since otherwise it changes the behavior for people using the pkgsStatic.buildPackages.rustc manually. They would now have to manually set +crt-static to get a static binary, which they might not expect since upstream is explicitly keeping the default.

[yu-re-ka]

Actually I don't think it's necessary. Most people who do this kind of stuff manually will use rustup.

[alyssais]

I think it's a good idea not to change the upstream behaviour where a user hasn't explicitly requested it, so I'm going to make this change. But I think it should be stdenv.targetPlatform so it's correct when building a musl cross compiler.

[yu-re-ka]

But I think it should be stdenv.targetPlatform so it's correct when building a musl cross compiler.

Not sure about this. When the compiler is a cross-compiler from glibc to musl, the build.rs scripts will also run on glibc and this setting does not matter.
If I remember correctly what hostPlatform and targetPlatform are...

For example, targetPlatform.isMusl would be true for pkgsStatic.buildPackages.rustc, even though the change is not needed there?

Or the opposite, if we are building a cross-compiler from musl to glibc, targetPlatform.isMusl will be false, but it should still apply the change because the build.rs will be run with musl.

[alyssais]

Not sure about this. When the compiler is a cross-compiler from glibc to musl, the build.rs scripts will also run on glibc and this setting does not matter.
If I remember correctly what hostPlatform and targetPlatform are...

But this is the change for the default value of crt-static for all Musl builds in rustc, not the one for build.rs scripts in Cargo.

For example, targetPlatform.isMusl would be true for pkgsStatic.buildPackages.rustc, even though the change is not needed there?

Yes, the actual check I would do here is stdenv.targetPlatform.isMusl && !stdenv.targetPlatform.isStatic.

Or the opposite, if we are building a cross-compiler from musl to glibc, targetPlatform.isMusl will be false, but it should still apply the change because the build.rs will be run with musl.

Not sure about this one. I'll have to test it.

[alyssais]

Or the opposite, if we are building a cross-compiler from musl to glibc, targetPlatform.isMusl will be false, but it should still apply the change because the build.rs will be run with musl.

Not sure about this one. I'll have to test it.

I tested this by building pkgsMusl.pkgsCross.aarch64-multiplatform.{fd,cargo}, and both built fine even without this change. I don't entirely understand why, but it seems like just enabling it when targetPlatform is non-static Musl works fine.

[yu-re-ka]

This is a much nicer solution. Thanks!

[risicle]

Builds for me, nixos x86_64

[lovesegfault]

Great investigation and creative solution!

Could you replicate your rationale/reasoning as a comment in the postPatch of cargo.nix?

Then add a c.f. in the substituteInPlace that the reader should go look in the cargo drv for an explanation.

Just trying to make sure we don't add more Chesterton's fences :P

[alyssais]

Update on this PR: I'm trying to fix cross-compilation of rustc first, so I can check I got the build/host/target in this PR right. Then I'll come back round to this.

[alyssais]

Just trying to make sure we don't add more Chesterton's fences :P

I will do, but IMO it's not really a Chesterton's fence if it's explained in this amount of detail in the commit message. :)

[alyssais]

New version:

• Changes the Cargo fix to apply to the build platform rather than the host platform. (Verified experimentally with pkg-config: set pkg-config-rs-compatible env vars #196333) that this is correct.
• Adds the much-requested comments.
• Applies the rustc fix only for dynamically-linked musl targets, to avoid unexpectedly changing upstream behaviour in other circumstances.
• (Re)sets the linker in the Cargo config override, which fixes cross-compiling from musl.

Builds I have tested (from x86_64) (with #196333 merged in):

• pkgsMusl.fd
• pkgsMusl.cargo
• pkgsStatic.fd
• pkgsMusl.pkgsStatic.fd pkgsMusl.pkgsStatic.buildPackages.rustc fails to build, I think because it's trying to statically link proc macros. So maybe we will have to do something about the crt-static default for the build platform after all…
• cargo
• fd
• pkgsCross.musl64.cargo
• pkgsCross.musl64.fd
• pkgsCross.aarch64-multiplatform.cargo
• pkgsCross.aarch64-multiplatform.fd
• pkgsMusl.pkgsCross.aarch64-multiplatform.cargo
• pkgsMusl.pkgsCross.aarch64-multiplatform.fd
• pkgsMusl.pkgsCross.aarch64-multiplatform.pkgsStatic.fd

Nothing has failed yet. I'm just waiting for the remainder of the builds.

[alyssais]

Closing in favour of #198311.