buildRustPackage: only set RUSTFLAGS if non-empty

[HeitorAugustoLN]

Closes #464392

Previously, the refactor of buildRustPackage caused RUSTFLAGS to be defined in the environment unconditionally, often resulting in an empty string (RUSTFLAGS="").

When RUSTFLAGS is set (even to an empty string), Cargo prioritizes it and ignores CARGO_TARGET_<TRIPLE>_RUSTFLAGS. This behavior broke packages like cosmic-comp that rely on target-specific flags for linking libraries (e.g., libEGL).

This change calculates the flags beforehand and uses lib.optionalAttrs to ensure RUSTFLAGS is only added to the environment if it contains a value. This restores the ability for Cargo to fall back to target-specific configuration variables.

tested with:

diff --git a/pkgs/by-name/ru/rustcat/package.nix b/pkgs/by-name/ru/rustcat/package.nix
index 65a241e99bc2..285452eaf751 100644
--- a/pkgs/by-name/ru/rustcat/package.nix
+++ b/pkgs/by-name/ru/rustcat/package.nix
@@ -3,6 +3,7 @@
   fetchFromGitHub,
   rustPlatform,
   versionCheckHook,
+  libglvnd,
 }:

 rustPlatform.buildRustPackage rec {
@@ -24,6 +25,18 @@ rustPlatform.buildRustPackage rec {

   versionCheckProgram = [ "${placeholder "out"}/bin/rcat" ];

+  buildInputs = [ libglvnd ];
+
+  env = {
+    CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS = toString (
+      map (arg: "-C link-arg=" + arg) [
+        "-Wl,--push-state,--no-as-needed"
+        "-lEGL"
+        "-Wl,--pop-state"
+      ]
+    );
+  };
+

And its getting linked:

❯ , readelf -d ./result/bin/rcat
Dynamic section at offset 0x1735d0 contains 32 entries:
  Tag                Type           Name/Value
  0x0000000000000001 (NEEDED)       Shared library: [libc.so.6]
  0x0000000000000001 (NEEDED)       Shared library: [ld-linux-x86-64.so.2]
  0x0000000000000001 (NEEDED)       Shared library: [libgcc_s.so.1]
  0x0000000000000001 (NEEDED)       Shared library: [libEGL.so.1]
  0x000000000000001d (RUNPATH)      Library runpath: [/nix/store/mm4v6w3wj8wi5qc6rwdrb2rkqs08whp9-libglvnd-1.7.0/lib:/nix/store/xx7cm72qy2c0643cm1ipngd87aqwkcdp-glibc-2.40-66/lib:/nix/store/xm08aqdd7pxcdhm0ak6aqb1v7hw5q6ri-gcc-14.3.0-lib/lib]

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

All good now, thank you!

[Aleksanaa]

Good, but this doesn't change the fact that platform-specific RUSTFLAGS will lose effect once RUSTFLAGS is set 🫠 Only bringing us back to previous status

[HeitorAugustoLN]

Good, but this doesn't change the fact that platform-specific RUSTFLAGS will lose effect once RUSTFLAGS is set 🫠 Only bringing us back to previous status

Right, but I am pretty sure that this is standard cargo behavior, so there's not much we can do about it

[thefossguy]

I think the relevant issue is upstream: rust-lang/cargo#13981
TL;DR: We specify --target (I think always) and that causes CARGO_TARGET_${TRIPLET}_RUSTFLAGS to not be respected.

[thefossguy]

Should we start using our own flags RUSTFLAGS_TARGET and RUSTFLAGS_HOST which then are merged? That would help mitigate overwriting RUSTFLAGS already defined for cross compilation.

[Aleksanaa]

Should we start using our own flags RUSTFLAGS_TARGET and RUSTFLAGS_HOST which then are merged? That would help mitigate overwriting RUSTFLAGS already defined for cross compilation.

Good idea, try if it works for cosmic?

[Aleksanaa]

This is the same as #464707

[Aleksanaa]

Should we start using our own flags RUSTFLAGS_TARGET and RUSTFLAGS_HOST which then are merged? That would help mitigate overwriting RUSTFLAGS already defined for cross compilation.

I don't see these variables exist?

[HeitorAugustoLN]

Closing in favor of #464707