buildRustPackage: refactor to support finalAttrs pattern

[TomaSajt]

I'll rebase onto staging once this PR is reviewed.

Supersedes #194475
Supersedes #288430
Supersedes #340798

Related:

• buildDotnetModule: add finalAttrs support #331398 (not perfect, IMO)
• maven.buildMavenPackage: support finalAttrs and overrideAttrs #359573 (made by me, very similar)
• buildPython*: allow providing a finalAttrs function #370232 (has .overridePythonAttrs to also worry about...)
• (partially) WIP: experiment with a hook based alternative to buildGoModule #353526

This PR refactors the buildRustPackage wrapper around mkDerivation to support the finalAttrs pattern better.

A non-goal was having 0 rebuilds.
The rebuilds are because of more attrs actually being passed to mkDerivation, improving overridability without having to add an extra .overrideRustAttrs utility, which would complicate things a lot.

---

Other than just being able to do something like

rustPlatform.buildRustPackage (finalAttrs: {
  a = finalAttrs.cargoHash;
  b = finalAttrs.cargoDeps;
  c = finalAttrs.doCheck
})

you'll also be able to override derivations much easier:

foo.overrideAttrs {
  ...
  cargoHash = "...";
}

cargoLock is something that can only be inside passthru, since it can't be converted to a string.
Because of this, the input cargoLock is moved to passthru.cargoLock.
This means overriding is a bit painful, but at least it's possible.

foo.overrideAttrs (finalAttrs: prevAttrs: {
  passthru = prevAttrs.passthru or {} // {
    cargoLock = {
      lockfile = ./new-Cargo.lock;
    };
  };
})

This means that to override a package that uses cargoLock to use cargoHash, you will need to set passthru.cargoLock to null:

drv.overrideAttrs (prevAttrs: {
  cargoHash = "...";
  passthru = prevAttrs.passthru or {} // {
    cargoLock = null
  };
})

You may also just override cargoDeps directly, like you would have done before this PR

drv.overrideAttrs {
  cargoDeps = ...;
}

---

Note:

I dislike the following function pattern:
{ name ? "default", ... }@attrs: args // { inherit name; } // modifiedAttrs

I used the following instead:
attrs: defaultValues // lib.removeAttrs attrs namesToDelete // modifiedAttrs

This means that we lose the functionArgs information, but I think the current version is more readable.
Also functionArgs only makes sense if the input to buildRustPackage is not a function-recursive attrset.

---

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[9999years]

finalAttrs pattern with 0 rebuilds can target master while the overridability commit can't. Guess that won't be happening.

I'm not sure why you believe this. PRs to master are definitely allowed to cause rebuilds, even lots of rebuilds. Here's a PR that caused 5001+ Linux rebuilds (the same amount as this PR) that was merged an hour ago: #373784

Check the closed PRs for many, many more examples: https://github.com/NixOS/nixpkgs/issues?q=label%3A%2210.rebuild-linux%3A+5001%2B%22+is%3Aclosed

The CONTRIBUTING.md says that sometimes PRs with too many rebuilds should target a different branch, but gives no specific guidance for how many rebuilds that is:

From time to time, changes between branches must be rebased, for example, if the number of new rebuilds they would cause is too large for the target branch.

We should assume that it's on a case-by-case basis, and that 5001+ rebuilds is definitely not too many.

[TomaSajt]

I was not planning on splitting this up. But also, I was not planning on pointing this at master, since it's unnecessary load on the builders. And also, many people would not be able to use the cache for a time on master because of this.

Merging into master is understandable in case of important security fixes, which this is not.

[9999years]

Also, this is a duplicate of #194475, which has at least been reviewed to some extent.

[TomaSajt]

Also, this is a duplicate of #194475, which has at least been reviewed to some extent.

Yes, see the first line of this PR.

This PR is slightly better in the sense that it allows overriding the cargoLock. But yes, very similar.

There were a few things I disliked about that PR:

First, is the way it first passes and "invalid" attrset (still has cargoLock) to mkDerivation and then quickly uses .overrideAttrs on the derivation to change it.
Being more explicit and constructing the attrset outside of mkDerivation is more obvious.

Second, (this is mostly just an issue I had with the current builder), it still had the many "unnecesary" defaults. These are present to be able to pass something to the fetchers. I used the custom takeAttrs function to circumvent this, removing some of this bloat.

[9999years]

Yes, see the first line of this PR.

Oops, sorry, my bad 🤦‍♀️

[getchoo]

I'm not sure why you believe this. PRs to master are definitely allowed to cause rebuilds, even lots of rebuilds.

Quoting the contributing guide:

The staging workflow exists to batch Hydra builds of many packages together. [...] It works by directing commits that cause mass rebuilds to a separate staging branch that isn't directly built by Hydra.

And while the definition of mass rebuilds is not "formally defined", the guide does give us a marker for it by saying:

As a rule of thumb, if the number of rebuilds is over 500, it can be considered a mass rebuild

So while yes, this is on a case-by case (such as in the rsync patch above, which was done due to it being a security fix and staging-next being blocked), it's perfectly reasonable to assume mass rebuilds are only fit for staging a vast majority of the time (as is the case with nearly every PR in your search link)

Regarding this patch specifically, it is causing rebuilds of every Rust package and it's dependents -- totaling up to over 31k packages -- for something that doesn't actually change the behavior of binaries. I think Toma was correct in targeting staging here

[Shawn8901]

I'm not sure why you believe this. PRs to master are definitely allowed to cause rebuilds, even lots of rebuilds. Here's a PR that caused 5001+ Linux rebuilds (the same amount as this PR) that was merged an hour ago: #373784

Context: The rules are not applied strictly, to allow security sensitive prs go to master directly. The linked pr was fixing High Severity security issues (that allowed RCE). I assume that there everyone would agree that it should not merged to staging to receive the update in 2-3 weeks.

non security sensitive mass rebuilds in general should target staging, in the past such prs also have been reverted in the past (when merged directly to master) to not let hydra eat full rebuilds

[drupol]

I guess this PR can be closed now?

[TomaSajt]

I'll close it for now.
There might be some ideas in this PR that might be used for improvements.