Enable stdenv customization through (buildPython*.override { inherit stdenv; })

[ShamrockLee]

Description of changes

Now that #366593 has restored the missing .override to buildPythonPackage and buildPythonApplication, this allow specifying the stdenv used by the build helpers through buildPython*.override (as most build helpers do) instead of passing it directly into buildPython*.

The reason to move to buildPythonPackage.override { stdenv = ...; } from buildPythonPackage { stdenv = ...; } includes:

1. The buildPython* { inherit stdenv; ...; } will be impossible/unfeasible when taking fixed-point arguments (buildPython* (finalAttrs: { ...; })). lib.extendMkDerivation is on its way, but we need a clean ground to integrate it into buildPython*.

2. stdenv is a dependency buildPythonPackage would (directly or indirectly) get from pkgs through callPackage. It makes better sense to change it through the .override attribute provided by callPackage.

3. In the future, we could get rid of all the special, impassable arguments, and pass everything properly either into stdenv.mkDerivation or through passthru. This way, we only need overrideAttrs to override attributes, without having to defined special overriders like overridePythonAttrs and for each language and framework. stdenv could neither be passed into stdenv.mkDerivation or through passthru, since stdenv.mkDerivaiton will add a stdenv attribute to the resulting derivation.

For package definition that used to be:

{
  lib,
  stdenv,
  buildPythonPackage,
  fetchPyPi,
}:
buildPythonPackage rec {
  inherit stdenv;
  pname = "...";
  version = "...";
  src = fetchPypi {
    inherit pname source;
    hash = "...";
  };
}

can now become

{
  lib,
  stdenv,
  buildPythonPackage,
  fetchPyPi,
}:
buildPythonPackage.override { inherit stdenv; } rec {
  pname = "...";
  version = "...";
  src = fetchPypi {
    inherit pname source;
    hash = "...";
  };
}

In the future, when buildPythonPackage also takes fixed-point arguments, we could write:

{
  lib,
  stdenv,
  buildPythonPackage,
  fetchPyPi,
}:
buildPythonPackage.override { inherit stdenv; } (finalAttrs: {
  pname = "...";
  version = "...";
  src = fetchPypi {
    inherit (finalAttrs) pname source;
    hash = "...";
  };
})

This PR depends on #366593.
Initial effort for #271387.

Things done

• Built on platform(s)
  • x86_64-linux (no rebuilds other than nixos-install-tools)
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

Priorities

Add a 👍 reaction to pull requests you find important.

[ShamrockLee]

Simplify a bit.

[ShamrockLee]

Reword the release note entry.

[MattSturgeon]

LGTM

[ShamrockLee]

@MattSturgeon I wrote some test cases locally (with the modularized Python tests.overriding test-cases cherry-picked from #267296), and they passed. Should I push it here or should we do it in a subsequent PR?

[ShamrockLee]

Anyway, let's merge before the all-breaking-changes restriction.

[MattSturgeon]

If the tests are testing this PR's changes and/or testing that this PR doesn't break existing behaviour, then pushing them to this PR probably makes sense.

[ShamrockLee]

Apology for not reacting earlier. I'll open a PR for the tests tomorrow morning (UTC+8).