NixOS test framework: add overriding methods

[roberth]

This is the first part of #299542, adding the test framework overriding machinery only, without the opinionated setPackage module.
These are useful primitives, regardless of how we'll end up using them in this repo.

The remainder of #299542 does not touch these additions.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• Nixpkgs 25.11 Release Notes (or backporting 24.11 and 25.05 Nixpkgs Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
• NixOS 25.11 Release Notes (or backporting 24.11 and 25.05 NixOS Release notes)
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other contributing documentation in corresponding paths.

---

Add a 👍 reaction to pull requests you find important.

[wolfgangwalther]

Reading through it, this LGTM.

I did not actually run the tests or try to use it, but #299542 had done so already, so confidence is still high.

[trofi]

Bisect says 63e2606 nixos.runTest: Add extend, overrideTestDerivation introduced attributes that fail to eval with error that can't be caught. For example:

$ nix eval --impure --expr 'with import ./. {}; bind.tests.kubernetes-dns-multi-node.extend'
error:
       … while evaluating a branch condition
         at nixos/tests/all-tests.nix:23:5:
           22|     val:
           23|     if isAttrs val then
             |     ^
           24|       if hasAttr "test" val then

       … while calling the 'isAttrs' builtin
         at nixos/tests/all-tests.nix:23:8:
           22|     val:
           23|     if isAttrs val then
             |        ^
           24|       if hasAttr "test" val then

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: function 'extend' called without required argument 'modules'
       at nixos/lib/testing/call-test.nix:31:7:
           30|     passthru.extend =
           31|       args@{
             |       ^
           32|         modules,

[wolfgangwalther]

Hm. .extend is a function, but is not expected to be "called" via command line like this. bind.tests.kubernetes-dns-multi-node is a derivation right? Why do you recurse into its attributes / where does that happen?

[trofi]

It happens as part of an evaluatability test of reachable attributes in nixpkgs I run time to time locally: https://trofi.github.io/posts/309-listing-all-nixpkgs-packages.html

I don't think extend is a valid function here. It does not get called via eval here either: it only gets evaluated (as a lambda) and fails.

$ nix repl -f .

nix-repl> builtins.isFunction bind.tests.kubernetes-dns-multi-node.extend
error:
       … while calling the 'isFunction' builtin
         at «string»:1:1:
            1| builtins.isFunction bind.tests.kubernetes-dns-multi-node.extend
             | ^

       … while evaluating a branch condition
         at nixos/tests/all-tests.nix:23:5:
           22|     val:
           23|     if isAttrs val then
             |     ^
           24|       if hasAttr "test" val then

       (stack trace truncated; use '--show-trace' to show the full, detailed trace)

       error: function 'extend' called without required argument 'modules'
       at nixos/lib/testing/call-test.nix:31:7:
           30|     passthru.extend =
           31|       args@{
             |       ^
           32|         modules,

[wolfgangwalther]

I see. This seems to happen, because discoverTests tries to call that function. This will affect all "old style" (handleTest / handleTestOn etc.) tests.

[roberth]

discoverTests and callTest. This is at least the third time they come up after a seemingly innocuous change.
When the runTest migration is complete we'll finally get rid of that, but for now I'll try and find a workaround again.