tests.overriding: refactor and allow non-bool expected value

[ShamrockLee]

• Refactor test package generation:
  • Use Nix languages equality operator to compare expr and expected at evaluation time, instead of comparing their string-interpolation at build time.
  • Use structured attributes to pass testResults instead of dumping all the values into buildCommand.
  • Aside from passthru.tests, add passthru.failures for lib.runTest-generated list of failed tests, similar to lib/tests/misc.
• Refactor test cases
  • Change all tests in the form

    {
      expr = realExpr == realExpected;
      expected = true;
    }

    to

    {
      expr = realExpr;
      expected = realExpected;
    }

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[MattSturgeon]

Much better! I had the same thought of "why isn't expr compared with expected like in other tests" when I looked at this test.

Some thoughts/ideas/suggestions below to discuss, but nothing blocking:

[MattSturgeon]

I don't understand a) why we need two test frameworks here (this derivation and runTests) and b) why this is called failures.

If we include this passthru, isn't it just an alternative to the below, that also attempts to print the expr and expected results using toPretty?

[ShamrockLee]

isn't it just an alternative to the below, that also attempts to print the expr and expected results using toPretty?

IIUC, the design of tests.overriding avoids context from test cases, which would break if we print expr and expected in buildCommand.

tests.overriding is essentially a collection of evaluation tests for build helpers. But we need it as a package test, and therefore the package part. The name failures comes from a search across Nixpkgs with keyword "lib.runTests", and 3 in 5 result are failures = lib.runTests {.

I added it as an alternative to building the package (nix eval -f . tests.overriding.failures), which hopefully ease staging change testing. Still, having two test frameworks is kinda redundant, and we could just tell people to look at tests.overriding.tests.

[MattSturgeon]

If there's precedent for failures = runTests elsewhere, then I'm fine with having it. It just felt (feels) like a bit of an odd pattern.

[MattSturgeon]

One nit, otherwise LGTM

[MattSturgeon]

I squashed 57ad481a13faac26777246b9bd02c1e0dd48ab20. Hopefully the rebase also helps with the weird CI error we were seeing.

[MattSturgeon]

nixpkgs-review result

Generated using nixpkgs-review-gha

Command: nixpkgs-review pr 456848
Commit: ffca1c4197f52aa58e1c0e600b874bd0e00999df (subsequent changes)
Merge: 506ec31c6bddfa78305a8f6665bd8adb851a14ca

Logs: https://github.com/MattSturgeon/nixpkgs-review-gha/actions/runs/19120892935

---

x86_64-linux

✅ 1 package built:

• tests.overriding

---

aarch64-linux

✅ 1 package built:

• tests.overriding

---

x86_64-darwin (sandbox = true)

✅ 1 package built:

• tests.overriding

---

aarch64-darwin (sandbox = true)

✅ 1 package built:

• tests.overriding

[ShamrockLee]

Ah! It is already in release-25.11.