NixOS: use overlays when nixpkgs.pkgs is set

[roberth]

Motivation for this change

In the current situation, when specifying nixpkgs.pkgs in NixOS, the nixpkgs.overlays option is ignored. Now that we have pkgs.extend since #47430, we can actually support the combination.

It makes the nixpkgs options a bit more predictable and when pkgs.nixosTest #47684 is merged, this will enable project-specific NixOS tests to reuse pkgs while also supporting test machine-specific overlays.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Fits CONTRIBUTING.md.

---

[matthewbauer]

I'm not so sure about this. While I see the use case of setting pkgs and overlays, there's no reason you can't set overlays yourself when you init pkgs. Since we don't do this for config, crossSystem, or localSystem, it seems a little unexpected. We probably should have an assertion for those so that you don't set pkgs and something that has no effect.

[roberth]

I was going to add the assertion when I ran into #49765. I actually wrote the assertion yesterday, but forgot to push it here. I don't think it is necessary to add the other assertion about the host system.

[roberth]

Regarding whether we should, this elegantly enables a use case pointed out by aszlig for nixosTest where one of the machines has an extra overlay. So that is one reason why you can't set overlays yourself when you init pkgs.

We could support the other options but I can't tell at this point how well that would work.

[nbp]

Sounds good to me.