top-level/packages-info: pre-evaluate output paths

[techninja1008]

packages.json is used by a variety of downstream services for a variety of purposes. Some of those services such as FloxHub and NixHub use it to enable an experience where a user can easily install packages by version, irrespective of nixpkgs versioning. A common optimization is that these services will pre-evaluate some or all of the packages in their index, to enable direct fetching from https://cache.nixos.org rather than having to locally evaluate nixpkgs.

This change modifies packages-info.nix so that the stubbed outputs section of a package's info, which would previously look like this:

{
  "debug": null,
  "out": null
}

Now looks like this:

{
  "debug": {
    "aarch64-linux": "/nix/store/bzb0vriv1a1kaz9zv6lzv62zkcvvm88r-python3-3.13.11-debug",
    "x86_64-linux": "/nix/store/gvhg8py0jr9v83ysqb86rr2s1yidppca-python3-3.13.11-debug"
  },
  "out": {
    "aarch64-linux": "/nix/store/ygzfhw5nxrn7qqfqmz2s9p6cikcjqqkh-python3-3.13.11",
    "x86_64-linux": "/nix/store/slhpx9glq7vl99bwi93bgrhn3syv98s1-python3-3.13.11"
  }
}

packages-info.nix (and make-tarball.nix) now takes in supportedSystems which it uses to determine which systems to evaluate.

This increases the time and memory usage of make-tarball, however in my testing it remains below 5 minutes, which is almost certainly worth it for the downstream optimization. This might necessitate re-tagging it as big-parallel as in #229132.

Things done

• Built on platform:
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• Tested, as applicable:
  • NixOS tests in nixos/tests.
  • Package tests at passthru.tests.
  • Tests in lib/tests or pkgs/test for functions and "core" functionality.
• Ran nixpkgs-review on this PR. See nixpkgs-review usage.
• Tested basic functionality of all binary files, usually in ./result/bin/.
• Nixpkgs Release Notes
  • Package update: when the change is major or breaking.
• NixOS Release Notes
  • Module addition: when adding a new NixOS module.
  • Module update: when the change is significant.
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

[techninja1008]

@vcunat do you mind taking a look at this? Thanks.

[vcunat]

A complication is that I don't really have an idea how packages.json is used.

[vcunat]

RAM usage will be quite high now. Roughly 60G after this PR, I'd say. (multiple of what it is) So if we do this, we certainly need that big-parallel flag. Also the total runtime increases to a multiple (of the 1 minute it was) and the packages.json.br bumps from 9.1 to 23.0 MiB.

I'd at least /cc @K900 for relations to PR #306095

[vcunat]

Maybe some consumers of the json won't be too happy about the increase? For example, does https://search.nixos.org/packages load this file (to everyone's browser) or something else? EDIT: it doesn't seem to be this dumb, from a couple minute's glance.

[MattSturgeon]

I don't yet have a strong opinion on whether this justifies the performance cost, but I've given the impl a quick once-over:

[MattSturgeon]

As of #492103 this has moved to pkgs/top-level/release-supported-systems.json

Suggested change

	supportedSystems ? builtins.fromJSON (builtins.readFile ../../ci/supportedSystems.json),
	supportedSystems ? builtins.fromJSON (builtins.readFile ./release-supported-systems.json),

[MattSturgeon]

Unsafe escaping/quoting is a bit of a pet peve of mine. Especially when going through multiple "layers" of syntax/evaluation.

Here we could use something like toVars, escapeShellArg, drv attrs, etc. But it'd be better to avoid the echo pipe entirely and instead write an actual JSON file that the nix expression can read.

E.g. using passAsFile or using __structuredAttrs and its $NIX_ATTRS_JSON_FILE file. That way the expression can do something like fromJSON (readFile $jsonFilePath).

Ideally we'd also avoid interpolating shell variables into the expression, using nix-instantiate --eval --expr '{src, json}: ...' --argstr src "$src" --argstr json "$jsonPath". Properly setting variables is much more robust than directly interpolating shell variables into the expression. That's a pre-existing issue though.

[MattSturgeon]

Would this flow more readably if the if was flipped?

Suggested change

	if builtins.all (v: v != builtins.currentSystem) supportedSystems then
	supportedSystems ++ [ builtins.currentSystem ]
	else
	supportedSystems;
	if builtins.elem builtins.currentSystem supportedSystems then
	supportedSystems
	else
	supportedSystems ++ [ builtins.currentSystem ];

[MattSturgeon]

Off-topic: a lot of this code could be simplified if we had access to a systemless instance of lib (i.e. import ../lib instead of pkgs.lib).

E.g. the above code is re-implementing lib.genAttrs and lib.optional.

[MattSturgeon]

This can be lib.genAttrs too.

[MattSturgeon]

Same, although pre-existing

[MattSturgeon]

Why the comment?

[MattSturgeon]

The or null will never trigger here. value is always present on a tryEval result, it defaults to value = false when success = false.

If you want to return null for unsuccessful evals, you need to check .success:

Suggested change

	value = (builtins.tryEval outPaths).value or null;
	value =
	let
	result = builtins.tryEval outPaths;
	in
	if result.success then result.value else null;

(You'd probably merge that let binding into your existing let block, of course.)

[jopejoe1]

RAM usage will be quite high now. Roughly 60G after this PR, I'd say. (multiple of what it is) So if we do this, we certainly need that big-parallel flag. Also the total runtime increases to a multiple (of the 1 minute it was)

Note that this gets build from scratch in every GitHub CI run and every nixpkgs/nixos Hydra jobset, so this was design with performance in mind, the transformation we did on outputs was only done because that's the format what nix-env spits out and that I did not yet have time to investigate what of our infra is making use of that. I will probably remove that transformation close to the next NixOS release so that we don't have such a long time when we need to support 2 versions of the format.

the packages.json.br bumps from 9.1 to 23.0 MiB.

Unrelated tangent: I also was not happy about the design how the cpe's where add also increased the size by more than it should have.