ci/eval: allow local comparison with rebuilds

[wolfgangwalther]

This allows running a full comparison between two commits locally.

What was previously eval.full is now called eval.all. The new eval.full takes a baseline argument for the comparison.

Fixes #437697 (comment)

@philiptaron could you give this a test-run? I only did some basic testing with "baseline == HEAD", but I think this should work with your example as well.

Things done

• Basic testing.
• More testing by @philiptaron
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[wolfgangwalther]

Ah, mh, something is not working properly, yet. the before/ directory in the combined-eval result is still empty.

[wolfgangwalther]

Ah, it's that readlink stuff again. So currently, you'd need to do this:

nix-build ci -A eval.full --arg baseline $(readlink ./baseline)

on the second step. Maybe we should fix this once and for all - but I have no idea where the ./baseline symlink is not followed properly...

[wolfgangwalther]

The problem is that, ./baseline is a symlink. And when we pass that into eval, it will be transformed into a path via builtins.path, which copies the symlink to the store. But a symlink in the store will only help when the target of that symlink is also made available to the builder that tries to access it. But that's not the case, thus this will fail.

When using readlink, the link will be resolved before calling nix, nix will identify the store path passed and correctly make that store path available inside the builder.

I don't think we can properly solve this here :/

However, I changed the README slightly. It now has these two commands:

BASELINE=$(nix-build ci -A eval.all --no-out-link)

nix-build ci -A eval.full --arg baseline $BASELINE

This works because it doesn't use another symlink.

[philiptaron]

This works because it doesn't use another symlink.

That's precisely the fix I was going to suggest. Thanks Wolfgang.

[philiptaron]

Approved with some comments and gripes.

[philiptaron]

There are some non-README.md docs in this ci/eval/compare/default.nix file that are also relevant.

[nixpkgs-ci]

Successfully created backport PR for release-25.05:

• [Backport release-25.05] ci/eval: allow local comparison with rebuilds #437965