ci/eval: fix local comparison with baseline

[wolfgangwalther]

Due to how we pass in existing store paths via CLI arguments for the diff and combine scripts, Nix didn't register a dependency on the store paths properly. This meant that some of the derivations that were built, didn't have the right store paths made available in the sandbox - leading to all kinds of "not found" errors.

We worked around this in CI by resolving the symlinks to the nix store beforehand. We tried to work around this locally by storing the nix store path in BASELINE, but this didn't fully work. By explicitly registering these store paths as dependencies, this should work across the board - without any magic required by the caller.

Follow up to #440895.

The error I would be getting would be a "file not found" in the python script to compare stats.

Things done

• Tested locally
• Fits CONTRIBUTING.md, pkgs/README.md, maintainers/README.md and other READMEs.

---

Add a 👍 reaction to pull requests you find important.

[wolfgangwalther]

Seems like this still doesn't work. It worked when I ran it on the same commit, but now it doesn't when I compare two different commits.

[wolfgangwalther]

OK, TIL about builtins.storePath. Rewrote the whole PR description (and the fix) - my previous approach was not even close.

This should now work correctly - and also remove the odd readlink workarounds we had in CI. Tested locally, let's see what CI says.

[Mic92]

looks simpler.

[wolfgangwalther]

Not sure whether it works 100% correctly, yet. The test workflow has this in its summary:

⚠️ Skipping comparison: stats directory is missing in the target commit.

[MattSturgeon]

I'm sure you're already aware of the different behaviour of path concatenation, interpolation, etc...

But just for my own understanding, and to try and understand the motivation behind the new/old impl, I'll ask the stupid questions anyway:

[wolfgangwalther]

I'm sure you're already aware of the different behaviour of path concatenation, interpolation, etc...

Let me assure you, that this is not the case :). This is all a big mystery to me!

[wolfgangwalther]

Not sure whether it works 100% correctly, yet. The test workflow has this in its summary:

⚠️ Skipping comparison: stats directory is missing in the target commit.

Ah. that's a cross-over between target branch and this PR: The eval "compare" step runs as nix-build nixpkgs/trusted/ci .... That means it uses the eval.compare code from the target branch, where the storepath change is not included. That's why the change to combinedDir doesn't take effect and the script can't find the stats folder.

I confirmed this works correctly when repeating these steps that CI does locally. Will be good after merge.

[MattSturgeon]

Another nice improvement! Thanks again @wolfgangwalther, and thanks for entertaining my belligerent questioning 😅

[nixpkgs-ci]

Successfully created backport PR for release-25.05:

• [Backport release-25.05] ci/eval: fix local comparison with baseline #441831