packages-config.nix: don't index stuff thats not getting evaled by jopejoe1 · Pull Request #434501 · NixOS/nixpkgs

jopejoe1 · 2025-08-17T16:03:47Z

Eval or remove all the package sets that had special treatment in packages-config.nix and only use it to filter stuff out from search like minimal-bootstrap and tests.

Things done

Add a 👍 reaction to pull requests you find important.

wolfgangwalther

While I like the general direction, this gives us 24924 new packages to build for hydra. I don't think we can do it like this.

pkgs/os-specific/windows/default.nix

pkgs/top-level/all-packages.nix

wolfgangwalther · 2025-08-17T16:14:39Z

pkgs/top-level/all-packages.nix

Commenting here, but that's for the whole commit: This also causes all these packages to additionally be built by hydra, right? I think that's a massive number of packages added.

I don't think we should or even can do this for all of these.

I see that you want to include all of these in Eval, and that makes sense. At the same time, we'd have to ensure that they're not built in Hydra, though.

It seems like most of these come from rPackages so it's probably fine to just leave that one out for now and see what can be done about that.

rPackages was disabled in ccd1029, like the others, too. The discussion for this was in #12203. It was also noted:

I suppose we could disable the builds for R since those packages tend to compile very quickly

So maybe, build-wise, adding R packages wouldn't be that much of a problem. Not sure. Eval times would certainly go up, though. For GHA, this is an increase of 15 seconds, from 2:30 to 2:45 right now. That should work.

I can't estimate what the consequences for Hydra would be, however.

@vcunat you were involved in that issue above back then as well. WDYT? Are we ready to enable rPackages again or not?

See also #126934

rPackages was disabled in ccd1029, like the others, too. The discussion for this was in #12203. It was also noted:

Note that some packages that have recurseIntoAttrs (e.g. ocamlPackages) were disabled this way, too. So it is also worth checking (though it may be hard with the time that has passed) why dontRecurseIntoAttrs was added for these package sets. I suspect that eval time on a simple nix-env (which is the stable way to list packages) was a factor, too.

rPackages was disabled in ccd1029, like the others, too. The discussion for this was in #12203. It was also noted:

I suppose we could disable the builds for R since those packages tend to compile very quickly

So maybe, build-wise, adding R packages wouldn't be that much of a problem. Not sure. Eval times would certainly go up, though. For GHA, this is an increase of 15 seconds, from 2:30 to 2:45 right now. That should work.

I can't estimate what the consequences for Hydra would be, however.

@vcunat you were involved in that issue above back then as well. WDYT? Are we ready to enable rPackages again or not?

I would very much like to see rPackages re-enabled if there's sufficient capacity. With it disabled, we have very little visibility of the state of the tree as the r-updates jobset has also been disabled in hydra since last year. Currently I'm running a hydra instance on my desktop to help with bumping rPackges while maintining a good overall state.

I've attached a plot using data I took from my hydra instance (buildoutputs table in postgres) showing the build times for packages. It shows that the vast majority of builds in the tree are quite small, with 99% of jobs finishing in under 60s. Evaluation seems to take ~ 108s.

See also #126934

I opened a similar PR for ocamlPackages a few months ago, #406555, but that one also included some cleanup so that it still passes current CI standards, and most of the other ones were done in #405039.

Let's split the commit that adds all the recurseIntoAttrs into separate commits in separate PRs - one for each package set that is touched. This will allow us to see rebuild numbers for each of these changes. Then we can judge these case-by-case.

I suppose we could disable the builds for R since those packages tend to compile very quickly

So maybe, build-wise, adding R packages wouldn't be that much of a problem.

For hydra.nixos.org, it seems build time of each packages affects much less the throughput than the number of packages to build. In other words, many small packages are much harder than a few large packages for hydra.nixos.org.

This is our experience for emacsPackages. emacsPackages are 6k (for one platform) small packages. Building them takes 30 minutes on my desktop. However, due to its large number, we were kindly asked by the infra team to target staging. (I totally support infra team's request and I am sorry we added some extra work for them. I mention these only to help the discussion here: estimating the potential influence of this PR.)

wolfgangwalther · 2025-08-17T16:18:51Z

pkgs/top-level/packages-config.nix

Not sure why this is removed. Isn't this required to get emacsPackages to show up on the search?

emacsPackages is behind config.allowAliases, which does not seem like something that should appear in search.

I don't understand the conclusion, tbh. Maybe your sentence is missing a "not"? I still wouldn't agree with it, but it would at least make more sense.

This file (packages-config.nix) sets allowAliases = false. This means that aliases will by default not show up. If emacsPackages is an alias - it won't show up. But emacs.pkgs is an attribute on a derivation, so won't be recursed into either. That means the emacs packages are not showing up anywhere this way.

Also consider:

nix-repl> emacsPackages.recurseForDerivations error: attribute 'recurseForDerivations' missing at «string»:1:1: 1| emacsPackages.recurseForDerivations | ^

(even if the alias was considered, or emacsPackages was not an alias to begin with, it still wouldn't recurse into that attrset)

The correct solution would probably be to move it out of aliases.nix so that it exists when allowAliases is false, as I see everything behind that as being deprecated, and if that is not the case for emacsPackages it should not be there.

nixpkgs/pkgs/top-level/aliases.nix

Line 639 in cefe143

emacsPackages = emacs.pkgs; # Added 2025-03-02

It has been intentionally moved there. emacsPackages in search is a workaround so that the package set gets indexed in search at all since it is impossible to get search to index emacs.pkgs. We do need this unless nix-env/search.nixos.org is sufficiently adapted.

That sounds like emacsPackages should not be in aliases.nix but in all-packages.nix

But emacs.pkgs is an attribute on a derivation, so won't be recursed into either. That means the emacs packages are not showing up anywhere this way.

Also consider:

nix-repl> emacsPackages.recurseForDerivations error: attribute 'recurseForDerivations' missing at «string»:1:1: 1| emacsPackages.recurseForDerivations | ^

First, we want to keep emacsPackages as a deprecated alias of emacs.pkgs and remove emacsPackages in the future.

Second, we want emacsPackages (emacs.pkgs actually) to be shown on search. And they actually are shown on search now.

nix-repl> emacs.pkgs.recurseForDerivations true

wolfgangwalther · 2025-08-17T16:20:50Z

pkgs/top-level/packages-config.nix

I don't think we can ultimately do that last change. For example, I'd like to add postgresqlPackages here, too, eventually (just didn't get to it, yet). You can currently search only for postgresql17Packages (and all other version numbers, specifically), but I argue it should be the other way around: No need to expose all the packages 5 times for different PG versions, it would be better if we only exposed postgresqlPackages in the search.

So adding the numbered ones here, like I left minimal-bootstrap and tests for, should work for those too.

Adding these here would only remove them, but not add postgresqlPackages itself, which is not showing up right now.

In this case it actually would, as postgresqlPackages is only not added to search because it is just a repeat of postgresql17Packages, and if those no longer exist, it would be found in search.

Oh, that would certainly explain my confusion about why postgresqlPackages is not showing up :)

Also, seems like lix is way faster with NIX_STATE_DIR=$TMPDIR NIX_PATH= time nix-env -f ../.. -qa --meta --json --show-trace --arg config 'import ./pkgs/top-level/packages-config.nix' > packages.json which gets used to generate the index for search.nixos.org.

On nix (Nix) 2.28.4: 370.68user 45.55system 7:00.56elapsed 98%CPU (0avgtext+0avgdata 8870956maxresident)k 0inputs+792480outputs (0major+2326175minor)pagefaults 0swaps
On nix (Lix, like Nix) 2.91.3: 28.24user 4.69system 0:33.22elapsed 99%CPU (0avgtext+0avgdata 8969504maxresident)k 0inputs+803856outputs (0major+2353707minor)pagefaults 0swaps

Yes, that regression is fixed with Nix 2.30.

I wonder whether the deduplication you mentioned is related to that performance difference... and maybe postgresqlPackages would show up automatically, if the Nix version used for building this tarball was changed?

We use Nix 2.30 in GHA for performance, but I think Hydra does not use it. It probably uses 2.28 or 2.29... and that's where the results for search are coming from.

I tested it with 2.30.2 and that still does the deduping.

This actually does not use the nix defined by Hydra, but its independently declared one:

nixpkgs/pkgs/top-level/make-tarball.nix

Line 23 in e4f2852

nix.out

pkgs/top-level/packages-config.nix

wolfgangwalther

Maybe we should split the PR for the moment and put the first 3 commits in a separate one? These should be uncontroversial, so we could do them already.

pkgs/os-specific/windows/msvcSdk/default.nix

jopejoe1 · 2025-08-25T18:27:55Z

Moved them into #436840 and #436841

these package set where already recoursed into

…eval

jopejoe1 · 2025-08-26T17:23:47Z

Split them up into:

jopejoe1 · 2026-01-27T15:24:40Z

This got split into seprate prs so no need for this anymore

jopejoe1 requested a review from wolfgangwalther August 17, 2025 16:03

jopejoe1 changed the base branch from master to staging August 17, 2025 16:04

nixpkgs-ci bot closed this Aug 17, 2025

nixpkgs-ci bot reopened this Aug 17, 2025

nix-owners bot requested review from maralorn and sternenseemann August 17, 2025 16:06

jopejoe1 force-pushed the no-speacial-treatment-for-search branch from 0d434b0 to 2040d66 Compare August 17, 2025 16:11

wolfgangwalther reviewed Aug 17, 2025

View reviewed changes

sternenseemann reviewed Aug 17, 2025

View reviewed changes

pkgs/top-level/packages-config.nix Outdated Show resolved Hide resolved

jopejoe1 force-pushed the no-speacial-treatment-for-search branch from 2040d66 to 3ecc9e2 Compare August 19, 2025 10:49

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Aug 23, 2025

windows.mingw_w64_pthreads: put deprecation behind alias

fbe17ff

jopejoe1 force-pushed the no-speacial-treatment-for-search branch from 3ecc9e2 to 343193f Compare August 25, 2025 18:01

wolfgangwalther reviewed Aug 25, 2025

View reviewed changes

pkgs/os-specific/windows/msvcSdk/default.nix Outdated Show resolved Hide resolved

nixpkgs-ci bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Aug 25, 2025

This was referenced Aug 25, 2025

windows: fix eval if ci enabled #436840

Merged

packages-config.nix: remove redundant package sets #436841

Merged

jopejoe1 added 5 commits August 25, 2025 20:34

windows.sdk: move throw behind meta.license check

1079516

packages-config.nix: remove redundant package sets

1cdfcf6

these package set where already recoursed into

packages-config.nix: eval stuff that's appearing on search.nixos.org

1f4ee10

packages-config.nix: move emcasPackages back to all-packages.nix and …

288d13a

…eval

packages-config.nix: only use to remove packages from search

04948a4

jopejoe1 force-pushed the no-speacial-treatment-for-search branch from 343193f to 04948a4 Compare August 25, 2025 18:34

nixpkgs-ci bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Aug 26, 2025

jopejoe1 closed this Jan 27, 2026

jopejoe1 deleted the no-speacial-treatment-for-search branch January 27, 2026 15:24

Uh oh!

Conversation

jopejoe1 commented Aug 17, 2025

Things done

Uh oh!

wolfgangwalther left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sternenseemann Aug 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jopejoe1 Aug 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jopejoe1 Aug 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wolfgangwalther Aug 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jian-lin Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wolfgangwalther left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jopejoe1 commented Aug 25, 2025

Uh oh!

jopejoe1 commented Aug 26, 2025 • edited by wolfgangwalther Loading Uh oh! There was an error while loading. Please reload this page.

sternenseemann Aug 17, 2025 •

edited

Loading

jopejoe1 Aug 19, 2025 •

edited

Loading

jopejoe1 Aug 17, 2025 •

edited

Loading

wolfgangwalther Aug 17, 2025 •

edited

Loading

jian-lin Aug 27, 2025 •

edited

Loading

jopejoe1 commented Aug 26, 2025 •

edited by wolfgangwalther

Loading