pythonPackages: ensure all derivations provide python modules by FRidh · Pull Request #194205 · NixOS/nixpkgs

FRidh · 2022-10-03T10:25:43Z

This adds a test to ensure no new uses of buildPythonApplication can be added to python-packages.nix.

Python packages can be grouped into two groups: 1) applications and 2) packages providing importable modules. In python-packages.nix we only want to have 2). 1) should be in the top-level package set.

Description of changes

Things done

mweinelt · 2022-10-03T11:13:16Z

Wondering if the eval failure is the result of the one remaining application package: openrazer-daemon. If so, this is pretty hard to glean from the trace.

#194095

FRidh · 2022-10-03T11:32:40Z

Ha, no, this test fails because I forgot about setuphooks. In this case the wrapPython hook:

error: wrapPython should use `buildPythonPackage` or `toPythonModule` if it is to be part of the Python packages set.

I wonder how to exempt those...

We really need types in Nixpkgs so we can check them and filter based on them.

SuperSandro2000 · 2022-10-03T18:12:59Z

Is there an eval penalty to this? and if so how high would it be?

FRidh · 2022-10-04T08:33:33Z

Is there an eval penalty to this? and if so how high would it be?

Additional eval load is negligible for this.

We really need types in Nixpkgs so we can check them and filter based on them.

Other solutions:

whitelist
mark setup hooks as such and allow hooks

Some packages to consider as well:

python (the interpreter), I think we could use toPythonModule on it.
recursivePthLoader. Should get rid of this.

FRidh · 2022-10-25T13:40:28Z

~~Cross is broken with this change.~~

This adds a test to ensure no new uses of `buildPythonApplication` can be added to `python-packages.nix`. Python packages can be grouped into two groups: 1) applications and 2) packages providing importable modules. In `python-packages.nix` we only want to have 2). 1) should be in the top-level package set. To achieve this, all setup hooks need to be marked as being a setup hook. For the setup hooks in the Python packages set this is done by creating a new builder, `makePythonHook`. Because there were issues with splicing, the file importing all the hooks is converted to an extension. All non-packages were moved out of `python-packages.nix` into `python-packages-base.nix`. The `keep` argument to `makeScopeWithSplicing was cleaned up as well; there is no need to keep this one manually in sync reducing the risk of breaking cross-compilation.

FRidh · 2022-10-26T12:54:54Z

Everything seems to be OK now. I am surprised to see rebuilds though.

FRidh · 2022-10-27T07:02:26Z

Some packages now get a duplicate package in a closure. I guess an incorrect hook is being used.

- Don't call from a different scope - Disable catching of conflicts because we do get a (installation-time only) conflict due to overrides in awscli.

This is the recommended way. It won't actually matter here though.

FRidh · 2022-10-27T07:45:07Z

Bytecode has differences in filenames, but that would be fixed if we would build wheels as intermediate step and use CA then.

andrevmatos · 2022-10-28T14:40:39Z

This seems to have broken neovim-remote on the stdenv call in doCheck.

FRidh · 2022-10-28T17:20:23Z

This seems to have broken neovim-remote on the stdenv call in doCheck.

#198281

nixos-discourse · 2022-11-01T12:24:16Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/i-cant-update-my-system-due-to-buildpythonpackage/22886/7

fixes NixOS#198319

Artturin · 2022-11-06T23:12:10Z

this broke the fix i did in #196052 (comment)

before 33d12e5:

nix-repl> pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs
[ «derivation /nix/store/hvb9yxgv1133cfhxxd869sibldvv2vdx-python3-3.10.7.drv» «derivation /nix/store/v880cnh4ml7czmivfbk3cdh93hz9yvbn-hook.drv» «derivation /nix/store/f243ab7wv92gqsmc9h7gr0qcnj0xcgdb-hook.drv» «derivation /nix/store/880lf8895bzn8d94lrr2y7ilgkxq0lc4-python-remove-tests-dir-hook.drv» «derivation /nix/store/fzjnhawfs1wpw58hcd1vxd9y750dc08y-python-remove-bin-bytecode-hook.drv» «derivation /nix/store/n8l59iparx98yfw8g5ydqmzmk3fdic75-setuptools-setup-hook.drv» «derivation /nix/store/7vyhynlaxlva1rpvfl0hpdl3qywi9qmc-pip-install-hook.drv» «derivation /nix/store/0l76r5pf69smrl631qa801sjkyb8isk6-python-namespaces-hook.sh.drv» «derivation /nix/store/zipz2sj6ncyllb2z53p0fmsirq601rj7-python-output-dist-hook.drv» ]

after 33d12e5:

nix-repl> pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs
[ «derivation /nix/store/bhz39ds4v02hn6x4py4mzjyilw4a589h-python3-aarch64-unknown-linux-gnu-3.10.7.drv» «derivation /nix/store/q0pdz5z18bfra7841qwdqq9i3g5na3w7-hook.drv» «derivation /nix/store/f243ab7wv92gqsmc9h7gr0qcnj0xcgdb-hook.drv» «derivation /nix/store/880lf8895bzn8d94lrr2y7ilgkxq0lc4-python-remove-tests-dir-hook.drv» «derivation /nix/store/fzjnhawfs1wpw58hcd1vxd9y750dc08y-python-remove-bin-bytecode-hook.drv» «derivation /nix/store/x9b2q51spifxvzsxj90if1h4nb617pys-setuptools-setup-hook.drv» «derivation /nix/store/zsh90vnqf3zblmrlyh8py5hxky2bk78a-pip-install-hook.drv» «derivation /nix/store/a4cyp1fcsnlppzrwwbk1ka5xhbj52hnm-python-namespaces-hook.sh.drv» «derivation /nix/store/2hc97nsdi375cv13014c4lardpxk2mgw-python-output-dist-hook.drv» ]

Artturin · 2022-11-17T21:48:03Z

pkgs/development/interpreters/python/python-packages-base.nix

+  inherit toPythonModule toPythonApplication;
+  inherit buildSetupcfg;
+
+  python = toPythonModule python;


toPythonModule is probably discarding the splicing and that's why the fix is broken

Will test later

`lib.extends hooks pythonPackagesFun` includes python which we want to splice what this fixes: ``` nix-repl> lib.elemAt pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs 0 «derivation /nix/store/39dkb51rciw6zwg0c2c44gpmpjapddxc-python3-aarch64-unknown-linux-gnu-3.10.12.drv» ``` to ``` nix-repl> lib.elemAt pkgsCross.aarch64-multiplatform.python3Packages.xpybutil.nativeBuildInputs 0 «derivation /nix/store/21ldw2dp26xvv9iyxn9x77a8yh4waqz5-python3-3.10.12.drv» ``` Before NixOS#194205 The keep python used to work by accident because self was passed from __splicedPackages https://github.com/NixOS/nixpkgs/pull/196052/files#diff-44ce3495c4f983ce64dd47c86a9d3e77bad210b2709c098a3806998dcd9b000bR213 But now it does not work because overrideAttrs is used on python in python-packages-base.nix Preferably we would [ splice the hooks too ](NixOS#228139) but we cannot do that until [Package sets within derivations (i.e. python3.pkgs) are not spliced](NixOS#211340) is fixed, because people often use `python3.pkgs.wrapPython` in `nativeBuildInputs` (it's correct but python3.pkgs should be python3Packages to get splicing.

fixes NixOS#198319

FRidh requested a review from mweinelt October 3, 2022 10:25

github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Oct 3, 2022

FRidh mentioned this pull request Oct 3, 2022

python3Packages: Use buildPythonPackage within the package set #194100

Merged

13 tasks

FRidh changed the base branch from staging to master October 25, 2022 11:10

FRidh force-pushed the ensure branch from 160feec to 29c6dbd Compare October 25, 2022 13:39

FRidh requested a review from jonringer as a code owner October 25, 2022 13:39

ofborg bot added 8.has: clean-up This PR removes packages or removes other cruft 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels Oct 25, 2022

FRidh force-pushed the ensure branch 4 times, most recently from 7e97f7e to 9566aa4 Compare October 26, 2022 12:01

FRidh force-pushed the ensure branch from 9566aa4 to 24dd69b Compare October 26, 2022 12:53

FRidh added 2 commits October 27, 2022 09:36

git-remote-codecommit: fix build

0170e88

- Don't call from a different scope - Disable catching of conflicts because we do get a (installation-time only) conflict due to overrides in awscli.

awscli: override best practice

2f43e21

This is the recommended way. It won't actually matter here though.

ofborg bot requested a review from zaninime October 27, 2022 07:48

FRidh merged commit 0c5c895 into NixOS:master Oct 27, 2022

FRidh deleted the ensure branch October 27, 2022 08:03

gador pushed a commit to gador/nixpkgs that referenced this pull request Nov 4, 2022

platformio: work around fallout from NixOS#194205

a3b7b77

fixes NixOS#198319

Artturin reviewed Nov 17, 2022

View reviewed changes

Artturin mentioned this pull request Apr 14, 2023

beautifulsoup4: fix cross #226121

Merged

12 tasks

Artturin mentioned this pull request Jul 24, 2023

python: don't use keep for hooks, so they will be spliced #228139

Draft

12 tasks

Artturin mentioned this pull request Jul 26, 2023

python3: splice python within python3Packages.callPackage #245475

Merged

12 tasks

rtimush pushed a commit to rtimush/nixpkgs that referenced this pull request Sep 21, 2023

platformio: work around fallout from NixOS#194205

60830f7

fixes NixOS#198319

natsukium mentioned this pull request Jan 24, 2024

python311Packages.recursive-pth-loader: rename from recursivePthLoader #279384

Merged

13 tasks

Uh oh!

Conversation

FRidh commented Oct 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Things done

Uh oh!

mweinelt commented Oct 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FRidh commented Oct 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SuperSandro2000 commented Oct 3, 2022

Uh oh!

FRidh commented Oct 4, 2022

Uh oh!

FRidh commented Oct 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FRidh commented Oct 26, 2022

Uh oh!

FRidh commented Oct 27, 2022

Uh oh!

FRidh commented Oct 27, 2022

Uh oh!

andrevmatos commented Oct 28, 2022

Uh oh!

FRidh commented Oct 28, 2022

Uh oh!

nixos-discourse commented Nov 1, 2022

Uh oh!

Artturin commented Nov 6, 2022

Uh oh!

Artturin Nov 17, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

FRidh commented Oct 3, 2022 •

edited

Loading

mweinelt commented Oct 3, 2022 •

edited

Loading

FRidh commented Oct 3, 2022 •

edited

Loading

FRidh commented Oct 25, 2022 •

edited

Loading