Fix infinite recursion for aliases in pkgs/top-level/python-aliases.nix #787

SemMulder · 2022-10-31T13:23:16Z

This fixes #750.

A better fix for this might be to set e.g. meta.isAlias = true in nixpkgs/blob/master/pkgs/top-level/python-aliases.nix. Let me know if that is a better route :).

K900 · 2022-11-01T09:09:49Z

One thing I thought of doing instead is just using the normalized nixpkgs attrset as a base, without adding normalization as an overlay. That should get things building again and also allow us to find some unnormalized names in overrides.

SemMulder · 2022-11-01T09:32:48Z

normalized nixpkgs attrset as a base

Do you mean using python.pkgs from nixpkgs as is, so without using the existing overlay mechanism offered in nixpkgs (i.e. packageOverrides)? Then normalizing python.pkgs. And then building our own overlay mechanism on top of that?

I think that would be a nicer solution as well. But I'm not sure if we can make it work:

One thing I'm not sure about is that then any package we include directly from the raw python.pkgs in the final "virtualenv" would ship with its own set of dependencies (i.e. with the versions in the raw python.pkgs). But those dependencies might also be specified in poetry.lock, but with a different version. That would result in collisions, I think.

Or can we avoid including any package from python.pkgs in the final virtualenv? Strictly speaking, I think we only need some packages (setuptools, pip etc.) to allow building the Python packages from poetry.lock. But then we would also need to duplicate the work already done in python.pkgs (e.g. specifying runtime dependencies, difficult build processes) I think? Not sure how much of that we are actually currently using or not?

K900 · 2022-11-01T09:34:04Z

No, I mean replacing python.pkgs with a normalized version of itself, so doing the same thing, but removing all the non-normalized names.

SemMulder · 2022-11-03T10:09:18Z

@K900 do you mean something like this?

cpcloud · 2022-11-07T12:34:19Z

@adisbladis Can you take a look at this? It'd be swell if we could address the infinite recursion issue with aliased packages.

NorfairKing · 2022-11-21T23:51:31Z

@adisbladis Gentle ping

NorfairKing · 2022-11-29T13:58:25Z

@adisbladis Ping. This is blocking an upgrade for us.

This fixes nix-community#750.

cpcloud · 2022-12-06T12:36:03Z

@adisbladis Any chance you can take a look at this? Otherwise, I'll try this out on a recent version of mkdocs which has this problem and if it successfully builds I'll merge this PR.

tfmoraes · 2022-12-06T21:24:12Z

I'm using this poetry2nix branch and it was working. But if I update torch to 1.13 I start to have a problem with infinite recursion. This is the output:

warning: Git tree '/home/thiago/Sources/github/invesalius3_nix_develop' is dirty
error: infinite recursion encountered

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/stdenv/generic/make-derivation.nix:311:7:

          310|       depsBuildBuild              = lib.elemAt (lib.elemAt dependencies 0) 0;
          311|       nativeBuildInputs           = lib.elemAt (lib.elemAt dependencies 0) 1;
             |       ^
          312|       depsBuildTarget             = lib.elemAt (lib.elemAt dependencies 0) 2;

       … while evaluating the attribute 'nativeBuildInputs' of the derivation 'python3.10-python3.10-wheel-0.38.4'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

       … while evaluating the attribute 'outPath'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/lib/customisation.nix:214:7:

          213|       drvPath = assert condition; drv.drvPath;
          214|       outPath = assert condition; drv.outPath;
             |       ^
          215|     };

       … while evaluating 'optionalString'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/lib/strings.nix:213:5:

          212|     # String to return if condition is true
          213|     string: if cond then string else "";
             |     ^
          214|

       … from call site

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/build-support/trivial-builders.nix:575:12:

          574|         cp ${script} $out/nix-support/setup-hook
          575|       '' + lib.optionalString (deps != []) ''
             |            ^
          576|         printWords ${toString deps} > $out/nix-support/propagated-build-inputs

       … while evaluating the attribute 'buildCommand' of the derivation 'wheel-unpack-hook.sh'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

       … while evaluating the attribute 'nativeBuildInputs' of the derivation 'python3.10-parso-0.8.3'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

       … while evaluating the attribute 'out.outPath'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/lib/customisation.nix:207:13:

          206|             drvPath = assert condition; drv.${outputName}.drvPath;
          207|             outPath = assert condition; drv.${outputName}.outPath;
             |             ^
          208|           };

       … while evaluating anonymous lambda

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/lib/lists.nix:646:25:

          645|    */
          646|   unique = foldl' (acc: e: if elem e acc then acc else acc ++ [ e ]) [];
             |                         ^
          647|

       … from call site

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/development/interpreters/python/python-packages-base.nix:52:6:

           51|     modules = lib.filter hasPythonModule drvs;
           52|   in lib.unique ([python] ++ modules ++ lib.concatLists (lib.catAttrs "requiredPythonModules" modules));
             |      ^
           53|

       … while evaluating 'requiredPythonModules'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/development/interpreters/python/python-packages-base.nix:50:27:

           49|   # Get list of required Python modules given a list of derivations.
           50|   requiredPythonModules = drvs: let
             |                           ^
           51|     modules = lib.filter hasPythonModule drvs;

       … from call site

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/development/interpreters/python/python-packages-base.nix:67:33:

           66|         pythonPath = [ ]; # Deprecated, for compatibility.
           67|         requiredPythonModules = requiredPythonModules drv.propagatedBuildInputs;
             |                                 ^
           68|       };

       … while evaluating 'requiredPythonModules'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/development/interpreters/python/python-packages-base.nix:50:27:

           49|   # Get list of required Python modules given a list of derivations.
           50|   requiredPythonModules = drvs: let
             |                           ^
           51|     modules = lib.filter hasPythonModule drvs;

       … from call site

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/development/interpreters/python/python-packages-base.nix:67:33:

           66|         pythonPath = [ ]; # Deprecated, for compatibility.
           67|         requiredPythonModules = requiredPythonModules drv.propagatedBuildInputs;
             |                                 ^
           68|       };

       … while evaluating 'requiredPythonModules'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/development/interpreters/python/python-packages-base.nix:50:27:

           49|   # Get list of required Python modules given a list of derivations.
           50|   requiredPythonModules = drvs: let
             |                           ^
           51|     modules = lib.filter hasPythonModule drvs;

       … from call site

       at /nix/store/qrfavlffxx8a8d3mkb5zp474z0835ryr-source/default.nix:279:23:

          278|       */
          279|       storePackages = requiredPythonModules (builtins.foldl' (acc: v: acc ++ v) [ ] (lib.attrValues inputAttrs));
             |                       ^
          280|     in

       … while evaluating the attribute 'poetryPackages'

       at /nix/store/qrfavlffxx8a8d3mkb5zp474z0835ryr-source/default.nix:283:7:

          282|       python = py;
          283|       poetryPackages = storePackages
             |       ^
          284|         ++ lib.optional hasScripts scriptsPackage

       … while evaluating anonymous lambda

       at /nix/store/qrfavlffxx8a8d3mkb5zp474z0835ryr-source/default.nix:351:39:

          350|     in
          351|     poetryPython.python.withPackages (ps: envPkgs ++ (extraPackages ps));
             |                                       ^
          352|

       … from call site

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/development/interpreters/python/with-packages.nix:3:19:

            2|
            3| f: let packages = f pythonPackages; in buildEnv.override { extraLibs = packages; }
             |                   ^

       … while evaluating 'requiredPythonModules'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/development/interpreters/python/python-packages-base.nix:50:27:

           49|   # Get list of required Python modules given a list of derivations.
           50|   requiredPythonModules = drvs: let
             |                           ^
           51|     modules = lib.filter hasPythonModule drvs;

       … from call site

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/development/interpreters/python/wrapper.nix:20:13:

           19|   env = let
           20|     paths = requiredPythonModules (extraLibs ++ [ python ] ) ;
             |             ^
           21|     pythonPath = "${placeholder "out"}/${python.sitePackages}";

       … while evaluating the attribute 'passAsFile'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/build-support/buildenv/default.nix:77:5:

           76|     # XXX: The size is somewhat arbitrary
           77|     passAsFile = if builtins.stringLength pkgs >= 128*1024 then [ "pkgs" ] else [ ];
             |     ^
           78|   }

       … while evaluating the attribute 'passAsFile' of the derivation 'python3-3.10.8-env'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

       … while evaluating the attribute 'buildInputs' of the derivation 'nix-shell'

       at /nix/store/wm2w45lg0dqrcfrn54dijw30ljpx5k8k-source/pkgs/stdenv/generic/make-derivation.nix:270:7:

          269|     // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
          270|       name =
             |       ^
          271|         let

adisbladis · 2022-12-07T04:01:13Z

@tfmoraes Just eyeballing that stack trace it looks to me like a different recursion issue than this one. Could you open a new issue?

adisbladis · 2022-12-07T04:02:52Z

This achieves the same thing I wanted to with normalisation, so I'm happy with this.

tfmoraes · 2022-12-07T14:23:49Z

Done @adisbladis! I opened this issue #850

SemMulder force-pushed the fix-750 branch from 4dccae6 to fb5566d Compare November 3, 2022 10:08

cpcloud requested a review from adisbladis November 7, 2022 12:33

mrene mentioned this pull request Nov 12, 2022

Infinite recursion #810

Closed

Fix infinite recursion for aliases in pkgs/top-level/python-aliases.nix

9dc514d

This fixes nix-community#750.

SemMulder force-pushed the fix-750 branch from fb5566d to 9dc514d Compare December 5, 2022 09:51

adisbladis merged commit ce618e3 into nix-community:master Dec 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix infinite recursion for aliases in pkgs/top-level/python-aliases.nix #787

Fix infinite recursion for aliases in pkgs/top-level/python-aliases.nix #787

SemMulder commented Oct 31, 2022

K900 commented Nov 1, 2022

SemMulder commented Nov 1, 2022

K900 commented Nov 1, 2022

SemMulder commented Nov 3, 2022

cpcloud commented Nov 7, 2022

NorfairKing commented Nov 21, 2022

NorfairKing commented Nov 29, 2022

cpcloud commented Dec 6, 2022

tfmoraes commented Dec 6, 2022

adisbladis commented Dec 7, 2022

adisbladis commented Dec 7, 2022

tfmoraes commented Dec 7, 2022

Fix infinite recursion for aliases in pkgs/top-level/python-aliases.nix #787

Fix infinite recursion for aliases in pkgs/top-level/python-aliases.nix #787

Conversation

SemMulder commented Oct 31, 2022

K900 commented Nov 1, 2022

SemMulder commented Nov 1, 2022

K900 commented Nov 1, 2022

SemMulder commented Nov 3, 2022

cpcloud commented Nov 7, 2022

NorfairKing commented Nov 21, 2022

NorfairKing commented Nov 29, 2022

cpcloud commented Dec 6, 2022

tfmoraes commented Dec 6, 2022

adisbladis commented Dec 7, 2022

adisbladis commented Dec 7, 2022

tfmoraes commented Dec 7, 2022