mkPythonDerivation: validate propagatedBuildInputs and buildInputs use a matching python

[grahamc]

It isn't so difficult to accidentally mix and match Python packages from different pythons. This can cause fairly mysterious build time and run time errors. This PR adds a trace-only warning:

trace: warning: Warning: Python version mismatch in 'hi-bogus':

The Python derivation 'hi-bogus' depends on a Python derivation
named 'foooo', but the two derivations use different versions
of python:

    hi-bogus uses /nix/store/d44wd6n98f93hjr6q1d1phhh1hw7a17d-python3-3.8.8
       foooo uses /nix/store/pp0fp5zds7wqhyx6zqnnxl2aqzf4a51r-python-2.7.18

Suggestion: change hi-bogus's propagatedBuildInputs to use a 'foooo'
built from the same version of Python in /home/grahamc/projects/github.com/NixOS/nixpkgs/test.nix:5:3.

which will prevent these mistakes from entering Nixpkgs but only present as a nuisance for users outside of nixpkgs.

Open questions for me:

• Should this refuse to build?
• Should there be a way to turn it off for specific inputs?
• Should this check apply to other inputs?

This has been made on behalf of Flox.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[FRidh]

Good idea!

[FRidh]

Should this refuse to build?

In principle I should say yes, because this will lead to mistakes. A potential, but bad case, is where one wants to rely on wrapPythonPrograms to add applications of different interpreters in propagatedBuildInputs to $PATH. Note this wouldn't detect it because pythonModule = false;; adding all propagatedBuildInputs to $PATH but its how it is now and out of scope here.

Should there be a way to turn it off for specific inputs?

Until someone reports a valid case where it is needed, I don't think we should.

Should this check apply to other inputs?

This is sufficient I think. nativeBuildInputs can indeed contain 2 and 3 mixed. Other cross inputs I think we can ignore.

[FRidh]

Should this refuse to build?

Note if it would block to build, it would need an entry in the changelog.

[lukegb]

nit: propogatedBuildInputs --> propagatedBuildInputs in the PR subject

[grahamc]

nit: propogatedBuildInputs --> propagatedBuildInputs in the PR subject

https://logs.nix.samueldr.com/nixos-dev/2021-04-20#4878138;

[grahamc]

This PR has an updated message, and uses throw now:

error: Python version mismatch in 'hi-bogus':

The Python derivation 'hi-bogus' depends on a Python derivation
named 'flake8', but the two derivations use different versions
of Python:

    'hi-bogus' uses /nix/store/d44wd6n98f93hjr6q1d1phhh1hw7a17d-python3-3.8.8
      'flake8' uses /nix/store/pp0fp5zds7wqhyx6zqnnxl2aqzf4a51r-python-2.7.18

Possible solutions:

  * If 'flake8' is a Python library, change the reference to 'flake8'
    in the propagatedBuildInputs of 'hi-bogus' to use a 'flake8' built from the same
    version of Python

  * If 'flake8' provides executables, move the reference to 'flake8' in
    'hi-bogus' from propagatedBuildInputs to nativeBuildInputs

 at /home/grahamc/projects/github.com/NixOS/nixpkgs/test.nix:5:3

(use '--show-trace' to show detailed location information)

[grahamc]

Note if it would block to build, it would need an entry in the changelog.

It now blocks the build, @FRidh. Which changelog do you mean? NixOS? Nixpkgs hasn't really had a changelog in years.

[FRidh]

It now blocks the build, @FRidh. Which changelog do you mean? NixOS? Nixpkgs hasn't really had a changelog in years.

Yes, NixOS, so it is add least findable without having to resort to git log and wade through all the merge commits.

• If 'flake8' provides executables, move the reference to 'flake8' in
  'hi-bogus' from propagatedBuildInputs to nativeBuildInputs

• If 'flake8' is used as a tool during the build, move the reference to 'flake8' in
  'hi-bogus' from propagatedBuildInputs to nativeBuildInputs
• If 'flake8' provides executables that need to be available during run-time, pass makeWrapperArgs = [ "--prefix PATH : ${lib.makeBinPath [ flake8 ]}" ];

[grahamc]

If 'flake8' provides executables that need to be available during run-time, pass makeWrapperArgs = [ "--prefix PATH : ${lib.makeBinPath [ flake8 ]}" ];

This one is tricky, we don't actually know what the attribute name of the derivation named 'flake8', and I'm not sure how to word this otherwise.

[FRidh]

I noticed the same when working on error messages on mk-python-derivation.nix in #102613 (e.g. 2a1f172) and chose to just use the pname. To me that seems "good enough". Yes, it will be confusing at times, but in my opinion better than nothing. Otherwise replace the "flake8" there with "<your package>".

[grahamc]

How about an ofborg check that really wants pname == attrname

[grahamc]

Current error looks ilke:

error: Python version mismatch in 'hi-bogus':

The Python derivation 'hi-bogus' depends on a Python derivation
named 'flake8', but the two derivations use different versions
of Python:

    'hi-bogus' uses /nix/store/d44wd6n98f93hjr6q1d1phhh1hw7a17d-python3-3.8.8
      'flake8' uses /nix/store/pp0fp5zds7wqhyx6zqnnxl2aqzf4a51r-python-2.7.18

Possible solutions:

  * If 'flake8' is a Python library, change the reference to 'flake8'
    in the propagatedBuildInputs of 'hi-bogus' to use a 'flake8' built from the same
    version of Python

  * If 'flake8' is used as a tool during the build, move the reference to
    'flake8' in 'hi-bogus' from propagatedBuildInputs to nativeBuildInputs

  * If 'flake8' provides executables that are called at run time, pass its
    bin path to makeWrapperArgs:

        makeWrapperArgs = [ "--prefix PATH : ${lib.makeBinPath [ flake8 ] }" ];

 at /home/grahamc/projects/github.com/NixOS/nixpkgs/test.nix:6:3

[FRidh]

Current error looks ilke:

Looks great to me!

How about an ofborg check that really wants pname == attrname

That would also need to consider variants, which would then be forced to fix their pname, for example like

hdf5-mpi = appendToName "mpi" (hdf5.override { ...

That seems like a good idea.

A problematic case would be Python packages. Their pname is actually different; the function you're editing now prefixes pythonX.Y to its name. An option there is to make that pname for example pythonXYPackages.flake. But then how do we deal with a dot in a pname? This clearly could get complex.

[grahamc]

Agreed, I don't think it would be reasonable to ensure pname == attrname. It might be a useful lint as part of a "packaging health report", though... Not sure. Something like this would be advisory, not errory.

[SuperSandro2000]

How about an ofborg check that really wants pname == attrname

Yes please. or nixpkgs-hammering.

[lukegb]

Any chance we can land this? The error looks good - I think we might want to file a followup issue against OfBorg for the pname==attrname check.

[stale]

I marked this as stale due to inactivity. → More info

[RaitoBezarius]

Can I do something to help get this merged?

[Artturin]

Either this should just be .name or both should use lib.getName, depends on if we want versions

       error: Python version mismatch in 'xpybutil-0.0.6':

       The Python derivation 'xpybutil-0.0.6' depends on a Python derivation
       named 'xpybutil', but the two derivations use different versions
       of Python:

           'xpybutil-0.0.6' uses /nix/store/fdqpyj613dr0v1l1lrzqhzay7sk4xg87-python3-3.10.10
                 'xpybutil' uses /nix/store/v7h4276ij0sj48qwrf4g4cm3pfc86mmw-python3-3.9.16

       Possible solutions:

         * If 'xpybutil' is a Python library, change the reference to 'xpybutil'
           in the propagatedBuildInputs of 'xpybutil-0.0.6' to use a 'xpybutil' built from the same
           version of Python

         * If 'xpybutil' is used as a tool during the build, move the reference to
           'xpybutil' in 'xpybutil-0.0.6' from propagatedBuildInputs to nativeBuildInputs

         * If 'xpybutil' provides executables that are called at run time, pass its
           bin path to makeWrapperArgs:

               makeWrapperArgs = [ "--prefix PATH : ${lib.makeBinPath [ xpybutil ] }" ];

diff --git a/pkgs/development/python-modules/xpybutil/default.nix b/pkgs/development/python-modules/xpybutil/default.nix
index cc574c58241..9236df77d84 100644
--- a/pkgs/development/python-modules/xpybutil/default.nix
+++ b/pkgs/development/python-modules/xpybutil/default.nix
@@ -1,4 +1,4 @@
-{ lib, buildPythonPackage, fetchFromGitHub, xcffib, pillow }:
+{ lib, buildPythonPackage, fetchFromGitHub, xcffib, pillow, python39 }:
 
 buildPythonPackage rec {
   pname = "xpybutil";
@@ -13,7 +13,7 @@ buildPythonPackage rec {
   };
 
   # pillow is a dependency in image.py which is not listed in setup.py
-  propagatedBuildInputs = [ pillow xcffib ];
+  propagatedBuildInputs = [ pillow xcffib python39.pkgs.xpybutil ];
 
   propagatedNativeBuildInputs = [ xcffib ];

[Artturin]

changed to .name and the suggestion below to lib.getName

[FRidh]

I have not tested this but the change looks good.

[Artturin]

error: Python version mismatch in 'python3.10-xpybutil-0.0.6':

The Python derivation 'python3.10-xpybutil-0.0.6' depends on a Python derivation
named 'python3.9-i3ipc-2.2.1', but the two derivations use different versions
of Python:

    'python3.10-xpybutil-0.0.6' uses /nix/store/fdqpyj613dr0v1l1lrzqhzay7sk4xg87-python3-3.10.10
        'python3.9-i3ipc-2.2.1' uses /nix/store/v7h4276ij0sj48qwrf4g4cm3pfc86mmw-python3-3.9.16

Possible solutions:

  * If 'python3.9-i3ipc-2.2.1' is a Python library, change the reference to i3ipc
    in the propagatedBuildInputs of 'python3.10-xpybutil-0.0.6' to use a i3ipc built from the same
    version of Python

  * If 'python3.9-i3ipc-2.2.1' is used as a tool during the build, move the reference to
    i3ipc in 'python3.10-xpybutil-0.0.6' from propagatedBuildInputs to nativeBuildInputs

  * If 'python3.9-i3ipc-2.2.1' provides executables that are called at run time, pass its
    bin path to makeWrapperArgs:

        makeWrapperArgs = [ "--prefix PATH : ${lib.makeBinPath [ i3ipc ] }" ];

[grahamc]

neat!