pypi2nix - generating packages from pypi

[garbas]

after this PL gets merged i'll slowly migrate packages from python-packages.nix to be generated with pypi2nix tool

[bjornfor]

@garbas: This looks really cool! Some questions:

Does it handle specific versions of python modules? I'm guessing yes because of this line in your patch:

propagatedBuildInputs = [ pythonPackages."Distutils2-1.0a4" ];

I'm guessing that python-packages.json is the hand-written input to pypi2nix? If so, why do you have to tell pypi2nix what the buildinputs of packages are? Doesn't the pypi database say that?

+      "lxml": {
+        "buildInputs": [ "pkgs.libxml2", "pkgs.libxslt" ],
+        "doCheck": false
+      },

Or are you actually overriding pypi dependency information (I see you have a "override" section).

I hope these are not too dumb questions! :-)

[offlinehacker]

Answer to second question:

These are native build inputs. There's no simple way to get these
dependencies, because names are different in different linux distros.
On Aug 12, 2013 7:46 PM, "Bjørn Forsman" notifications@github.com wrote:

@garbas https://github.com/garbas: This looks really cool! Some
questions:

Does it handle specific versions of python modules? I'm guessing yes
because of this line in your patch:

propagatedBuildInputs = [ pythonPackages."Distutils2-1.0a4" ];

I'm guessing that python-packages.json is the hand-written input to
pypi2nix? If so, why do you have to tell pypi2nix what the buildinputs of
packages are? Doesn't the pypi database say that?

•  "lxml": {
  

•    "buildInputs": [ "pkgs.libxml2", "pkgs.libxslt" ],
  

•    "doCheck": false
  

•  },
  

Or are you actually overriding pypi dependency information (I see you
have a "override" section).

I hope these are not too dumb questions! :-)

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/815#issuecomment-22511290
.

[bjornfor]

Ah, of course, these are c library dependencies. I thought they were python modules (how silly of me). But what about this one:

+      "zope.testing": {
+        "buildInputs": [ "zope.location" ]
+      },

Zope is pure python, right?

[offlinehacker]

Missing in setup.py, am i right @garbas?
On Aug 12, 2013 8:20 PM, "Bjørn Forsman" notifications@github.com wrote:

Ah, of course, these are c library dependencies. I thought they were
python modules (how silly of me). But what about this one:

•  "zope.testing": {
  

•    "buildInputs": [ "zope.location" ]
  

•  },
  

Zope is pure python, right?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/815#issuecomment-22513747
.

[garbas]

Quoting Bjørn Forsman (2013-08-12 19:45:15)

@garbas: This looks really cool! Some questions:

Does it handle specific versions of python modules? I'm guessing yes because of
this line in your patch:

propagatedBuildInputs = [ pythonPackages."Distutils2-1.0a4" ];

yes its handling specific version of python packages. later on i will also add
"aliases" which could be used for specific version, eg:

{ name: "example"
aliases: ['', '1', '1.2']
}

if there is version 1.2.3 for example package it would produce something like

self."example" = self."example-1.2.3";
self."example-1" = self."example-1.2.3";
self."example-1.2" = self."example-1.2.3";
self."example-1.2.3" = buildPythonPackage {...};

i'm also not sure of exact

I'm guessing that python-packages.json is the hand-written input to pypi2nix?
If so, why do you have to tell pypi2nix what the buildinputs of packages are?
Doesn't the pypi database say that?

•  "lxml": {
  

•    "buildInputs": [ "pkgs.libxml2", "pkgs.libxslt" ],
  

•    "doCheck": false
  

•  },
  

Or are you actually overriding pypi dependency information (I see you have a
"override" section).

as @offlinehacker pointed out, you can not get buildInputs from pypi. thats why
i enable you to extend attribute set which is passed to buildPythonPackage
function. for example if "packageA" has a dependency on "packageB" and
"packageC" and for some reason pypi doesnt list "packageC" as dependency
(happens in some corner cases) then we can still define it in
python-packages.json, eg:

"packageA": {
"propagatedBuildInputs": [ "packageC" ]
}

will be generated to something like:

self."packageA-1.2.3" = buildPythonPackage {
...
propagatedBuildInputs = [ self."packageB-1.2.3" self."packageC-1.2.3" ];
...
}

I hope these are not too dumb questions! :-)

no questions are dumb, tnx for taking time to look into it.

Rok Garbas - http://www.garbas.si

[bjornfor]

@garbas: Thanks for the explanation.

I tried to run pypi2nix and got this message:

$ git checkout origin/pr/815
$ nix-build -A python27Packages.pypi2nix
$ ./result/bin/pypi2nix
Traceback (most recent call last):
  File "/nix/store/27gwi3xgy3m6ksnr89ks5p93dciyqvi0-python2.7-pypi2nix-1.0_f0bed8037538de481fab19af0521cb771c148aff/bin/.pypi2nix-wrapped", line 9, in <module>
    load_entry_point('pypi2nix==0.0.1', 'console_scripts', 'pypi2nix')()
  File "/nix/store/27gwi3xgy3m6ksnr89ks5p93dciyqvi0-python2.7-pypi2nix-1.0_f0bed8037538de481fab19af0521cb771c148aff/lib/python2.7/site-packages/pypi2nix-0.0.1-py2.7.egg/pypi2nix/cli.py", line 80, in main
    f.close()
UnboundLocalError: local variable 'f' referenced before assignment

Reading the code, I see that it expects a json file as sys.argv[1]. But it'd be nice to just have a "Usage: pypi2nix ...." message instead of this traceback :-)

Also, if I pass it a non-existing path it prints the same traceback. Could you improve the error handling?

Lastly, you have an "eggsdir = '/home/rok/.buildout/eggs'" line in pypi2nix/cli.py and this eggsdir variable is used further down in the code.

[garbas]

@bjornfor tnx for feedback

• i've removed my custom stuff and point to default buildout eggs cache directory (will be created if it doesnt exist). note that if you are running pypi2nix for the first time it will take quite some time since it has to pull down all the dependencies from pyramid and plone (around 300). this will be only needed once.
• add nicer error when you forget to provide json file ("Usage:...")

[bjornfor]

@garbas: Thanks! +1 for merge :-)

[edolstra]

I think this broke the Nixpkgs channel: http://hydra.nixos.org/build/5713172

attribute `Distutils2-1.0a4' missing

Could you run nix-env -f /my/nixpkgs -qa '*' --drv-path (or nixpkgs-lint, which also does that) to discover these problems before pushing?

[garbas]

Quoting Eelco Dolstra (2013-08-15 03:25:10)

I think this broke the Nixpkgs channel: http://hydra.nixos.org/build/5713172

attribute `Distutils2-1.0a4' missing

Could you run nix-env -f /my/nixpkgs -qa '*' --drv-path (or nixpkgs-lint, which
also does that) to discover these problems before pushing?

fixing right now.

Rok Garbas - http://www.garbas.si