[wip] Proposal/fetchpypi generalized fetching from pypi by RonnyPfannschmidt · Pull Request #22257 · NixOS/nixpkgs

RonnyPfannschmidt · 2017-01-29T11:31:43Z

Motivation for this change

add a more general tool to fetch packages from pypi using fetchurl

step1 is adding the pypi.io base mirror and testing the first helper on a few packages
the next steps are adding wheel fetching and updating more packages

followup to #22256 22256

Things done

Tested using sandboxing
(nix.useSandbox on NixOS,
or option build-use-sandbox in nix.conf
on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- Linux
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

mention-bot · 2017-01-29T11:31:53Z

@RonnyPfannschmidt, thanks for your PR! By analyzing the history of the files in this pull request, we identified @edolstra, @peti, @cmfwyp and @FRidh to be potential reviewers.

RonnyPfannschmidt · 2017-01-29T16:40:43Z

current failures are because setuptools/wheel are not implied build dependencies of pip, i beleive i should split the pr

LnL7 · 2017-01-29T16:52:06Z

pkgs/top-level/python-packages.nix


+  fetchPypiMirror = ({ name ? null , pname ? null, version ? null, kind, ext, sha256}:
+    let
+      _name = if name == null then "${pname}-${version}" else name;


you can do this with the default values of the attributeset

i dont follow

I think he means

fetchPypiMirror = ({kind, ext, sha256, ...} @ args: let _name = args.name or "${pname}-${version}";

LnL7 · 2017-01-29T17:35:58Z

pkgs/top-level/python-packages.nix

+        url = "mirror://pypiio/${kind}/${builtins.substring 0 1 _name}/${_pdrv.name}/${_name}${ext}";
+   });
+
+  fetchSource = args: fetchPypiMirror (args // { kind = "source"; ext=".tar.gz"; });


I used a types attributeset with a bunch of common extensions, that might be a bit more extensible

care to elaborate

garbas · 2017-01-29T17:49:57Z

pkgs/build-support/fetchurl/mirrors.nix

+  pypiio = [
+    https://pypi.io/packages/
+  ];
  pypi = [


shouldn't pypi.io be part of the pypi mirror?

different root base url, the pypi mirror is having a bad starting segment not useful for the different types

garbas · 2017-01-29T18:01:08Z

pkgs/top-level/python-packages.nix

+    in
+      pkgs.fetchurl {
+        inherit sha256;
+        url = "mirror://pypiio/${kind}/${builtins.substring 0 1 _name}/${_pdrv.name}/${_name}${ext}";


this actually won't work in all cases :) I know your code should work, but I found out that there are many different ways how URL is generated on pypi.python.org. Also, the below code should make fetchSourceZip obsolete.

fetchpypi = name: version: hash: pkgs.fetchurl { urls = [ "https://pypi.python.org/packages/source/${builtins.substring 0 1 name}/${name}/${name}-${version}.tar.gz" "https://pypi.python.org/packages/source/${builtins.substring 0 1 name}/${name}/${pkgs.lib.toLower name}-${version}.tar.gz" "https://pypi.python.org/packages/source/${builtins.substring 0 1 name}/${name}/${name}-${version}.tgz" "https://pypi.python.org/packages/source/${builtins.substring 0 1 name}/${name}/${pkgs.lib.toLower name}-${version}.tgz" "https://pypi.python.org/packages/source/${builtins.substring 0 1 name}/${name}/${name}-${version}.zip" "https://pypi.python.org/packages/source/${builtins.substring 0 1 name}/${name}/${pkgs.lib.toLower name}-${version}.zip" ]; md5 = "${hash}"; };

It should work if if normalized names are used, pip does the same since 9.x.

I know it should work with normalized names, but my experience tells me different :)

Oh, perhaps it's only for the metadata then? That's why I used pip download for my implementation.

lovely idea

Mic92 · 2017-01-29T18:22:57Z

see also #16005 by @FRidh, which introduce a buildPyPIPackage

RonnyPfannschmidt · 2017-01-29T19:47:25Z

@Mic92 thanks for the reference, i believe the 2 pull requests are orthogonal - my functions could be used as part of the functions made by @FRidh tho

Inspired by #22257. It is recommend to use fetchpypi and, when fetching a wheel, to pass `format = "wheel"` (typically you can just `inherit format;`). The hash type is fixed (for now) because this is the hash type PyPI uses.

FRidh · 2017-02-01T13:54:55Z

I intend to merge 695ff0d

FRidh · 2017-02-01T13:55:37Z

Though maybe more urls have to be added as @garbas wrote.

RonnyPfannschmidt · 2017-02-01T14:09:12Z

i haven't gotten around updating, i#ll try to take a look after work

RonnyPfannschmidt · 2017-02-02T10:19:55Z

@FRidh i belive this is obsolete now?

FRidh · 2017-02-10T16:21:42Z

i belive this is obsolete now?

Yes, I think it is. Thanks for the proposal though!

RonnyPfannschmidt added 2 commits January 29, 2017 11:47

fetchurl: add pypi.io base url under new name

fcda7d4

introduce python.fetchSource, test it on acoustics

c142e12

python: introduce fetchers for wheel and sdists

a87167c

RonnyPfannschmidt changed the title ~~[wip} Proposal/fetchpypi generalized fetching from pypi~~ [wip] Proposal/fetchpypi generalized fetching from pypi Jan 29, 2017

LnL7 reviewed Jan 29, 2017

View reviewed changes

garbas reviewed Jan 29, 2017

View reviewed changes

FRidh added the 6.topic: python Python is a high-level, general-purpose programming language. label Jan 30, 2017

FRidh mentioned this pull request Feb 1, 2017

Python package updates and new fetchers #22345

Merged

7 tasks

RonnyPfannschmidt added 4 commits February 2, 2017 07:55

python: simplify fetchPypiMirror as suggested

0516fd2

fetchPypi: add url variations with name normalization

5ba74e2

fechPypi: style fix

8da067e

fetchpypi: add forgotten .tgz extension

df6185b

FRidh closed this Feb 10, 2017

Uh oh!

Conversation

RonnyPfannschmidt commented Jan 29, 2017

Motivation for this change

Things done

Uh oh!

mention-bot commented Jan 29, 2017

Uh oh!

RonnyPfannschmidt commented Jan 29, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

garbas Jan 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

garbas Jan 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LnL7 Jan 29, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Mic92 commented Jan 29, 2017

Uh oh!

RonnyPfannschmidt commented Jan 29, 2017

Uh oh!

FRidh commented Feb 1, 2017

Uh oh!

FRidh commented Feb 1, 2017

Uh oh!

RonnyPfannschmidt commented Feb 1, 2017

Uh oh!

RonnyPfannschmidt commented Feb 2, 2017

Uh oh!

FRidh commented Feb 10, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

garbas Jan 29, 2017 •

edited

Loading

garbas Jan 29, 2017 •

edited

Loading

LnL7 Jan 29, 2017 •

edited

Loading