Skip to content

[wip] Proposal/fetchpypi generalized fetching from pypi #22257

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
RonnyPfannschmidt wants to merge 7 commits into from
Closed

[wip] Proposal/fetchpypi generalized fetching from pypi #22257

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
fcda7d4
fetchurl: add pypi.io base url under new name
RonnyPfannschmidt Jan 29, 2017
c142e12
introduce python.fetchSource, test it on acoustics
RonnyPfannschmidt Jan 29, 2017
a87167c
python: introduce fetchers for wheel and sdists
RonnyPfannschmidt Jan 29, 2017
0516fd2
python: simplify fetchPypiMirror as suggested
RonnyPfannschmidt Feb 2, 2017
5ba74e2
fetchPypi: add url variations with name normalization
RonnyPfannschmidt Feb 2, 2017
8da067e
fechPypi: style fix
RonnyPfannschmidt Feb 2, 2017
df6185b
fetchpypi: add forgotten .tgz extension
RonnyPfannschmidt Feb 2, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions pkgs/build-support/fetchurl/mirrors.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,6 +386,10 @@ rec {
];

# Python PyPI mirrors

pypiio = [
https://pypi.io/packages/
];
pypi = [

@garbas garbas Jan 29, 2017 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't pypi.io be part of the pypi mirror?

@RonnyPfannschmidt RonnyPfannschmidt Jan 29, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

different root base url, the pypi mirror is having a bad starting segment not useful for the different types

https://pypi.io/packages/source/
];
Expand Down
28 changes: 21 additions & 7 deletions pkgs/top-level/python-packages.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,20 @@ let

graphiteVersion = "0.9.15";

fetchPypiMirror = ({ name ? null , pname ? null, version ? null, kind, ext, sha256}:
let
_name = if name == null then "${pname}-${version}" else name;

@LnL7 LnL7 Jan 29, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can do this with the default values of the attributeset

@RonnyPfannschmidt RonnyPfannschmidt Jan 29, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i dont follow

@layus layus Feb 1, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think he means

fetchPypiMirror = ({kind, ext, sha256, ...} @ args:
  let
    _name = args.name or "${pname}-${version}";

_pdrv = builtins.parseDrvName _name;
in
pkgs.fetchurl {
inherit sha256;
url = "mirror://pypiio/${kind}/${builtins.substring 0 1 _name}/${_pdrv.name}/${_name}${ext}";

@garbas garbas Jan 29, 2017 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this actually won't work in all cases :) I know your code should work, but I found out that there are many different ways how URL is generated on pypi.python.org. Also, the below code should make fetchSourceZip obsolete.

  fetchpypi = name: version: hash: pkgs.fetchurl {
      urls = [
        "https://pypi.python.org/packages/source/${builtins.substring 0 1 name}/${name}/${name}-${version}.tar.gz"
        "https://pypi.python.org/packages/source/${builtins.substring 0 1 name}/${name}/${pkgs.lib.toLower name}-${version}.tar.gz"
        "https://pypi.python.org/packages/source/${builtins.substring 0 1 name}/${name}/${name}-${version}.tgz"
        "https://pypi.python.org/packages/source/${builtins.substring 0 1 name}/${name}/${pkgs.lib.toLower name}-${version}.tgz"
        "https://pypi.python.org/packages/source/${builtins.substring 0 1 name}/${name}/${name}-${version}.zip"
        "https://pypi.python.org/packages/source/${builtins.substring 0 1 name}/${name}/${pkgs.lib.toLower name}-${version}.zip"
      ];
      md5 = "${hash}";
  };

@LnL7 LnL7 Jan 29, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should work if if normalized names are used, pip does the same since 9.x.

@garbas garbas Jan 29, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know it should work with normalized names, but my experience tells me different :)

@LnL7 LnL7 Jan 29, 2017 •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, perhaps it's only for the metadata then? That's why I used pip download for my implementation.

@RonnyPfannschmidt RonnyPfannschmidt Jan 29, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lovely idea

});

fetchSource = args: fetchPypiMirror (args // { kind = "source"; ext=".tar.gz"; });

@LnL7 LnL7 Jan 29, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used a types attributeset with a bunch of common extensions, that might be a bit more extensible

@RonnyPfannschmidt RonnyPfannschmidt Jan 29, 2017

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

care to elaborate

fetchSourceZip = args: fetchPypiMirror (args // { kind = "source"; ext = ".zip";});
fetchWheel = args: fetchPypiMirror (args // rec { kind = if "source" then "py3" else "py2"; ext = "-${kind}-none-any.whl";});
fetchUniversalWheel = args: fetchPypiMirror (args // rec { kind = "py2.py3"; ext = "-${kind}-none-any.whl";});
in {

inherit python bootstrapped-pip pythonAtLeast pythonOlder isPy26 isPy27 isPy33 isPy34 isPy35 isPy36 isPyPy isPy3k mkPythonDerivation buildPythonPackage buildPythonApplication;
Expand All @@ -53,14 +67,14 @@ in {

acoustics = buildPythonPackage rec {
pname = "acoustics";
name = "acoustics-${version}";
version = "0.1.2";
name = pname + "-" + version;

buildInputs = with self; [ cython pytest ];
propagatedBuildInputs = with self; [ numpy scipy matplotlib pandas tabulate ];

src = pkgs.fetchurl {
url = "mirror://pypi/${builtins.substring 0 1 pname}/${pname}/${name}.tar.gz";
src = fetchSource {
inherit name;
sha256 = "b75a47de700d01e704de95953a6e969922b2f510d7eefe59f7f8980ad44ad1b7";
};

Expand Down Expand Up @@ -88,8 +102,8 @@ in {

propagatedBuildInputs = with self; [ discid six parsedatetime isodate Babel pytimeparse ];

src = pkgs.fetchurl {
url = "mirror://pypi/a/agate/${name}.tar.gz";
src = fetchSource {
inherit name;
sha256 = "0h2w30a0zhylivz86d823a05hvg8w8p61lmm855z1wwkgml9l9d4";
};
};
Expand Down Expand Up @@ -755,8 +769,8 @@ in {
almir = buildPythonPackage rec {
name = "almir-0.1.8";

src = pkgs.fetchurl {
url = "mirror://pypi/a/almir/${name}.zip";
src = fetchSourceZip {
inherit name;
sha256 = "5dc0b8a5071f3ff46cd2d92608f567ba446e4c733c063b17d89703caeb9868fe";
};

Expand Down