Skip to content

[wip] Proposal/fetchpypi generalized fetching from pypi #22257

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
RonnyPfannschmidt wants to merge 7 commits into from
Closed

[wip] Proposal/fetchpypi generalized fetching from pypi #22257

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
fcda7d4
fetchurl: add pypi.io base url under new name
RonnyPfannschmidt Jan 29, 2017
c142e12
introduce python.fetchSource, test it on acoustics
RonnyPfannschmidt Jan 29, 2017
a87167c
python: introduce fetchers for wheel and sdists
RonnyPfannschmidt Jan 29, 2017
0516fd2
python: simplify fetchPypiMirror as suggested
RonnyPfannschmidt Feb 2, 2017
5ba74e2
fetchPypi: add url variations with name normalization
RonnyPfannschmidt Feb 2, 2017
8da067e
fechPypi: style fix
RonnyPfannschmidt Feb 2, 2017
df6185b
fetchpypi: add forgotten .tgz extension
RonnyPfannschmidt Feb 2, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions pkgs/build-support/fetchurl/mirrors.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,6 +386,10 @@ rec {
];

# Python PyPI mirrors

pypiio = [
https://pypi.io/packages/
];
pypi = [
Copy link
Member

@garbas garbas Jan 29, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't pypi.io be part of the pypi mirror?

Copy link
Contributor Author

@RonnyPfannschmidt RonnyPfannschmidt Jan 29, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

different root base url, the pypi mirror is having a bad starting segment not useful for the different types

https://pypi.io/packages/source/
];
Expand Down
37 changes: 29 additions & 8 deletions pkgs/top-level/python-packages.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ let
packages = ( self:

let

_firstChar = strings.substring 0 1;
pythonAtLeast = versionAtLeast python.pythonVersion;
pythonOlder = versionOlder python.pythonVersion;
isPy26 = python.majorVersion == "2.6";
Expand All @@ -19,7 +21,7 @@ let
isPy35 = python.majorVersion == "3.5";
isPy36 = python.majorVersion == "3.6";
isPyPy = python.executable == "pypy";
isPy3k = strings.substring 0 1 python.majorVersion == "3";
isPy3k = _firstChar python.majorVersion == "3";

callPackage = pkgs.newScope self;

Expand All @@ -36,7 +38,26 @@ let
buildPythonApplication = args: buildPythonPackage ({namePrefix="";} // args );

graphiteVersion = "0.9.15";


mkurls = ({name, kind, ext}:
let
drvs = map builtins.parseDrvName [ name (toLower name) ];
in map ({name, version}: "mirror://pypiio/${kind}/${_firstChar name}/${name}/${name}-${version}${ext}") drvs
);

fetchPypiMirror = ({ kind, exts, sha256, ...} @ args:
let
name = args.name or "${args.pname}-${args.version}";
in
pkgs.fetchurl {
inherit sha256;
urls = concatMap (ext: mkurls { inherit name kind ext;}) exts;
});

fetchSource = args: fetchPypiMirror (args // { kind = "source"; exts=[".tar.gz" ".zip" ".tgz"]; });
fetchWheel = args: fetchPypiMirror (args // rec { kind ="py${_firstChar python.majorVersion}"; exts = ["-${kind}-none-any.whl"];});
fetchUniversalWheel = args: fetchPypiMirror (args // rec { kind = "py2.py3"; exts = ["-${kind}-none-any.whl"];});
in {

inherit python bootstrapped-pip pythonAtLeast pythonOlder isPy26 isPy27 isPy33 isPy34 isPy35 isPy36 isPyPy isPy3k mkPythonDerivation buildPythonPackage buildPythonApplication;
Expand All @@ -53,14 +74,14 @@ in {

acoustics = buildPythonPackage rec {
pname = "acoustics";
name = "acoustics-${version}";
version = "0.1.2";
name = pname + "-" + version;

buildInputs = with self; [ cython pytest ];
propagatedBuildInputs = with self; [ numpy scipy matplotlib pandas tabulate ];

src = pkgs.fetchurl {
url = "mirror://pypi/${builtins.substring 0 1 pname}/${pname}/${name}.tar.gz";
src = fetchSource {
inherit name;
sha256 = "b75a47de700d01e704de95953a6e969922b2f510d7eefe59f7f8980ad44ad1b7";
};

Expand Down Expand Up @@ -88,8 +109,8 @@ in {

propagatedBuildInputs = with self; [ discid six parsedatetime isodate Babel pytimeparse ];

src = pkgs.fetchurl {
url = "mirror://pypi/a/agate/${name}.tar.gz";
src = fetchSource {
inherit name;
sha256 = "0h2w30a0zhylivz86d823a05hvg8w8p61lmm855z1wwkgml9l9d4";
};
};
Expand Down Expand Up @@ -755,8 +776,8 @@ in {
almir = buildPythonPackage rec {
name = "almir-0.1.8";

src = pkgs.fetchurl {
url = "mirror://pypi/a/almir/${name}.zip";
src = fetchSourceZip {
inherit name;
sha256 = "5dc0b8a5071f3ff46cd2d92608f567ba446e4c733c063b17d89703caeb9868fe";
};

Expand Down