python3: add C++ compiler support for distutils by veprbl · Pull Request #39576 · NixOS/nixpkgs

veprbl · 2018-04-26T22:05:35Z

This implements 095095c4 ('python: add C++ compiler support for distutils')
for python3. Should fix various problems with python packages on darwin.

Motivation for this change

Even in python3 the distutils will try to invoke C compiler to compile C++ files, which works against GCC and Apple Clang, but not with clang that we ship.

This expands on #27649 to play well with numpy/cython (see 46dd9df)

Fixes: #27649

Things done

Tested using sandboxing (nix.useSandbox on NixOS, or option build-use-sandbox in nix.conf on non-NixOS)
Built on platform(s)
- NixOS
- macOS
- other Linux distributions
Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
Tested execution of all binary files (usually in ./result/bin/)
Fits CONTRIBUTING.md.

This implements 095095c4 ('python: add C++ compiler support for distutils') for python3. Should fix various problems with python packages on darwin.

GrahamcOfBorg · 2018-04-26T22:29:21Z

Success on x86_64-linux (full log)

Attempted: python3

Partial log (click to expand)

strip is /nix/store/jfnxjmhyjin5h5mz8w26g9mq0x8gy5kg-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5/lib  /nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5/bin
patching script interpreter paths in /nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5
/nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5/bin/python3.6m-config: interpreter directive changed from "/bin/sh" to "/nix/store/522ligjgrfhf2yfvy9xq7l1s788mwpwz-bash-4.4-p19/bin/sh"
/nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5/lib/python3.6/ctypes/macholib/fetch_macholib: interpreter directive changed from "/bin/sh" to "/nix/store/522ligjgrfhf2yfvy9xq7l1s788mwpwz-bash-4.4-p19/bin/sh"
/nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5/lib/python3.6/config-3.6m-x86_64-linux-gnu/makesetup: interpreter directive changed from " /bin/sh" to "/nix/store/522ligjgrfhf2yfvy9xq7l1s788mwpwz-bash-4.4-p19/bin/sh"
/nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5/lib/python3.6/config-3.6m-x86_64-linux-gnu/install-sh: interpreter directive changed from "/bin/sh" to "/nix/store/522ligjgrfhf2yfvy9xq7l1s788mwpwz-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5...
wrong ELF type
/nix/store/5yh2c6al9258dxi4m80c0q2b15ak1xdx-python3-3.6.5

GrahamcOfBorg · 2018-04-26T22:31:58Z

Success on aarch64-linux (full log)

Attempted: python3

Partial log (click to expand)

strip is /nix/store/i3k26ygq416sr5ng9f87041f3zcid9di-binutils-2.30/bin/strip
stripping (with command strip and flags -S) in /nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5/lib  /nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5/bin
patching script interpreter paths in /nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5
/nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5/lib/python3.6/config-3.6m-aarch64-linux-gnu/install-sh: interpreter directive changed from "/bin/sh" to "/nix/store/axf1bsmkzs2ckj68ihr5dhdi8z0sgwd9-bash-4.4-p19/bin/sh"
/nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5/lib/python3.6/config-3.6m-aarch64-linux-gnu/makesetup: interpreter directive changed from " /bin/sh" to "/nix/store/axf1bsmkzs2ckj68ihr5dhdi8z0sgwd9-bash-4.4-p19/bin/sh"
/nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5/lib/python3.6/ctypes/macholib/fetch_macholib: interpreter directive changed from "/bin/sh" to "/nix/store/axf1bsmkzs2ckj68ihr5dhdi8z0sgwd9-bash-4.4-p19/bin/sh"
/nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5/bin/python3.6m-config: interpreter directive changed from "/bin/sh" to "/nix/store/axf1bsmkzs2ckj68ihr5dhdi8z0sgwd9-bash-4.4-p19/bin/sh"
checking for references to /build in /nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5...
wrong ELF type
/nix/store/iv0psyf4zjgs4x6zl8dns69bzqgbiq2a-python3-3.6.5

GrahamcOfBorg · 2018-04-26T22:57:37Z

Success on aarch64-linux (full log)

Attempted: python3

Partial log (click to expand)

/nix/store/m00ppbhglkn98qiwafbqgh7fif0w9six-python3-3.6.5

GrahamcOfBorg · 2018-04-26T22:57:50Z

Success on x86_64-linux (full log)

Attempted: python3

Partial log (click to expand)

/nix/store/gpqyw2sm11kmzrdnpw7pgdr0y7h1813x-python3-3.6.5

GrahamcOfBorg · 2018-04-26T23:22:34Z

Failure on x86_64-darwin (full log)

Attempted: python3

Partial log (click to expand)

strip is /nix/store/bbm9qyr0mbpy46nvsz27kb06v8z19bx0-cctools-binutils-darwin/bin/strip
patching script interpreter paths in /nix/store/8axs3yl256g4wpibm6p7iypv0r7cn4pg-curl-7.59.0-devdoc
building '/nix/store/d62swpidd2ipkyq9pqvmyhhn5rq79brb-python-3.x-distutils-C++.patch.drv'...

trying https://bugs.python.org/file47046/python-3.x-distutils-C++.patch
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10947  100 10947    0     0  26635      0 --:--:-- --:--:-- --:--:-- 26700
cannot build derivation '/nix/store/rr5bm03pza8qfn58bl14g382miwccbfy-python3-3.6.5.drv': 2 dependencies couldn't be built
error: build of '/nix/store/rr5bm03pza8qfn58bl14g382miwccbfy-python3-3.6.5.drv' failed

GrahamcOfBorg · 2018-04-27T00:02:06Z

Failure on x86_64-darwin (full log)

Attempted: python3

Partial log (click to expand)

/nix/store/7jsv3azj6gdxwscbq4h81b5s9lafvanc-Security-osx-10.9.5/Library/Frameworks/Security.framework/Headers/cssmtype.h:2023:3: note: 'CSSM_QUERY' has been explicitly marked deprecated here
} CSSM_QUERY DEPRECATED_IN_MAC_OS_X_VERSION_10_7_AND_LATER, *CSSM_QUERY_PTR DEPRECATED_IN_MAC_OS_X_VERSION_10_7_AND_LATER;
  ^
In file included from DeviceOnHold.c:38:
In file included from ./SystemConfiguration.framework/Headers/SystemConfiguration.h:121:
In file included from ./SystemConfiguration.framework/Headers/SCPreferences.h:38:
In file included from /nix/store/7jsv3azj6gdxwscbq4h81b5s9lafvanc-Security-osx-10.9.5/Library/Frameworks/Security.framework/Headers/Security.h:42:
building of '/nix/store/f0kppx79iwfxvvpjm133bxnv3fbrilaa-configd-osx-10.8.5.drv' timed out after 1800 seconds
cannot build derivation '/nix/store/q19y11kzv19fwfn30zdsrs1l2rwqs98k-python3-3.6.5.drv': 1 dependencies couldn't be built
�[31;1merror:�[0m build of '/nix/store/q19y11kzv19fwfn30zdsrs1l2rwqs98k-python3-3.6.5.drv' failed

veprbl · 2018-04-27T21:30:20Z

I've verified that hasDistutilsCxxPatch is needed for tests of python3Packages.cython to succeed (same as I was previously seeing for python2Packages.cython). Perhaps this was not caught in #27649 since the tests were completely disabled until recently (#37396).

knedlsepp · 2018-04-29T21:50:06Z

@FRidh: This would unbreak Cython on darwin.

knedlsepp · 2018-04-29T21:49:16Z

pkgs/development/interpreters/python/cpython/3.4/default.nix

    ++ optionals x11Support [ tcl tk libX11 xproto ]
    ++ optionals stdenv.isDarwin [ CF configd ];

+  hasDistutilsCxxPatch = !(stdenv.cc.isGNU or false);


How about needsDistutilsCxxPatch?

@knedlsepp The meaning depends on the context. The point to have this attribute in the first place is to query python.hasDistutilsCxxPatch from numpy so that its numpy counterpart can be engaged (it won't work otherwise). But PR's are welcome if you still want to change it.

knedlsepp · 2018-05-04T19:55:16Z

@FRidh Any concerns?

vcunat · 2018-07-03T08:22:07Z

After minor update 78e0521 the patch no longer applies (staging).

veprbl · 2018-07-03T15:42:57Z

I guess this is caused by python/cpython@10f715d

veprbl requested a review from FRidh as a code owner April 26, 2018 22:05

veprbl mentioned this pull request Apr 26, 2018

pythonPackages.cython: 0.27.3 -> 0.28.1 #37396

Closed

8 tasks

GrahamcOfBorg added the 6.topic: python Python is a high-level, general-purpose programming language. label Apr 26, 2018

veprbl force-pushed the pr/py3_distutils_fix branch from 96ae2dc to ea08d81 Compare April 26, 2018 22:14

GrahamcOfBorg added 10.rebuild-darwin: 501+ This PR causes many rebuilds on Darwin and should normally target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. labels Apr 26, 2018

python3: add C++ compiler support for distutils

31010e0

This implements 095095c4 ('python: add C++ compiler support for distutils') for python3. Should fix various problems with python packages on darwin.

veprbl force-pushed the pr/py3_distutils_fix branch from ea08d81 to 31010e0 Compare April 26, 2018 22:27

7c6f434c mentioned this pull request Apr 26, 2018

Chromium meta error broke the bot testing pipeline #39575

Closed

GrahamcOfBorg added 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. and removed 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. labels Apr 26, 2018

knedlsepp mentioned this pull request Apr 29, 2018

python3: Add C++ compiler support for distutils (on non gcc systems) #27649

Closed

8 tasks

knedlsepp reviewed Apr 29, 2018

View reviewed changes

knedlsepp mentioned this pull request Apr 29, 2018

'-fno-strict-overflow'-warnings litter certain builds on darwin #39687

Closed

matthewbauer merged commit 1d3fe7d into NixOS:staging May 9, 2018

vcunat mentioned this pull request Jul 3, 2018

python35, python36: fix reading large files on darwin #39627

Merged

8 tasks

veprbl mentioned this pull request Jul 3, 2018

python3 distutils patch maintenance #42925

Merged

9 tasks

veprbl deleted the pr/py3_distutils_fix branch December 1, 2020 16:56

Uh oh!

Conversation

veprbl commented Apr 26, 2018

Motivation for this change

Things done

Uh oh!

GrahamcOfBorg commented Apr 26, 2018

Uh oh!

GrahamcOfBorg commented Apr 26, 2018

Uh oh!

GrahamcOfBorg commented Apr 26, 2018

Uh oh!

GrahamcOfBorg commented Apr 26, 2018

Uh oh!

GrahamcOfBorg commented Apr 26, 2018

Uh oh!

GrahamcOfBorg commented Apr 27, 2018

Uh oh!

veprbl commented Apr 27, 2018

Uh oh!

knedlsepp commented Apr 29, 2018

Uh oh!

knedlsepp Apr 29, 2018

Choose a reason for hiding this comment

Uh oh!

veprbl Apr 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

knedlsepp commented May 4, 2018

Uh oh!

vcunat commented Jul 3, 2018

Uh oh!

veprbl commented Jul 3, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Comments

veprbl Apr 29, 2018 •

edited

Loading