Fix #6163: Default to setuptools.build_meta:legacy #6212

ncoghlan · 2019-01-28T13:26:13Z

The main setuptools PEP 517 backend is intended for
explicit usage in pyproject.toml, when the project
authors can ensure that their setup.py runs without
that directory being implicitly on sys.path.

For implicit usage, setuptools now offers a separate
legacy backend that more closely mimics direct
execution of the setup.py script.

ncoghlan · 2019-01-28T13:35:40Z

Note that the CI is expected to fail, as successfully testing this currently requires a manual build step as described in #6210 (comment)

However, I wanted to give folks reassurance that the horrible hack in #6210 really is temporary, and just a way to take a bit of the urgency off the setuptools folks, since this approach involves a public API change for them, and hence shouldn't be rushed.

ncoghlan · 2019-01-28T14:05:49Z

Note that this approach poses a problem for projects with a non-wildcard setuptools dependency in their build requirements:

[ncoghlan@localhost pip]$ python3 src/pip install --user PyInstaller==3.4
Collecting PyInstaller==3.4
  Downloading https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz (3.5MB)
    100% |████████████████████████████████| 3.5MB 5.8MB/s 
  Installing build dependencies ... done
Some build dependencies for PyInstaller==3.4 from https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz#sha256=a5a6e04a66abfcf8761e89a2ebad937919c6be33a7b8963e1a961b55cb35986b conflict with PEP 517/518 supported requirements: setuptools==40.7.0 is incompatible with setuptools>=40.7.0.post20190128.

ncoghlan · 2019-01-28T14:18:24Z

The above error was just an artifact of my failing to pass in the local directory with the pre-built wheel file in it. However, solving that still wasn't enough to get a successful build:

[ncoghlan@localhost pip]$ python3 src/pip install --no-cache --user -f /tmp/setuptools_build_meta_legacy/ PyInstaller==3.4
Looking in links: /tmp/setuptools_build_meta_legacy/
Collecting PyInstaller==3.4
  Downloading https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz (3.5MB)
    100% |████████████████████████████████| 3.5MB 11.7MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: macholib>=1.8 in /home/ncoghlan/.local/lib/python3.7/site-packages (from PyInstaller==3.4) (1.11)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from PyInstaller==3.4) (39.0.1)
Requirement already satisfied: altgraph in /home/ncoghlan/.local/lib/python3.7/site-packages (from PyInstaller==3.4) (0.16.1)
Requirement already satisfied: pefile>=2017.8.1 in /home/ncoghlan/.local/lib/python3.7/site-packages (from PyInstaller==3.4) (2018.8.8)
Requirement already satisfied: future in /home/ncoghlan/.local/lib/python3.7/site-packages (from pefile>=2017.8.1->PyInstaller==3.4) (0.17.1)
Exception:
Traceback (most recent call last):
  File "src/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "src/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "src/pip/_internal/wheel.py", line 904, in build
    assert have_directory_for_build
AssertionError

ncoghlan · 2019-01-28T15:04:49Z

Ah, that assertion error looks like an unrelated bad interaction with the --no-cache option (I tested #6210 first, so I hadn't needed --no-cache there, I only added it in when testing this one).

Testing with cache-dir instead gives a successful install as we'd expect:

[ncoghlan@localhost pip]$ python3 src/pip install --cache-dir /tmp/pip_cache_dir --user -f /tmp/setuptools_build_meta_legacy/ PyInstaller==3.4
Looking in links: /tmp/setuptools_build_meta_legacy/
Collecting PyInstaller==3.4
  Downloading https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz (3.5MB)
    100% |████████████████████████████████| 3.5MB 8.2MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: macholib>=1.8 in /home/ncoghlan/.local/lib/python3.7/site-packages (from PyInstaller==3.4) (1.11)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from PyInstaller==3.4) (39.0.1)
Requirement already satisfied: altgraph in /home/ncoghlan/.local/lib/python3.7/site-packages (from PyInstaller==3.4) (0.16.1)
Requirement already satisfied: pefile>=2017.8.1 in /home/ncoghlan/.local/lib/python3.7/site-packages (from PyInstaller==3.4) (2018.8.8)
Requirement already satisfied: future in /home/ncoghlan/.local/lib/python3.7/site-packages (from pefile>=2017.8.1->PyInstaller==3.4) (0.17.1)
Building wheels for collected packages: PyInstaller
  Building wheel for PyInstaller (PEP 517) ... done
  Stored in directory: /tmp/pip_cache_dir/wheels/09/ee/0a/94ef5d39074625f49e2e9cf97ac30cb7a87e1a7458ed195b8d
Successfully built PyInstaller
Installing collected packages: PyInstaller
Successfully installed PyInstaller-3.4

ncoghlan · 2019-01-29T12:49:33Z

Turns out there's already an issue filed for the AssertionError I ran into above: #6197

ncoghlan · 2019-02-01T14:58:31Z

Closing in favour of #6229 as a near-term resolution of #6163 (see #6163 (comment) for more details)

We'll presumably revisit this in a future release, but that can be a new issue and new PR (with the tests rolled in to #6229, there really won't be a lot of changes needed to update to a new setuptools backend later)

ncoghlan · 2019-02-01T16:09:18Z

Reopened, but removed the reference to #6163 from the title.

Assuming #6229 is accepted as the near term resolution of #6163, there'll need to be a follow-up issue to replace the workaround with the proper long term fix that doesn't require any xfail markers in the test suite.

ncoghlan · 2019-02-02T05:23:13Z

Just noting that once switching to the updated PEP 517 implementation, the setuptools version used is mentioned in docs/pip.rst

ncoghlan · 2019-02-05T12:24:21Z

It's looking like this is going to be the actual resolution for #6163 in pip 19.0.2, so updating based on some assumptions about the next setuptools release and added the issue reference back to the PR title.

However, it will remain WIP until setuptools is published and we know exactly what the new version number and backend name actually are.

uranusjr · 2019-02-06T02:13:35Z

The new version has been released as 40.8.0 🎉

$ ./pip-6212/bin/python -m pip install git+https://github.com/ncoghlan/pip.git@issue-6163-use-setuptools-build-meta-legacy-backend
Collecting git+https://github.com/ncoghlan/pip.git@issue-6163-use-setuptools-build-meta-legacy-backend
  Cloning https://github.com/ncoghlan/pip.git (to revision issue-6163-use-setuptools-build-meta-legacy-backend) to /private/var/folders/52/hsf94kc92lxbgn7gx3vy2qsr0000gn/T/pip-req-build-osolyj99
Branch 'issue-6163-use-setuptools-build-meta-legacy-backend' set up to track remote branch 'issue-6163-use-setuptools-build-meta-legacy-backend' from 'origin'.
Switched to a new branch 'issue-6163-use-setuptools-build-meta-legacy-backend'
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: pip
  Building wheel for pip (PEP 517) ... done
  Stored in directory: /private/var/folders/52/hsf94kc92lxbgn7gx3vy2qsr0000gn/T/pip-ephem-wheel-cache-hrif2deg/wheels/a5/ca/56/003c9018bedcc3b24f5e337d9f0a885835d232a79fe0cb11ff
Successfully built pip
Installing collected packages: pip
  Found existing installation: pip 19.0.1
    Uninstalling pip-19.0.1:
      Successfully uninstalled pip-19.0.1
Successfully installed pip-19.1.dev0

$ ./pip-6212/bin/python -m pip install pyinstaller
Collecting pyinstaller
  Downloading https://files.pythonhosted.org/packages/03/32/0e0de593f129bf1d1e77eed562496d154ef4460fd5cecfd78612ef39a0cc/PyInstaller-3.4.tar.gz (3.5MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting pefile>=2017.8.1 (from pyinstaller)
  Downloading https://files.pythonhosted.org/packages/ed/cc/157f20038a80b6a9988abc06c11a4959be8305a0d33b6d21a134127092d4/pefile-2018.8.8.tar.gz (62kB)
Collecting altgraph (from pyinstaller)
  Downloading https://files.pythonhosted.org/packages/0a/cc/646187eac4b797069e2e6b736f14cdef85dbe405c9bfc7803ef36e4f62ef/altgraph-0.16.1-py2.py3-none-any.whl
Requirement already satisfied: setuptools in ./pip-6212/lib/python3.7/site-packages (from pyinstaller) (40.8.0)
Collecting macholib>=1.8 (from pyinstaller)
  Downloading https://files.pythonhosted.org/packages/41/f1/6d23e1c79d68e41eb592338d90a33af813f98f2b04458aaf0b86908da2d8/macholib-1.11-py2.py3-none-any.whl
Collecting future (from pefile>=2017.8.1->pyinstaller)
  Downloading https://files.pythonhosted.org/packages/90/52/e20466b85000a181e1e144fd8305caf2cf475e2f9674e797b222f8105f5f/future-0.17.1.tar.gz (829kB)
Building wheels for collected packages: pyinstaller
  Building wheel for pyinstaller (PEP 517) ... done
  Stored in directory: /Users/uranusjr/Library/Caches/pip/wheels/09/ee/0a/94ef5d39074625f49e2e9cf97ac30cb7a87e1a7458ed195b8d
Successfully built pyinstaller
Building wheels for collected packages: pefile, future
  Building wheel for pefile (setup.py) ... done
  Stored in directory: /Users/uranusjr/Library/Caches/pip/wheels/d9/c6/56/a372ee1059440413d55d13f78adfd16c8719553a5b4efdd0c1
  Building wheel for future (setup.py) ... done
  Stored in directory: /Users/uranusjr/Library/Caches/pip/wheels/0c/61/d2/d6b7317325828fbb39ee6ad559dbe4664d0896da4721bf379e
Successfully built pefile future
Installing collected packages: future, pefile, altgraph, macholib, pyinstaller
Successfully installed altgraph-0.16.1 future-0.17.1 macholib-1.11 pefile-2018.8.8 pyinstaller-3.4

pradyunsg

These changes all look good to me @ncoghlan!

If it isn't too much pain, could you rewrite the commit history of this PR? If it is more work than you'd like, I'll be happy to do so myself.

src/pip/_internal/pyproject.py

ncoghlan · 2019-02-06T10:50:22Z

Squashed everything into one commit and removed the out of date comment. Hopefully CI will agree this is now good to go :)

ncoghlan · 2019-02-06T10:56:59Z

Missed a couple of test cases that referenced the exact implicit setuptools version. I also have a couple of markup and docstring fixes to add.

The main setuptools PEP 517 backend is intended for explicit usage in `pyproject.toml`, when the project authors can ensure that their `setup.py` runs without that directory being implicitly on `sys.path`. For implicit usage, setuptools now offers a separate legacy backend that more closely mimics direct execution of the `setup.py` script.

ncoghlan · 2019-02-06T11:32:32Z

@pradyunsg @pfmoore Latest push looks good to go to me - if you spot anything that needs tweaking before merging, please don't feel obliged to wait for me to comment before fixing it (I'm about to head to bed and won't be back online until tomorrow evening my time).

pradyunsg · 2019-02-06T11:48:37Z

Great! Thanks @ncoghlan! :)

lock · 2019-05-29T00:05:13Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

ncoghlan changed the title ~~Issue #6163: Default to setuptools.build_meta_legacy backend~~ WIP Issue #6163: Default to setuptools.build_meta_legacy backend Jan 28, 2019

ncoghlan mentioned this pull request Jan 28, 2019

Issue #6163: Temporary workaround for legacy setup.py files #6210

Closed

This was referenced Jan 28, 2019

Add build_meta_legacy backend pypa/setuptools#1652

Merged

pip 19.0 fails to install packages that import to-be-installed package from CWD #6163

Closed

cjerdonek mentioned this pull request Jan 29, 2019

Release 19.0 -- New year, New release #6106

Closed

ncoghlan mentioned this pull request Jan 29, 2019

have_directory_for_build AssertionError when installing with --no-cache-dir (19.0.1) #6197

Closed

cjerdonek added type: bugfix C: PEP 517 impact Affected by PEP 517 processing labels Jan 30, 2019

takluyver mentioned this pull request Feb 1, 2019

pep517 should either have wheel as an install dependency, or provide methods to get package dependencies without the wheel tool pypa/pyproject-hooks#39

Closed

ncoghlan closed this Feb 1, 2019

ncoghlan changed the title ~~WIP Issue #6163: Default to setuptools.build_meta_legacy backend~~ WIP Issue TBD: Default to setuptools.build_meta_legacy backend Feb 1, 2019

ncoghlan reopened this Feb 1, 2019

ncoghlan mentioned this pull request Feb 2, 2019

Issue #6163: Postpone using PEP 517 by default #6229

Closed

pradyunsg changed the title ~~WIP Issue TBD: Default to setuptools.build_meta_legacy backend~~ [WIP] Default to setuptools.build_meta_legacy backend Feb 3, 2019

pradyunsg changed the title ~~[WIP] Default to setuptools.build_meta_legacy backend~~ WIP Issue TBD: Default to setuptools.build_meta_legacy backend Feb 3, 2019

ncoghlan changed the title ~~WIP Issue TBD: Default to setuptools.build_meta_legacy backend~~ WIP Issue #6163: Default to setuptools setup.py compatibility backend Feb 5, 2019

pradyunsg added this to the 19.0 milestone Feb 6, 2019

pradyunsg approved these changes Feb 6, 2019

View reviewed changes

src/pip/_internal/pyproject.py Outdated Show resolved Hide resolved

ncoghlan force-pushed the issue-6163-use-setuptools-build-meta-legacy-backend branch from e9043b2 to 500c87a Compare February 6, 2019 10:47

ncoghlan changed the title ~~WIP Issue #6163: Default to setuptools setup.py compatibility backend~~ Fix #6163: Default to setuptools.build_meta:__legacy__ Feb 6, 2019

ncoghlan force-pushed the issue-6163-use-setuptools-build-meta-legacy-backend branch from 500c87a to 682cff7 Compare February 6, 2019 10:58

pfmoore approved these changes Feb 6, 2019

View reviewed changes

pradyunsg approved these changes Feb 6, 2019

View reviewed changes

pradyunsg merged commit e92560b into pypa:master Feb 6, 2019

jorisvandenbossche mentioned this pull request Feb 8, 2019

Specify build requirements in pyproject.toml (PEP 517) pandas-dev/pandas#25227

Closed

4 tasks

lock bot added the auto-locked Outdated issues that have been locked by automation label May 29, 2019

lock bot locked as resolved and limited conversation to collaborators May 29, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #6163: Default to setuptools.build_meta:legacy #6212

Fix #6163: Default to setuptools.build_meta:legacy #6212

ncoghlan commented Jan 28, 2019 •

edited

Loading

ncoghlan commented Jan 28, 2019 •

edited

Loading

ncoghlan commented Jan 28, 2019

ncoghlan commented Jan 28, 2019

ncoghlan commented Jan 28, 2019

ncoghlan commented Jan 29, 2019

ncoghlan commented Feb 1, 2019

ncoghlan commented Feb 1, 2019

ncoghlan commented Feb 2, 2019

ncoghlan commented Feb 5, 2019

uranusjr commented Feb 6, 2019 •

edited

Loading

pradyunsg left a comment

ncoghlan commented Feb 6, 2019

ncoghlan commented Feb 6, 2019

ncoghlan commented Feb 6, 2019

pradyunsg commented Feb 6, 2019

lock bot commented May 29, 2019

Fix #6163: Default to setuptools.build_meta:__legacy__ #6212

Fix #6163: Default to setuptools.build_meta:__legacy__ #6212

Conversation

ncoghlan commented Jan 28, 2019 • edited Loading

ncoghlan commented Jan 28, 2019 • edited Loading

ncoghlan commented Jan 28, 2019

ncoghlan commented Jan 28, 2019

ncoghlan commented Jan 28, 2019

ncoghlan commented Jan 29, 2019

ncoghlan commented Feb 1, 2019

ncoghlan commented Feb 1, 2019

ncoghlan commented Feb 2, 2019

ncoghlan commented Feb 5, 2019

uranusjr commented Feb 6, 2019 • edited Loading

pradyunsg left a comment

Choose a reason for hiding this comment

ncoghlan commented Feb 6, 2019

ncoghlan commented Feb 6, 2019

ncoghlan commented Feb 6, 2019

pradyunsg commented Feb 6, 2019

lock bot commented May 29, 2019

Fix #6163: Default to setuptools.build_meta:legacy #6212

Fix #6163: Default to setuptools.build_meta:legacy #6212

ncoghlan commented Jan 28, 2019 •

edited

Loading

ncoghlan commented Jan 28, 2019 •

edited

Loading

uranusjr commented Feb 6, 2019 •

edited

Loading