"conflict with the backend dependencies" error running pip install in editable mode with pyproject.toml / PEP-517 enabled project

[cjw296]

Environment

• pip version: 19.0.2
• Python version: 2.7.15
• OS: travis xenial

Description

Around a day ago, my cron-triggered builds on Travis for one projects started failing with Some build dependencies for file:///home/travis/build/cjw296/carly conflict with the backend dependencies: attrs==19.1.0 is incompatible with attrs<19.0 ,>=18.2. when being installed by poetry. Looks like under the hood, poetry is calling ['/home/travis/virtualenv/python2.7.15/bin/pip', 'install', '-e', '/ home/travis/build/cjw296/carly', '--no-deps'] which would suggest that dependencies should be ignored.

The full build log for this first failure is here: https://travis-ci.org/cjw296/carly/jobs/501041034

There is a poetry issue open for this here: python-poetry/poetry#826

It's unclear whether this is a poetry bug or a pip bug, but it appears that either downgrading to pip 18.x or removing the build-system section from pyproject.toml stop this happening.

What's super confusing is that my particular builds are run nightly and they only started failing on 3rd March, even though neither pip nor poetry had any releases around that date, and I haven't made any code changes to carly since 20th Feb.

Expected behavior

This build log shows expected behaviour:
https://travis-ci.org/cjw296/carly/jobs/500733549

How to Reproduce

These are the steps:
https://github.com/cjw296/carly/blob/6ec7eedea9c8405ea6a61f733cb7ee9dfaa9b801/.travis.yml#L21-L23

These should be run against https://github.com/cjw296/carly/tree/6ec7eedea9c8405ea6a61f733cb7ee9dfaa9b801

Output

$ curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python
Retrieving Poetry metadata
# Welcome to Poetry!
This will download and install the latest version of Poetry,
a dependency and package manager for Python.
It will add the `poetry` command to Poetry's bin directory, located at:
$HOME/.poetry/bin
This path will then be added to your `PATH` environment variable by
modifying the profile files located at:
$HOME/.profile
$HOME/.bash_profile
You can uninstall at any time with `poetry self:uninstall`,
or by executing this script with the --uninstall option,
and these changes will be reverted.
Installing version: 0.12.11
  - Downloading poetry-0.12.11-linux.tar.gz (8.00MB)
Poetry (0.12.11) is installed now. Great!
To get started you need Poetry's bin directory ($HOME/.poetry/bin) in your `PATH`
environment variable. Next time you log in this will be done
automatically.
To configure your current shell run `source $HOME/.poetry/env`
install.2
0.00s$ source $HOME/.poetry/env
23.50s$ poetry install
Updating dependencies
Resolving dependencies... (5.7s)
Package operations: 12 installs, 1 update, 0 removals
Writing lock file
  - Installing idna (2.8)
  - Updating six (1.11.0 -> 1.12.0)
  - Installing automat (0.7.0)
  - Installing constantly (15.1.0)
  - Installing hyperlink (18.0.0)
  - Installing incremental (17.5.0)
  - Installing pyhamcrest (1.9.0)
  - Installing txaio (18.8.1)
  - Installing zope.interface (4.6.0)
  - Installing autobahn (18.12.1)
  - Installing coverage (4.4.2)
  - Installing testfixtures (6.6.0)
  - Installing twisted (18.9.0)
  - Installing carly (0.9.1)
                                                                                
[EnvCommandError]                                                
Command ['/home/travis/virtualenv/python2.7.15/bin/pip', 'install', '-e', '/        
home/travis/build/cjw296/carly', '--no-deps'] errored with the following out  
put:                                                                          
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020.  
 Please upgrade your Python as Python 2.7 won't be maintained after that dat  
e. A future version of pip will drop support for Python 2.7.                  
Obtaining file:///home/travis/build/cjw296/carly                              
  Installing build dependencies: started                                      
  Installing build dependencies: finished with status 'done'                  
  Getting requirements to build wheel: started                                
  Getting requirements to build wheel: finished with status 'done'            
Some build dependencies for file:///home/travis/build/cjw296/carly conflict   
with the backend dependencies: attrs==19.1.0 is incompatible with attrs<19.0  
,>=18.2.                                                                      
                                                                                
install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]
The command "poetry install" failed and exited with 1 during .
Your build has been stopped.

[cjerdonek]

Note that it looks like attrs just released 19.1, which could explain why it just started happening:
https://github.com/python-attrs/attrs/releases . Can you simplify to a reproducer that doesn’t involve poetry?

[cjerdonek]

Does poetry auto-generate a setup.py? Because the logs show pip being used to install carly in editable mode, but pip doesn't support editable mode without a setup.py: #6206 , and carly doesn't seem like it has a setup.py.

[cjw296]

Good spot on the attrs release. What's interesting here is that while carly specifies "attrs 18.x", poetry doesn't depend on attrs at all. (although, of course, one of its dependencies might?)

Answers in order:

• Reproducer without poetry: I suspect poetry is intrinsic in this, so no ;-) pip 19 introduced support for build-system in a pyproject.toml, if I remember rightly, and carly that section pointing at poetry.

• Does poetry auto-generate a setup.py? Not in the checkout, as far I'm aware. There's mention of it in the changelog, but I'd been under the impression it was just in the sdist.

[cjerdonek]

If you look at poetry's source code, here is where it auto-generates a setup.py prior to calling pip install in editable mode: https://github.com/sdispater/poetry/blob/0222bf6bcbe0a55bfe2c5d593beb667a47b8a752/poetry/console/commands/install.py#L85-L89

if has_setup:
    self.line("<warning>A setup.py file already exists. Using it.</warning>")
else:
    with setup.open("w", encoding="utf-8") as f:
        f.write(decode(builder.build_setup()))

try:
    self.env.run("pip", "install", "-e", str(setup.parent), "--no-deps")

[cjw296]

Some pdb'ing later, yes, it looks like poetry does generate a setup.py (and then delete it):

(Pdb) print e
Command '['/Users/chris/Library/Caches/pypoetry/virtualenvs/carly-py2.7/bin/pip', 'install', '-e', '/Users/chris/vcs/git/carly', '--no-deps']' returned non-zero exit status 1

$ cat /Users/chris/vcs/git/carly/setup.py 
# -*- coding: utf-8 -*-
from distutils.core import setup

packages = \
['carly']

package_data = \
{'': ['*']}

install_requires = \
['Twisted>=18.9,<19.0', 'attrs>=18.2,<19.0']

setup_kwargs = {
    'name': 'carly',
    'version': '0.9.1',
    'description': 'A tool for putting messages into and collecting responses from Twisted servers using real networking',
    'long_description': None,
    'author': 'Chris Withers',
    'author_email': '[email protected]',
    'url': None,
    'packages': packages,
    'package_data': package_data,
    'install_requires': install_requires,
}


setup(**setup_kwargs)
(carly-py2.7)(master) Mac Pro:carly chris:

[cjw296]

@cjerdonek - hah! crossing the streams... okay, so what is requiring attrs >=19?

[cjerdonek]

I don't know.. I would suggest trying to reproduce the issue (if there is one) independent of poetry and Travis CI, as there are currently a lot of confounding factors. Hopefully this provides more of a start. You might also want to try examining what's installed in the environment at the time poetry invokes pip install.

[cjw296]

Right, here's the answer courtesy of @wimglenn's excellent johnnydep for poetry:

name                                         summary
-------------------------------------------  -------------------------------------------------------------------------
poetry                                       Python dependency management and packaging made easy.
├── cachecontrol[filecache]<0.13.0,>=0.12.4  httplib2 caching for requests
│   ├── lockfile>=0.9                        Platform-independent file locking module
│   ├── msgpack                              MessagePack (de)serializer.
│   └── requests                             Python HTTP for Humans.
│       ├── certifi>=2017.4.17               Python package for providing Mozilla's CA Bundle.
│       ├── chardet<3.1.0,>=3.0.2            Universal encoding detector for Python 2 and 3
│       ├── idna<2.9,>=2.5                   Internationalized Domain Names in Applications (IDNA)
│       └── urllib3<1.25,>=1.21.1            HTTP library with thread-safe connection pooling, file post, and more.
├── cachy<0.3,>=0.2                          Cachy provides a simple yet effective caching library.
├── cleo<0.7.0,>=0.6.7                       Cleo allows you to create beautiful and testable command-line interfaces.
│   ├── pastel<0.2.0,>=0.1.0                 Bring colors to your terminal.
│   └── pylev<2.0,>=1.3                      A pure Python Levenshtein implementation that's not freaking GPL'd.
├── html5lib<2.0,>=1.0                       HTML parser based on the WHATWG HTML specification
│   ├── six>=1.9                             Python 2 and 3 compatibility utilities
│   └── webencodings                         Character encoding aliases for legacy web content
├── jsonschema<4.0,>=3.0a3                   An implementation of JSON Schema validation for Python
│   ├── attrs>=17.4.0                        Classes Without Boilerplate
│   ├── pyrsistent>=0.14.0                   Persistent/Functional/Immutable data structures
│   │   └── six                              Python 2 and 3 compatibility utilities
│   ├── setuptools                           Easily download, build, install, upgrade, and uninstall Python packages
│   └── six>=1.11.0                          Python 2 and 3 compatibility utilities
├── pkginfo<2.0,>=1.4                        Query metadatdata from sdists / bdists / installed packages.
├── pyparsing<3.0,>=2.2                      Python parsing module
├── pyrsistent<0.15.0,>=0.14.2               Persistent/Functional/Immutable data structures
│   └── six                                  Python 2 and 3 compatibility utilities
├── requests-toolbelt<0.9.0,>=0.8.0          A utility belt for advanced users of python-requests
│   └── requests<3.0.0,>=2.0.1               Python HTTP for Humans.
│       ├── certifi>=2017.4.17               Python package for providing Mozilla's CA Bundle.
│       ├── chardet<3.1.0,>=3.0.2            Universal encoding detector for Python 2 and 3
│       ├── idna<2.9,>=2.5                   Internationalized Domain Names in Applications (IDNA)
│       └── urllib3<1.25,>=1.21.1            HTTP library with thread-safe connection pooling, file post, and more.
├── requests<3.0,>=2.18                      Python HTTP for Humans.
│   ├── certifi>=2017.4.17                   Python package for providing Mozilla's CA Bundle.
│   ├── chardet<3.1.0,>=3.0.2                Universal encoding detector for Python 2 and 3
│   ├── idna<2.9,>=2.5                       Internationalized Domain Names in Applications (IDNA)
│   └── urllib3<1.25,>=1.21.1                HTTP library with thread-safe connection pooling, file post, and more.
├── shellingham<2.0,>=1.1                    Tool to Detect Surrounding Shell
└── tomlkit<0.6.0,>=0.5.1                    Style preserving TOML library

So, jsonschema depends on attrs >17.4, but it looks like the time pip install -e happens happens, pip has already picked the latest attrs 19.x.

Here's a hypothesis: pip is picking the build system out of pyproject.toml during the pip install -e , installing poetry in the build env, but then getting sad when carly requests attrs 18.x, even though that's an install-time dependency.

To your comments about "confounding factors", I'm afraid I'm not sure I agree. Travis is great as we know it's using a sterile, up-to-date and reproducible environment (Circle CI does the same). The fact that poetry is involved is somewhat the crux of the matter here...

[pradyunsg]

Pinging @sdispater in case he has inputs here.

[cjerdonek]

My point was just that if an issue can't be reproduced independent of Travis and poetry, then it's not clear the issue lies with pip. If it's an issue with poetry, then it should be resolved on poetry's tracker rather than here..

[cjerdonek]

pip is picking the build system out of pyproject.toml during the pip install -e

I don't believe pip looks at pyproject.toml when run in editable mode (that's what I was saying above).

[cjw296]

Here's a minimal reproducer: https://github.com/cjw296/pip6314

It looks like pip is confusing install-time requirements of carly with build-time requirements.
I suspect this confusion comes from both a setup.py and a pyproject.toml being present.

It's unclear why, but the [tool.poetry.dependencies] section in pyproject.toml is required, otherwise you get a different error, from inside poetry.

I'm speculating, but this feels like a combination of poetry being unusual in generating a dynamic setup.py, and potentially bug(s) in both pip's PEP-517 implementation and poetry's (ab)use of pip.

[cjerdonek]

Thanks for the simple reproducer!

Some preliminary thoughts are below. Note that this is just me thinking out loud and shouldn't be interpreted as anything ironclad.

Without having investigated fully, my initial instinct is that maybe pip should be erroring out if it's in the position of being instructed to use PEP 517 when -e is passed. One reason is that I don't think pip supports this yet, and if it seems to, I don't think it was intentional and its behavior thought through.

Another reason is that PEP 517 doesn't support editable installs yet (link):

This [PEP 517] originally specified another hook, install_editable, to do an editable install (as with pip install -e). It was removed due to the complexity of the topic, but may be specified in a later PEP.

In addition, an advantage to erroring out now is that it preserves pyproject.toml with the build-backend key as being the "switch" to turning on PEP 517. Otherwise, if pip "supported" it now and PEP 517 adds support for editable installs later, existing applications would break if pip tried to implement the new behavior (unless PEP 517's description happened to follow pip's implementation exactly).

In terms of what's happening now, to avoid this issue it seems like it would address the issue and could only help if poetry either--

1. passes --no-use-pep517 when it invokes pip install -e, or

2. ensures that pyproject.toml doesn't trigger PEP 517 behavior, either by removing the pyproject.toml when it auto-generates a setup.py, or replaces the pyproject.toml with one without the build-backend key (and updating the auto-generated setup.py accordingly).

[cjw296]

1. Sadly not:

$ pip install --no-use-pep517  -e .
Obtaining file:///Users/chris/vcs/git/pip6314
Disabling PEP 517 processing is invalid: project specifies a build backend of poetry.masonry.api in pyproject.toml

2. I think you're right here, I suspect poetry should move pyproject.toml out of the way.

While it's probably "correct" that pip should error out when it finds both PEP 517 when doing a -e, in this case it would turn a bug that can be worked around into a total show stopper :-/

[cjw296]

If --no-use-pep517 could really mean that and totally ignore pyproject.toml, that feels right, and would make the change required to poetry less hacky than moving pyproject.toml out of the way...

[cjerdonek]

Hmm, I'm a little surprised --no-use-pep517 didn't work. It looks like it was coded that way deliberately:

pip/src/pip/_internal/pyproject.py

Lines 84 to 93 in 4589ed4

	elif build_system and "build-backend" in build_system:
	if use_pep517 is not None and not use_pep517:
	raise InstallationError(
	"Disabling PEP 517 processing is invalid: "
	"project specifies a build backend of {} "
	"in pyproject.toml".format(
	build_system["build-backend"]
	)
	)
	use_pep517 = True

Maybe that logic can still be adjusted (e.g. at least in the case of editable mode which isn't currently supported by PEP 517 anyways).

Regarding poetry moving pyproject.toml out of the way, it doesn't seem so hacky to me given that poetry is already manually changing things by generating a setup.py and running pip install -e. Given that this is gearing up for a setup.py-based build, it seems natural that poetry would also have to remove the thing that would cause a PEP 517-based build rather than a setup.py-based build.

[pfmoore]

If PEP 517 is being used, -e should be rejected - as @cjerdonek said, PEP 517 doesn't support editable installs.

Hmm, I'm a little surprised --no-use-pep517 didn't work. It looks like it was coded that way deliberately

That is correct. If you try to insist on not using PEP 517 in a project that has a pyproject.toml specifying a build backend, then you will change the behaviour, as a non-PEP 517 build will use setuptools rather than the specified backend (yes, it's possible that pyproject.toml says to use setuptools as a backend, but checking for that isn't easy, and the situation is rare enough that I don't think it's warranted). That sort of change in behaviour is not what --no-use-pep517 was intended for, so we reject it to make sure people don't mistakenly break their build.

I'd take a hardline view here. Given that PEP 517 doesn't support editable installs, projects shouldn't be using PEP 517 if they want to support them. If this is a problem, then the solution should be for someone to do the work of defining a PEP 517 compatible standard for editable installs. Then pip (and any other frontends that are developed) can support it cleanly for any backend that provides editable install support.

Regarding poetry moving pyproject.toml out of the way, it doesn't seem so hacky to me given that poetry is already manually changing things by generating a setup.py and running pip install -e. Given that this is gearing up for a setup.py-based build, it seems natural that poetry would also have to remove the thing that would cause a PEP 517-based build rather than a setup.py-based build.

+1 on this. I think that the standards should support poetry, and I'd be more than happy if the poetry project engaged with the standards process to make sure that happens. But as things stand right now, poetry seems to be doing its own thing and so adding some additional customisation to cover this situation seems entirely reasonable to me.

[cjerdonek]

Okay, thanks a lot for weighing in here, @pfmoore. Then the way forward on this issue in the short-term (separate from the longer term effort to expand PEP 517 to cover editable installs) is a PR for pip to "fail fast" with a clear message if someone is trying to install in editable mode a project that requires PEP 517 to be used.

[pfmoore]

Yes, that sounds reasonable.

[cjw296]

Hmm, while I don't disagree on the reasoning, that "bugfix" will kill poetry or force poetry users to say on pip 18, at a time when @sdispater doesn't appear to be available to work on poetry.

@pfmoore - as far as "poetry seems to be doing its own thing" goes, I think you pointed out why that is: there's no standard for editable installs, so I feel your wording is a little harsh. I wonder how flit does editable installs? @takluyver? (or does flit only deal with building wheels, rather than managing a development environment?)

[pfmoore]

Sorry if I sounded harsh, maybe I wasn't clear. All I meant was that as poetry is already modifying files here, moving pyproject.toml out of the way seems like a reasonable and fairly simple fix. (Basically I was just agreeing with @cjerdonek)

Flit doesn't support editable installs. That's precisely the reason editable installs were dropped from PEP 517 - there's no clear view on how any backend other than setuptools can handle them, so short of blessing setuptools' behaviour (and risking getting into a situation where no other backend can offer editable support, because the spec is too tided to details of how setuptools does things) it was better to defer support. Well, that plus the fact that everyone was so burned out getting as far as we did, that no-one wanted to open up yet another can of worms at that point.

[cjerdonek]

Hmm, while I don't disagree on the reasoning, that "bugfix" will kill poetry or force poetry users to say on pip 18, at a time when @sdispater doesn't appear to be available to work on poetry.

Remember @cjw296 that there's a fairly targeted (perhaps even simple and straightforward) change that poetry can make to resolve things, as we discussed above. So I wouldn't say it would "kill" poetry.

[takluyver]

Flit doesn't support editable installs.

Flit does, but not through the PEP 517 APIs, so you can't use pip to get an editable install of a Flit-packaged project. You can run flit install --symlink, or someone added --pth-file more recently for some platform which doesn't have good symlink support. 😉

My rationale is partly that the point of an editable install is to test a package you are actively working on, so it makes sense to set it up through a package authoring tool (flit) rather than a package consumption tool (pip). Plus, as you say, we were all exhausted by what we did try to standardise.

[pfmoore]

Thanks for the clarification. Your rationale makes sense, although there's a strong push from certain quarters for everything to be accessible via pip (or more precisely "the build frontend"). So I suspect the issue of having editable install support as an extension to PEP 517 could re-emerge at some point.

[takluyver]

Yup, I'm not against trying to specify some hook for editable installs. But I'm happy enough to let it rest for now.

[cjerdonek]

I started work on a PR for this here: #6331. (There is still a failing test I need to sort out after making the change.)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.