Why is the EditableBuilder running pip without --no-deps ?

[dmerejkowsky]

We've got a problem involving this repo at work and we are still trying to figure out exactly what the problem is - when we do, we'll update this issue :)

But during our investigation, we came across these lines in poet's EditableBuilder sources:

class EditableBuilder(Builder):
    def build(self):
        return self._setup_build()

    def _setup_build(self):
        builder = SdistBuilder(self._poetry, self._env, self._io)
        setup = self._path / "setup.py"
        ...
        try:
           ...
           self._env.run_pip("install", "-e", str(self._path))
        finaly:
          ...

My question is: why is pip run without the --no-deps flags? Is this an oversight, or is something else going on here?

[finswimmer]

Hello @dmerejkowsky,

as far as a understood this part of the code, it is only used to install your current project in editable mode. This step is always the very last step during poetry install. So all dependencies of this package should be already installed at this point.

I don't know if it would hurts the --no-deps flag at this place as well.

fin swimmer

[dmerejkowsky]

it is only used to install your current project in editable mode [...] all dependencies of this package should be already installed at this point.

That's what I thought, too. Guess I'll make a PR and see what happens :)

[dmerejkowsky]

So, here's what we've figured out.

Context

The tankersdk project

We have a project named tankersdk that uses CFFI's extensions.

In order to build the project, we use a proprietary module called ci, which in turn depends on conan (to build the C++ libraries used by the extensions).

Here are the relevant snippets of code:

# In pyproject.toml
[tool.poetry]
name = "tankersdk"
build = "build.py"

[tool.poetry.dependencies]
python = "^3.7"

cffi = "^1.13.2"
ci = "2020.2.2093"

# In build.py
def build(setup_kwargs):
     setup_kwargs["cffi_modules"] = [ .... ]

The Conan project

It should be noted the conan project contains the following configurations:

# conans/requirements_osx.txt
idna==2.6 # Solving conflict, somehow is installing 2.7 when requests require 2.6

# in conan/setup.py
project_requirements = get_requires("conans/requirements.txt")
if platform.system() == "Darwin":
        project_requirements.extend(get_requires("conans/requirements_osx.txt"))

The poetry lock

Finally, note that we've generated the lock on Linux, so the resolved deps are:

# in poetry.lock
[[package]]
category = "main"
name = "idna"
...
version = "2.9"

[[package]]
category = "main"
name = "conan"
...
version = "1.19.2"

but the package is built on Linux, macOS, and Windows.

Also, despite what the comment says, conan installed via poetry has no problem running on macOS - with conan==1.19.2 and idna==2.9

The bug

So, during the macOS CI we run poetry install at the root, of the tankersdk project, in order to build the extension, which in turn calls pip install -e.

And that's where things go wrong: pip sees the setup.py generated by poetry, starts parsing the requirements of the ci packages, then the conan package, and since sys.platform equals darwin, it tries to install idna==2.6 and fails with:

 pip._vendor.pkg_resources.ContextualVersionConflict: 
  (idna 2.9 
  (/.../virtualenvs/tankersdk-UeWgm17z-py3.7/lib/python3.7/site-packages),  
  Requirement.parse('idna==2.6'), {'conan'})

[abn]

@dmerejkowsky jfyi; the conan issue was resolved in conan-io/conan#6797 via conan-io/conan#6798.

[dmerejkowsky]

That's good to know, thanks !

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.