Solver breaks with related dependencies that are both conditional

[anthonymichaelclark]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS Monterey 12.3.1

• Poetry version: 1.1.13

• Link of a Gist with the contents of your pyproject.toml file: Here

Issue

The solver breaks when faced with dependencies A and B that are both conditional but where A depends on B only sometimes.

Consider these lines from the attached pyproject.toml

dbt = { version = "0.19.2", markers = "sys_platform != 'darwin'" }
dbt-bigquery = [
    { version = "0.19.2", markers = "sys_platform != 'darwin'" },
    { version = "^1.0.0", markers = "sys_platform == 'darwin' "},
]

The intent here is "Install dbt==0.19.2 and dbt-bigquery==0.19.2 when sys_platform is not 'darwin'; else install dbt-bigquery==^1.0.0. When trying to install on a machine using sys_platform == 'darwin' I get:

Updating dependencies
Resolving dependencies... (12.4s)

  SolverProblemError

  Because analytics-dbt depends on dbt (0.19.2) which depends on dbt-bigquery (0.19.2), dbt-bigquery is required.
  So, because analytics-dbt depends on dbt-bigquery (^1.0.0), version solving failed.

  at /opt/homebrew/Cellar/poetry/1.1.13/libexec/lib/python3.10/site-packages/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

The same error is reached when those pyproject.toml lines are replaced by

dbt = { version = "0.19.2", markers = "sys_platform != 'darwin'" }
dbt-bigquery = { version = "^1.0.0", markers = "sys_platform == 'darwin'" },

By contrast, note that this pyproject.toml:

[tool.poetry]
name = "analytics-dbt"
version = "0.1.0"
description = ""
authors = ["Flex"]

[tool.poetry.dependencies]
python = "^3.7"
dbt = "0.19.2"
dbt-bigquery = "0.19.2"
agate = "1.6.1"
genson = "^1.2.2"

[tool.poetry.dev-dependencies]
ipdb = "^0.13.4"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

resolves.

[abn]

cc: @radoering

[radoering]

This is quite similar to the issue described in #4695 however not the same. The issue here is that a restricted dependency actually is kind of a hidden multiple constraint dependency that probably has to be handled as such (but isn't yet).

To stay with the concrete example a possible solution might be:

If dbt = { version = "0.19.2", markers = "sys_platform != 'darwin'" } would be be treated as a multiple constraint dependency with two overrides

1. not required for darwin
2. 0.19.2 for other platforms

#4695 would enter the game and ignore combinations that do not make sense.

I'll have a look into this if I find some time. Not sure yet how complex such a change will be.

[radoering]

I implemented the concept mentioned in #5506 (comment) and it seems to resolve this issue. Unfortunately, it's not mergeable due to performance issues. Nevertheless, I created a (draft) PR that someone who's interested can take as a starting point or inspiration.

[anthonymichaelclark]

@radoering Thanks for sharing! I'd be keen to dig deeper and see what I can do.

[jorenham]

I have worked around this problem by creating a custom PEP503c-ompliant jax repository that can be used in a pyproject.toml, see: https://github.com/jorenham/jax_pep503

[pdeszynski]

I have a similar issue for markers, for e.g. having 2 packages: package1 and package2 with following pyproject.toml files:

[tool.poetry]
name = "package1"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.10"
torch = [
    { version="^1.13.0", markers = "sys_platform == 'darwin'", python = "<3.11" },
]
package2 = { path = "../package2" }


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
name = "package2"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = "^3.10"
torch = [
    { version=">=1.12.0", markers = "sys_platform == 'darwin'", python = "<3.11" },
    { version="1.12.0", markers = "sys_platform == 'linux'", python = "<3.11" },
]


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

when poetry update is invoked on MacOS machine it will trigger an error:

Because package2 (0.1.0) @ file:///.../package2 depends on torch (1.12.0)
 and package1 depends on torch (^1.13.0), package2 is forbidden.
So, because package1 depends on package2 (0.1.0) @ file:///.../package2, version solving failed.

Here's a log from running it with -vvv flag: https://gist.github.com/pdeszynski/d26f8d224319afaade853e2226c1ee40

Still, when running poetry install in a package2 it will choose newer version of torch:

poetry install                                                                   
Installing dependencies from lock file

Package operations: 2 installs, 0 updates, 0 removals

  • Installing typing-extensions (4.5.0)
  • Installing torch (1.13.1)

poetry --version
Poetry (version 1.4.0)

[Schmetzler]

Just to give some cents to it... I have the same issue, which is somewhat a dealbreaker for my current setup... or can we provide platformspecific pyproject.tomls?

tensorflow = {version = "2.12.0", platform = "linux"}
tensorflow-cpu = [
        {version = "2.10.1", platform = "windows"},
        {version = "2.12.0", platform = "linux"},
]
tensorflow-intel = {version = "2.10.1", platform = "windows"}

Gets the following error:

Because tensorflow-cpu (2.10.1) depends on keras (>=2.10.0,<2.11)
 and tensorflow (2.12.0) depends on keras (>=2.12.0,<2.13), tensorflow-cpu (2.10.1) is incompatible with tensorflow (2.12.0).
So, because signlanguagetranslator depends on both tensorflow (2.12.0) and tensorflow-cpu (2.10.1), version solving failed.

Is there a possibility to make a platform dependent dependency section. E.g. something like:

[tool.poetry.dependencies.platform.windows]
tensorflow-cpu = "2.10.1"
tensorflow-intel = "2.10.1"
...

[tool.poetry.dependencies.platform.linux]
tensorflow = "2.12.0"
tensroflow-cpu = "2.12.0"
...

[trim21]

Just to give some cents to it... I have the same issue, which is somewhat a dealbreaker for my current setup... or can we provide platformspecific pyproject.tomls?

I dont' think platform-specific override is a good solution, consider we alreay have platform=... option.

I think the "right" solution would be supporting soluting to multiple version of same package

[osemenovsky]

Hi! Here from #9573. Is this issue being fixed in the foreseeable future?

[radoering]

Unfortunately, I do not think so. #6969 is the only approach so far.

[osemenovsky]

Understood. Thanks for the feedback.