Handle version solving of python just like any other package

[ucodery]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Darwin 19.6.0

  • not platform-specific

• Poetry version: 1.2.0b2.dev0

  • I believe this affects all versions of poetry

Issue

poetry handles the required python version of a package different than the required version of any python package. Specifically, when there is a ceiling on the python version of a require package's python requirement but not on the requiring package's python requirement, poetry raises a SolverProblemError.

A concrete example of this is when a package, A, depends on python>=3.0 and package B, and package B depends on python^3. This will fail to solve, showing something like The current project's Python requirement (>=3.0) is not compatible with some of the required packages Python requirement: b requires Python >=3.0,<4, so it will not be satisfied for Python >=4. This has let to a lot of user confusion (I've found at least #5591 and #1930, which link to further issues).

I believe that this behavior is confusing to users because they can see that there is some overlap between the two python required versions; most likely that overlap includes the actual python versions currently being use by poetry to solve. This is further confusing because python packages (all other entries in tool.poetry.dependencies except python) do solve when there are multiple requirement ranges for a single package as long as all ranges overlap at at least one real version. They do not have to match completely as python does. A real example of this diverging behavior is show in the test file attached at the end of this report.

The workaround often proposed to this solve error is to restrict the python version your project is compatible. But this just propagates the most pessimistic upper-bound of python to downstream projects. Left alone, this will make ranges such as ^, ~ even more so, a poison that will spread to entire branches of python dependencies. One low-level package using ^3 will force all its users, direct and indirect, to also place this upper-bound on 4 even though there is no practical environment that would actually fail this ceiling.

Low ceilings of python packages are not forced on downstream users, don't force python ceilings.

import pytest
import pytest_cases

from cleo.io.null_io import NullIO
from poetry.core.packages.project_package import ProjectPackage

from poetry.factory import Factory
from poetry.puzzle import Solver
from poetry.puzzle.exceptions import SolverProblemError
from poetry.puzzle.provider import Provider as BaseProvider
from poetry.repositories.installed_repository import InstalledRepository
from poetry.repositories.pool import Pool
from poetry.repositories.repository import Repository
from tests.helpers import get_package


class Provider(BaseProvider):
    def set_package_python_versions(self, python_versions: str) -> None:
        self._package.python_versions = python_versions
        self._python_constraint = self._package.python_constraint


@pytest.fixture()
def io() -> NullIO:
    return NullIO()


@pytest.fixture()
def package() -> ProjectPackage:
    return ProjectPackage("root", "1.0")


@pytest.fixture()
def installed() -> InstalledRepository:
    return InstalledRepository()


@pytest.fixture()
def locked() -> Repository:
    return Repository()


@pytest.fixture()
def repo() -> Repository:
    return Repository()


@pytest.fixture()
def pool(repo: Repository) -> Pool:
    return Pool([repo])


@pytest.fixture()
def solver(
    package: ProjectPackage,
    pool: Pool,
    installed: InstalledRepository,
    locked: Repository,
    io: NullIO,
) -> Solver:
    return Solver(
        package,
        pool,
        installed,
        locked,
        io,
        provider=Provider(package, pool, io, installed=installed),
    )


# There should always be an overlap between the two ranges that includes, at least, 3.5
range1, range2 = pytest_cases.param_fixtures(
    "range1,range2",
    [
        # These ranges solve for both python and some package
        # (same-range, same-range)
        (">=3.4,<4", ">=3.4,<4"),
        # (sub-range, super-range)
        ("^3.4", ">=3.4"),
        ("^3.5", ">=3.4"),
        ("~3.5", ">=3.4"),
        (">=3.5,<4", ">=3.4"),
        ("==3.5", ">=3.4"),
        # All below raise SolverProblemError for python but not some package
        # (super-range, sub-range)
        (">=3.4", "^3.4"),
        (">=3.4", "^3.5"),
        (">=3.4", "~3.5"),
        (">=3.4", ">=3.5,<4"),
        (">=3.4", "==3.5"),
        # (overlapping-range, overlapping-range)
        (">=3.4,<=3.5", ">=3.5,<=3.6"),
    ]
)


def test_python_ranges(
    solver: Solver, repo: Repository, package: ProjectPackage, range1: str, range2: str,
):
    solver.provider.set_package_python_versions(range1)
    package.add_dependency(Factory.create_dependency("A", "*"))

    package_a = get_package("A", "1.0")
    package_a.python_versions = range2

    repo.add_package(package_a)

    solver.solve()
    # which one to use? half will solve, as all do for packages,
    # half will raise an exception
    #with pytest.raises(SolverProblemError):
        #solver.solve()


def test_package_ranges(
    solver: Solver, repo: Repository, package: ProjectPackage, range1: str, range2: str,
):
    package.add_dependency(Factory.create_dependency("A", "*"))
    package.add_dependency(Factory.create_dependency("B", range1))

    package_a = get_package("A", "3.5")
    package_a.add_dependency(Factory.create_dependency("B", range2))
    package_b = get_package("B", "3.5")

    repo.add_package(package_a)
    repo.add_package(package_b)

    solver.solve()

[finswimmer]

Hello @ucodery,

there are mainly two reasons why Poetry treated the python "dependency" this way:

1. The lock file created by Poetry is not only valid for the current activated Python. It is valid for all Python versions the project aims to be compatible with. It seems to be a unique feature in the python ecosystem at the moment and I guess this is why there confusions so often. If the project aims to be compatible with any python version higher than e.g. 3.7 but one or more dependency has defined an upper boundary it is nor longer possible to lock for any python version the project aims to be compatible.

2. Unlike other tools, Poetry makes sure that the dependency definition are valid at the time writing. Whether to respect or ignore an upper boundary for the python version is not a decision made by Poetry. This is a question that must be discussed in the whole Python community. There is some discussion here: https://discuss.python.org/t/requires-python-upper-limits/12663

As it stand now, Poetry will not change the handling of the python requirements made by project. I hope I could give you enough insights to understand why.

fin swimmer

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.