Core doesn't install python version specific dependencies #3347

DamienMatias · 2020-11-10T13:27:53Z

I am on the latest Poetry version.
I have searched the issues of this repo and believe that this is not a duplicate.
If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: MacOS Catalina
Poetry version: 1.1.4
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/DamienMatias/27b7f35d2b392e9f90c0f5b097e035f4#file-gistfile1-txt

Issue

Hello guys, I'm currently working on a package that is meant to be compatible with python ^3.6 and we added dataclasses as a dependency only for python 3.6 which gives us the following pyproject.toml

[tool.poetry]
name = "toto"
version = "0.1.0"
description = ""
authors = ["Damien Matias"]

[tool.poetry.dependencies]
python = "^3.6.1"
dataclasses = {version = "^0.7", python = ">=3.6.1,<3.7"}

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

The issue is if I try to install the package using pip in a python 36 venv, it won't install dataclasses even though it's specified

❯ pip install .
Looking in indexes: https://pypi.org/simple
Processing /private/tmp/toto
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: toto
  Building wheel for toto (PEP 517) ... done
  Created wheel for toto: filename=toto-0.1.0-py3-none-any.whl size=1038 sha256=187322497ce6ccb76aa5c8b658ec322315269284b48540e69f7ed2827e60d1ea
  Stored in directory: /Users/dmatias/Library/Caches/pip/wheels/77/f6/1b/c1cd1dd9e985f0598509bf1e578353c80a4c92bcd09e765e87
Successfully built toto
Installing collected packages: toto
  Attempting uninstall: toto
    Found existing installation: toto 0.1.0
    Uninstalling toto-0.1.0:
      Successfully uninstalled toto-0.1.0
Successfully installed toto-0.1.0

Maybe this is more a https://github.com/python-poetry/poetry-core issue ?

Thank you in advance :)

The text was updated successfully, but these errors were encountered:

jules-ch · 2020-11-10T15:13:36Z

Replace the comma with a space

[tool.poetry]
name = "toto"
version = "0.1.0"
description = ""
authors = ["Damien Matias"]

[tool.poetry.dependencies]
python = "^3.6.1"
dataclasses = {version = "^0.7", python = ">=3.6.1 <3.7"}



[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

$ pip install .
Processing /home/jules/Code/poetry-test
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: dataclasses<0.8,>=0.7; python_version >= "3.6" and python_version < "3.7" in ./.venv/lib/python3.6/site-packages (from toto==0.1.0) (0.7)
Building wheels for collected packages: toto
  Building wheel for toto (PEP 517) ... done
  Created wheel for toto: filename=toto-0.1.0-py3-none-any.whl size=1001 sha256=376a18632f84aeaa4e29996b8d5a9f249e2fa7fef89d1d8f379c2540c81c38c9
  Stored in directory: /home/jules/.cache/pip/wheels/2c/d1/b4/cc1c7773c3ac2413562b07a53e8996e9980d5e85e9297d37a6
Successfully built toto
Installing collected packages: toto
Successfully installed toto-0.1.0

jules-ch · 2020-11-10T15:16:15Z

I agree documentation is not very explicit with what is considered a valid python specification for a package.

https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies

jules-ch · 2020-11-10T15:31:53Z

Seems to be a bug though, parse_constraint in poetry.core is working with ">=3.6.1,<3.7"

DamienMatias · 2020-11-10T15:33:33Z

Thank you for your answer ! I tried with your snipped and I still get the following :

❯ pip install .     
Looking in indexes: https://pypi.org/simple, https://pypi.datalab.ehda.co/simple
Processing /private/tmp/test-dataclasses
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: toto
  Building wheel for toto (PEP 517) ... done
  Created wheel for toto: filename=toto-0.1.0-py3-none-any.whl size=1021 sha256=daa1cbccb45a9b1fa7db18ad19e3e6a3832118fde1269c32aec02c7870de17be
  Stored in directory: /Users/dmatias/Library/Caches/pip/wheels/0c/50/55/503544bf3982bd0723f09d927e804dba4b51cb2501351c5bf9
Successfully built toto
Installing collected packages: toto
  Attempting uninstall: toto
    Found existing installation: toto 0.1.0
    Uninstalling toto-0.1.0:
      Successfully uninstalled toto-0.1.0
Successfully installed toto-0.1.0

I'm using in this venv : Python 3.6.10 and Pip 20.2.4. Here is a gist of

pip install . -vvv

https://gist.github.com/DamienMatias/b1e2d1f4aba0b952ecc40678bee27e24

Do you mind sharing your python and pip version so that I can compare ?

jules-ch · 2020-11-10T15:48:33Z

My bad it seems to be related when specifying patch version of python in the python constraints.

[tool.poetry]
name = "toto"
version = "0.1.0"
description = ""
authors = ["Damien Matias"]

[tool.poetry.dependencies]
python = "^3.6.1"
dataclasses = {version = "^0.7", python = ">=3.6,<3.7"}



[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

This one is working correctly

DamienMatias · 2020-11-10T15:52:06Z

It worked, thank you so much ! :D

DamienMatias · 2020-11-10T15:52:55Z

It works but maybe this is not a normal behavior ?

jules-ch · 2020-11-10T16:00:40Z

It seems to be related on how install_requires works with https://www.python.org/dev/peps/pep-0496/`. It should not include a patch version.

$ python --version
Python 3.6.12
$ python -m pip install "dataclasses ; python_version >= '3.6.1'"
Ignoring dataclasses: markers 'python_version >= "3.6.1"' don't match your environment
$ python -m pip install "dataclasses ; python_version >= '3.6'"
Collecting dataclasses
  Using cached dataclasses-0.7-py3-none-any.whl (18 kB)
Installing collected packages: dataclasses
Successfully installed dataclasses-0.7

I agree this is a weird for the user. It should be simple to add a validator for python_version to catch this before encountering this behavior.

DamienMatias · 2020-11-10T16:03:42Z

It makes sense, thanks again !

sinoroc · 2020-11-10T16:06:55Z

Maybe related: #2480

jules-ch · 2020-11-13T11:35:26Z

IMO @DamienMatias you should reopen this. This is poetry-core related.

When specifying python in your toml. poetry should translate in python_full_version marker.

It seems poetry-core got the machinery to do this in poetry.core.package.dependency:Dependency._create_nested_marker which create python_full_version marker accordingly..

But when using pip install with the PEP517 backend, dependency_markers are created with poetry.core.package.utils.utils:create_nested_marker here:
https://github.com/python-poetry/poetry-core/blob/727907df37016c1451b68b00d3dc2268956c3cc4/poetry/core/factory.py#L279

There seems to be code duplication here.

For a quick fix we can use :

if python_versions:
                    dependency.python_versions = python_versions
                    marker = marker.intersect(
                        parse_marker(
                            dependency._create_nested_marker(
                                "python_version", dependency.python_constraint
                            )
                        )
                    )

But we should probably refactor those two function & just use one.

from poetry.core.factory import Factory
from pathlib import Path


poetry = Factory().create_poetry(Path(".").resolve())
print(poetry.package.requires[0].marker)

Actual : python_version >= "3.6.1" and python_version < "3.7"
Expected : python_full_version >= "3.6.1" and python_version < "3.7"

Any Core contributors can look into this?

finswimmer · 2020-11-15T19:38:44Z

Awesome @jules-ch! I went down the dependency resolution rabbit hole as well the last day, and come to the same problematic lines, but did not figure put how to fix it. Your fix is working, but I would like @sdispater or @abn to have a look on this as well. I think they can better estimate if your solution has unwanted impact.

Thanks a lot!

fin swimmer

There seems to be a bug out where poetry isn't installing certain dependencies with patch python version constraints. python-poetry/poetry#3347

github-actions · 2024-03-02T08:04:44Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

DamienMatias added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Nov 10, 2020

DamienMatias closed this as completed Nov 10, 2020

DamienMatias reopened this Nov 10, 2020

DamienMatias closed this as completed Nov 12, 2020

finswimmer reopened this Nov 15, 2020

finswimmer added the area/solver Related to the dependency resolver label Nov 15, 2020

br3ndonland mentioned this issue Dec 19, 2020

Improve separation between FastAPI and Starlette br3ndonland/inboard#20

Merged

1 task

eugenetriguba added a commit to eugenetriguba/lirc that referenced this issue Apr 18, 2021

Fix poetry python constraint

cd4d980

There seems to be a bug out where poetry isn't installing certain dependencies with patch python version constraints. python-poetry/poetry#3347

sdispater mentioned this issue May 11, 2021

Fix the way python markers with a precision >= 3 are handled python-poetry/poetry-core#178

Merged

2 tasks

abn closed this as completed in python-poetry/poetry-core#178 May 11, 2021

sdispater mentioned this issue May 21, 2021

[1.0] Fix the way python markers with a precision >= 3 are handled python-poetry/poetry-core#180

Merged

abn removed the status/triage This issue needs to be triaged label Mar 3, 2022

dimbleby mentioned this issue May 21, 2022

Strange skipping of dependencies when python requirement is specified ("Not needed for the current environment") #2187

Closed

3 tasks

github-actions bot locked as resolved and limited conversation to collaborators Mar 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Core doesn't install python version specific dependencies #3347

Core doesn't install python version specific dependencies #3347

DamienMatias commented Nov 10, 2020

jules-ch commented Nov 10, 2020

jules-ch commented Nov 10, 2020

jules-ch commented Nov 10, 2020

DamienMatias commented Nov 10, 2020

jules-ch commented Nov 10, 2020 •

edited

Loading

DamienMatias commented Nov 10, 2020 •

edited

Loading

DamienMatias commented Nov 10, 2020

jules-ch commented Nov 10, 2020 •

edited

Loading

DamienMatias commented Nov 10, 2020

sinoroc commented Nov 10, 2020

jules-ch commented Nov 13, 2020

finswimmer commented Nov 15, 2020

github-actions bot commented Mar 2, 2024

Core doesn't install python version specific dependencies #3347

Core doesn't install python version specific dependencies #3347

Comments

DamienMatias commented Nov 10, 2020

Issue

jules-ch commented Nov 10, 2020

jules-ch commented Nov 10, 2020

jules-ch commented Nov 10, 2020

DamienMatias commented Nov 10, 2020

jules-ch commented Nov 10, 2020 • edited Loading

DamienMatias commented Nov 10, 2020 • edited Loading

DamienMatias commented Nov 10, 2020

jules-ch commented Nov 10, 2020 • edited Loading

DamienMatias commented Nov 10, 2020

sinoroc commented Nov 10, 2020

jules-ch commented Nov 13, 2020

finswimmer commented Nov 15, 2020

github-actions bot commented Mar 2, 2024

jules-ch commented Nov 10, 2020 •

edited

Loading

DamienMatias commented Nov 10, 2020 •

edited

Loading

jules-ch commented Nov 10, 2020 •

edited

Loading