Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enviroment markers ignored when multiple version options coexist with legacy source #6710

Closed
4 tasks done
seansfkelley opened this issue Oct 4, 2022 · 12 comments · Fixed by python-poetry/poetry-core#497 or #6747
Closed
4 tasks done
Labels
area/sources Releated to package sources/indexes/repositories kind/bug Something isn't working as expected status/duplicate Duplicate issues

Comments

@seansfkelley
Copy link

seansfkelley commented Oct 4, 2022

  • I am on the latest stable Poetry version, installed using a recommended method.
  • I have searched the issues of this repo and believe that this is not a duplicate.
  • I have consulted the FAQ and blog for any relevant entries or release notes.
  • If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Possibly related: #3561, #6215, #5506,

Issue

When I attempt poetry install on macOS, Poetry tries to install torch=1.11.0+cpu, which does not have appropriate native builds:

poetry install   
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing torch (1.11.0+cpu): Failed

  RuntimeError

  Unable to find installation candidates for torch (1.11.0+cpu)

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chooser.py:103 in choose_for
       99│ 
      100│             links.append(link)
      101│ 
      102│         if not links:
    → 103│             raise RuntimeError(f"Unable to find installation candidates for {package}")
      104│ 
      105│         # Get the best link
      106│         chosen = max(links, key=lambda link: self._sort_key(package, link))
      107│ 

If I remove the non-Darwin option and leave only this one:

{ version = "1.11.0", markers = "sys_platform == 'darwin'", source = "pypi" }

Poetry installs it without issue.

If I instead replace the non-PyPI option with some other, distinct option and remove the pytorch legacy source:

{ version = "1.12.0", markers = "sys_platform != 'darwin'", source = "pypi" }

(note the version and source are different from the original non-Darwin option)

I can then freely change around the markers on both versions and reinstall and it will always select the intended version according to the rules.

The behavior of apparently ignoring markers only seems to happen when the source is not PyPI.

The non-Darwin source is used in a Docker build, and, if I remove the multiple options and simply set torch = "1.11.0+cpu", I am able to correctly install torch==1.11.0+cpu in my Docker build using this source configuration.

@seansfkelley seansfkelley added kind/bug Something isn't working as expected status/triage This issue needs to be triaged labels Oct 4, 2022
@radoering radoering added area/sources Releated to package sources/indexes/repositories and removed status/triage This issue needs to be triaged labels Oct 5, 2022
@radoering
Copy link
Member

Duplicates #5205 (comment). Will be fixed in upcoming 1.3 via #6679

@neersighted neersighted added the status/duplicate Duplicate issues label Oct 5, 2022
@neersighted neersighted closed this as not planned Won't fix, can't repro, duplicate, stale Oct 5, 2022
@seansfkelley
Copy link
Author

#6679 does not appear to resolve this. This morning I installed Poetry from master:

curl -sSL https://install.python-poetry.org | python3 - --git https://github.com/python-poetry/poetry.git@master

The self-reported version is only 1.3.0.dev0 and doesn't specify the commit hash. I assume this includes the fix, but I don't know how to check which exact commit is running.

Using the same pyproject.toml linked in the original comment:

$ poetry lock -vvv 
Loading configuration file /Users/seankelley/Library/Preferences/pypoetry/config.toml
Adding repository pytorch (https://download.pytorch.org/whl) and setting it as secondary
Creating virtualenv root-2Uq3A66e-py3.10 in /Users/seankelley/Library/Caches/pypoetry/virtualenvs
Using virtualenv: /Users/seankelley/Library/Caches/pypoetry/virtualenvs/root-2Uq3A66e-py3.10
Updating dependencies
Resolving dependencies...
   1: fact: root is 0.1.0
   1: derived: root    
   0: Duplicate dependencies for torch                                                                                                                        
   0: Different requirements found for torch (1.11.0+cpu) with markers sys_platform != "darwin" and torch (1.11.0) with markers sys_platform == "darwin".
   1: Version solving took 0.003 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('root', '0.1.0'): {'torch': <Dependency torch (==1.11.0+cpu)>}}).
   1: fact: root is 0.1.0
   1: derived: root
   1: fact: root depends on torch (1.11.0+cpu)
   1: selecting root (0.1.0)                                                                                                                                  
   1: derived: torch (==1.11.0+cpu)                                                                                                                           
   1: fact: torch (1.11.0+cpu) depends on typing-extensions (*)
   1: selecting torch (1.11.0+cpu)
   1: derived: typing-extensions
<c1>Source (pytorch):</c1> Authorization error accessing https://download.pytorch.org/whl/typing-extensions/
   1: selecting typing-extensions (4.3.0)
   1: Version solving took 0.538 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('root', '0.1.0'): {'torch': <Dependency torch (==1.11.0)>}}).
   1: fact: root is 0.1.0
   1: derived: root
   1: fact: root depends on torch (1.11.0)
   1: selecting root (0.1.0)
   1: derived: torch (==1.11.0)
   1: fact: torch (1.11.0) depends on typing-extensions (*)
   1: selecting torch (1.11.0)                                                                                                                                
   1: derived: typing-extensions
<c1>Source (pytorch):</c1> Authorization error accessing https://download.pytorch.org/whl/typing-extensions/
   1: selecting typing-extensions (4.3.0)
   1: Version solving took 0.136 seconds.
   1: Tried 1 solutions.
   0: Complete version solving took 0.677 seconds with 2 overrides
   0: Resolved with overrides: ({Package('root', '0.1.0'): {'torch': <Dependency torch (==1.11.0+cpu)>}}), ({Package('root', '0.1.0'): {'torch': <Dependency torch (==1.11.0)>}})

Writing lock file

$ poetry install
Installing dependencies from lock file

Package operations: 2 installs, 0 updates, 0 removals

  • Installing typing-extensions (4.3.0)
  • Installing torch (1.11.0+cpu): Failed

  RuntimeError

  Unable to find installation candidates for torch (1.11.0+cpu)

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chooser.py:103 in choose_for
       99│ 
      100│             links.append(link)
      101│ 
      102│         if not links:
    → 103│             raise RuntimeError(f"Unable to find installation candidates for {package}")
      104│ 
      105│         # Get the best link
      106│         chosen = max(links, key=lambda link: self._sort_key(package, link))
      107│ 

The lockfile format has changed a lot and it does seem to keep them separate as indicated in #6679, but the net effect is the same.

poetry.lock
# This file is automatically @generated by Poetry and should not be changed by hand.

[[package]]
name = "torch"
version = "1.11.0"
description = "Tensors and Dynamic neural networks in Python with strong GPU acceleration"
category = "main"
optional = false
python-versions = ">=3.7.0"

[[package.files]]
file = "torch-1.11.0-cp310-cp310-manylinux1_x86_64.whl"
hash = "sha256:62052b50fffc29ca7afc0c04ef8206b6f1ca9d10629cb543077e12967e8d0398"

[[package.files]]
file = "torch-1.11.0-cp310-cp310-manylinux2014_aarch64.whl"
hash = "sha256:866bfba29ac98dec35d893d8e17eaec149d0ac7a53be7baae5c98069897db667"

[[package.files]]
file = "torch-1.11.0-cp310-cp310-win_amd64.whl"
hash = "sha256:951640fb8db308a59d9b510e7d1ad910aff92913323bbe4bc75435347ddd346d"

[[package.files]]
file = "torch-1.11.0-cp310-none-macosx_10_9_x86_64.whl"
hash = "sha256:5d77b5ece78fdafa5c7f42995ff9474399d22571cd6b2de21a5d666306a2ff8c"

[[package.files]]
file = "torch-1.11.0-cp310-none-macosx_11_0_arm64.whl"
hash = "sha256:b5a38682769b544c875ecc34bcb81fbad5c922139b61319aacffcfd8a32f528c"

[[package.files]]
file = "torch-1.11.0-cp37-cp37m-manylinux1_x86_64.whl"
hash = "sha256:f82d77695a60626f2b7382d85bc566de8a6b3e50d32080755abc040db802e419"

[[package.files]]
file = "torch-1.11.0-cp37-cp37m-manylinux2014_aarch64.whl"
hash = "sha256:b96654d42566080a134e784705f33f8536b3b95b5dcde357ed7879b1692a5f78"

[[package.files]]
file = "torch-1.11.0-cp37-cp37m-win_amd64.whl"
hash = "sha256:8ee7c2e8d7f7020d5bfbc1bb91b9591044c26bbd0cee5e4f694cfd7ed8649260"

[[package.files]]
file = "torch-1.11.0-cp37-none-macosx_10_9_x86_64.whl"
hash = "sha256:6860b1d1bf0bb0b67a6bd47f85a0e4c825b518eea13b5d6101999dbbcbd5bc0c"

[[package.files]]
file = "torch-1.11.0-cp38-cp38-manylinux1_x86_64.whl"
hash = "sha256:4322aa29f50da7f404db06cdf30896ea67b09f673af4a985afc7162bc897864d"

[[package.files]]
file = "torch-1.11.0-cp38-cp38-manylinux2014_aarch64.whl"
hash = "sha256:e4d2e0ddd652f30e94cff750220324ec45705d4ecc69658f773b3cb1c7a28dd0"

[[package.files]]
file = "torch-1.11.0-cp38-cp38-win_amd64.whl"
hash = "sha256:34ce5ea4d8d85da32cdbadb50d4585106901e9f8a3527991daa70c13a09de1f7"

[[package.files]]
file = "torch-1.11.0-cp38-none-macosx_10_9_x86_64.whl"
hash = "sha256:0ccc85cd06227a3edf809e2c795fd5762c3d4e8a38b5c9f744c6e7cf841361bb"

[[package.files]]
file = "torch-1.11.0-cp38-none-macosx_11_0_arm64.whl"
hash = "sha256:c1554e49d74f1b2c3e7202d77056ba2dd7465437585bac64062b580f714a44e9"

[[package.files]]
file = "torch-1.11.0-cp39-cp39-manylinux1_x86_64.whl"
hash = "sha256:58c7814502b1c129a650d7092033bbb0bbd64faf1a7941631aaa1aeaddc37570"

[[package.files]]
file = "torch-1.11.0-cp39-cp39-manylinux2014_aarch64.whl"
hash = "sha256:831cf588f01dda9409e75576741d2823453990dee2983d670f2584b37a01adf7"

[[package.files]]
file = "torch-1.11.0-cp39-cp39-win_amd64.whl"
hash = "sha256:44a1d02fd20f827f0f36dc26fdcfc45e793806a6ad52769a22260655a77a4369"

[[package.files]]
file = "torch-1.11.0-cp39-none-macosx_10_9_x86_64.whl"
hash = "sha256:50fd9bf85c578c871c28f1cb0ace9dfc6024401c7f399b174fb0f370899f4454"

[[package.files]]
file = "torch-1.11.0-cp39-none-macosx_11_0_arm64.whl"
hash = "sha256:0e48af66ad755f0f9c5f2664028a414f57c49d6adc37e77e06fe0004da4edb61"

[package.dependencies]
typing-extensions = "*"

[[package]]
name = "torch"
version = "1.11.0+cpu"
description = "Tensors and Dynamic neural networks in Python with strong GPU acceleration"
category = "main"
optional = false
python-versions = ">=3.7.0"

[[package.files]]
file = "torch-1.11.0+cpu-cp310-cp310-linux_x86_64.whl"
hash = "sha256:32fa00d974707c0183bc4dd0c1d69e853d0f15cc60f157b71ac5718847808943"

[[package.files]]
file = "torch-1.11.0+cpu-cp310-cp310-win_amd64.whl"
hash = "sha256:bd984fa8676b2f7c9611b40af3a7c168fb90be3e29028219f822696bb357f472"

[[package.files]]
file = "torch-1.11.0+cpu-cp37-cp37m-linux_x86_64.whl"
hash = "sha256:50008b82004b9d91e036cc199a57f863b6f8978b8a222176f9a4435fce181dd8"

[[package.files]]
file = "torch-1.11.0+cpu-cp37-cp37m-win_amd64.whl"
hash = "sha256:7bbd8b77a59e628a7cb84289a3a26adc7e28dd7213c7f666537f26e714fb1721"

[[package.files]]
file = "torch-1.11.0+cpu-cp38-cp38-linux_x86_64.whl"
hash = "sha256:22997df8f3a3f9faed40ef9e7964d1869cafa0317cc4a5b115bfdf69323e8884"

[[package.files]]
file = "torch-1.11.0+cpu-cp38-cp38-win_amd64.whl"
hash = "sha256:0dbdddc7452a2c42250df369e4968b62589ab0ac1b9d14e27701eb4fc3839ad1"

[[package.files]]
file = "torch-1.11.0+cpu-cp39-cp39-linux_x86_64.whl"
hash = "sha256:544c13ef120531ec2f28a3c858c06e600d514a6dfe09b4dd6fd0262088dd2fa3"

[[package.files]]
file = "torch-1.11.0+cpu-cp39-cp39-win_amd64.whl"
hash = "sha256:7198bf5c69464459bd79526c6a4eaad2806db886443ee2f4e8e7a492bccf03ef"

[package.dependencies]
typing-extensions = "*"

[package.source]
type = "legacy"
url = "https://download.pytorch.org/whl"
reference = "pytorch"

[[package]]
name = "typing-extensions"
version = "4.3.0"
description = "Backported and Experimental Type Hints for Python 3.7+"
category = "main"
optional = false
python-versions = ">=3.7"

[[package.files]]
file = "typing_extensions-4.3.0-py3-none-any.whl"
hash = "sha256:25642c956049920a5aa49edcdd6ab1e06d7e5d467fc00e0506c44ac86fbfca02"

[[package.files]]
file = "typing_extensions-4.3.0.tar.gz"
hash = "sha256:e6d2677a32f47fc7eb2795db1dd15c1f34eff616bcaf2cfb5e997f854fa1c4a6"

[metadata]
lock-version = "2.0"
python-versions = "^3.10"
content-hash = "0adf8e52b6bf98c0eb83b81ef7d74dda816801f3b07f34d126de0c7fb03ba08d"

@dimbleby
Copy link
Contributor

dimbleby commented Oct 5, 2022

1.11.0 and 1.11.0+cpu are too similar: local versions are supposed to be fully compatible with the original.

So at install time when looking in the lockfile for something that satisfies 1.11.0, poetry is happy to choose 1.11.0+cpu: that's supposed to be compatible with 1.11.0, and also more recent.

I don't see anything much to be done about this.

@radoering
Copy link
Member

But shouldn't we consider the locked package source?

@dimbleby
Copy link
Contributor

dimbleby commented Oct 5, 2022

At installation time poetry is presented with the following puzzle: "install torch=1.11.0, the options available to you are per this lockfile".

Considering only version constraints and markers the 1.11.0+cpu package is then the best solution.

@seansfkelley
Copy link
Author

seansfkelley commented Oct 5, 2022

I agree that PyTorch's packaging is a bit odd and the whole cpu alternate build would maybe benefit from being specified using extras instead. That said, the way I understand markers (at least in this case) is that this version specification should be treated as if it doesn't exist for the purposes of installing on macOS.

Edit to respond:

Considering only version constraints and markers the 1.11.0+cpu package is then the best solution.

It doesn't seem to be considering the markers, because +cpu is not a legal choice on my system, right?

@neersighted
Copy link
Member

neersighted commented Oct 5, 2022

The intention was to keep them separate as per #6679 -- they are interchangeable versions, but I should be able to specify two identical package specs from different sources, and get the correct version from the specified source based on my multiple constraint markers.

@dimbleby
Copy link
Contributor

dimbleby commented Oct 5, 2022

@neersighted neersighted reopened this Oct 5, 2022
@dimbleby
Copy link
Contributor

dimbleby commented Oct 5, 2022

maybe as simple as

--- a/src/poetry/puzzle/provider.py
+++ b/src/poetry/puzzle/provider.py
@@ -859,7 +859,7 @@ class Provider:
                 # the complete_name (including features).
                 dependency.name == package.name
                 and (
-                    dependency.source_type is None
+                    (dependency.source_type is None and dependency.source_name is None)
                     or dependency.is_same_source_as(package)
                 )
                 and dependency.constraint.allows(package.version)

??

where the condition being tweaked represents "the dependency doesn't care from what source it is satisfied"

But I'll leave it to someone more motivated to add a unit test proving the point

@dimbleby
Copy link
Contributor

dimbleby commented Oct 5, 2022

actually I think it's all a bigger mess than that

  • what a Dependency calls source_name is stored as source_reference on a Package. Probably the source_name attribute should be removed altogether, in favour of source_reference, so that these became consistent.
  • per the not-great fix above, a Dependency with an explicit source_name is not universally satisfied by any package
  • is_same_source_as() ought to know something about all of this - it currently believes that if both packages have source type None then "no need to check further"
  • using source type None to mean "I don't care" on dependencies but "pypi" on packages is pretty confusing

ugh...

@radoering
Copy link
Member

The other PR is still required for a fix.

Copy link

github-actions bot commented Mar 1, 2024

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area/sources Releated to package sources/indexes/repositories kind/bug Something isn't working as expected status/duplicate Duplicate issues
Projects
None yet
4 participants