Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

--universal collapses version markers for same version #4899

Closed
lbowenwest opened this issue Jul 8, 2024 · 17 comments
Closed

--universal collapses version markers for same version #4899

lbowenwest opened this issue Jul 8, 2024 · 17 comments
Assignees
@charliermarsh
Labels
bug Something isn't working resolver Related to the package resolver

Comments

@lbowenwest
Copy link

Trying to use uv to install specific versions of torch on a specific platforms with a universal lock file, on macos silicon we want to install 2.3.1 but on x64 systems we want to install 2.3.1+cpu to avoid including the gpu dependencies.

A minimal reproducible example requirements.txt is

--find-links https://download.pytorch.org/whl/torch_stable.html

torch==2.3.1+cpu ; platform_machine == 'x86_64'
torch==2.3.1 ; platform_machine != 'x86_64'

When doing uv pip compile requirements.txt on the appropriate system it finds the correct version, however when running uv pip compile requirements.txt --universal it produces the following line

torch==2.3.1+cpu ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via -r requirements.txt
full verbose output 
$ uv pip compile requirements.txt --universal --verbose


DEBUG uv 0.2.22 (Homebrew 2024-07-07)
DEBUG Starting Python discovery for any Python
DEBUG Looking for exact match for request any Python
DEBUG Searching for Python interpreter in system path
DEBUG Found cpython 3.12.4 at `/Users/lygon/Workspace/rye-test/.direnv/python-3.12/bin/python3` (active virtual environment)
DEBUG Using Python 3.12.4 interpreter at .direnv/python-3.12/bin/python3 for builds
DEBUG Using request timeout of 30s
DEBUG No cache entry for: https://download.pytorch.org/whl/torch_stable.html
DEBUG Found 5724 packages in `--find-links` entry: https://download.pytorch.org/whl/torch_stable.html
DEBUG Solving with installed Python version: 3.12.4
DEBUG Solving with target Python version: >=3.12.4
DEBUG Splitting resolution on root over torch
DEBUG Adding direct dependency: torch{platform_machine != 'x86_64'}==2.3.1+cpu
DEBUG Adding direct dependency: torch{platform_machine == 'x86_64'}==2.3.1+cpu
DEBUG Pre-fork split universal took 0.003s
DEBUG Solving split platform_machine == 'x86_64' (python_version >= '3.12' and python_full_version >= '3.12.4')
DEBUG Found fresh response for: https://pypi.org/simple/torch/
DEBUG Searching for a compatible version of torch{platform_machine == 'x86_64'} (==2.3.1+cpu)
DEBUG Selecting: torch==2.3.1+cpu (torch-2.3.1+cpu-cp310-cp310-linux_x86_64.whl)
DEBUG Adding transitive dependency for torch==2.3.1+cpu: torch==2.3.1+cpu
DEBUG Adding transitive dependency for torch==2.3.1+cpu: torch{platform_machine == 'x86_64'}==2.3.1+cpu
DEBUG Searching for a compatible version of torch{platform_machine == 'x86_64'} (==2.3.1+cpu)
DEBUG Selecting: torch==2.3.1+cpu (torch-2.3.1+cpu-cp310-cp310-linux_x86_64.whl)
DEBUG Found stale response for: https://download.pytorch.org/whl/cpu/torch-2.3.1%2Bcpu-cp310-cp310-linux_x86_64.whl
DEBUG Sending revalidation request for: https://download.pytorch.org/whl/cpu/torch-2.3.1%2Bcpu-cp310-cp310-linux_x86_64.whl
DEBUG Found not-modified response for: https://download.pytorch.org/whl/cpu/torch-2.3.1%2Bcpu-cp310-cp310-linux_x86_64.whl
DEBUG Adding transitive dependency for torch==2.3.1+cpu: filelock*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: typing-extensions>=4.8.0
DEBUG Adding transitive dependency for torch==2.3.1+cpu: sympy*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: networkx*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: jinja2*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: fsspec*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: mkl{platform_system == 'Windows'}>=2021.1.1, <=2021.4.0
DEBUG Searching for a compatible version of torch (==2.3.1+cpu)
DEBUG Selecting: torch==2.3.1+cpu (torch-2.3.1+cpu-cp310-cp310-linux_x86_64.whl)
DEBUG Adding transitive dependency for torch==2.3.1+cpu: filelock*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: typing-extensions>=4.8.0
DEBUG Adding transitive dependency for torch==2.3.1+cpu: sympy*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: networkx*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: jinja2*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: fsspec*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: mkl{platform_system == 'Windows'}>=2021.1.1, <=2021.4.0
DEBUG Found fresh response for: https://pypi.org/simple/filelock/
DEBUG Searching for a compatible version of filelock (*)
DEBUG Selecting: filelock==3.15.4 (filelock-3.15.4-py3-none-any.whl)
DEBUG Found fresh response for: https://pypi.org/simple/jinja2/
DEBUG Found fresh response for: https://pypi.org/simple/typing-extensions/
DEBUG Found fresh response for: https://pypi.org/simple/mkl/
DEBUG Found fresh response for: https://pypi.org/simple/fsspec/
DEBUG Found fresh response for: https://pypi.org/simple/sympy/
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/ae/f0/48285f0262fe47103a4a45972ed2f9b93e4c80b8fd609fa98da78b2a5706/filelock-3.15.4-py3-none-any.whl.metadata
DEBUG Searching for a compatible version of typing-extensions (>=4.8.0)
DEBUG Selecting: typing-extensions==4.12.2 (typing_extensions-4.12.2-py3-none-any.whl)
DEBUG Found fresh response for: https://pypi.org/simple/networkx/
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/26/9f/ad63fc0248c5379346306f8668cda6e2e2e9c95e01216d2b8ffd9ff037d0/typing_extensions-4.12.2-py3-none-any.whl.metadata
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/31/80/3a54838c3fb461f6fec263ebf3a3a41771bd05190238de3486aae8540c36/jinja2-3.1.4-py3-none-any.whl.metadata
DEBUG Searching for a compatible version of sympy (*)
DEBUG Selecting: sympy==1.12.1 (sympy-1.12.1-py3-none-any.whl)
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/61/53/e18c8c97d0b2724d85c9830477e3ebea3acf1dcdc6deb344d5d9c93a9946/sympy-1.12.1-py3-none-any.whl.metadata
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/5e/44/73bea497ac69bafde2ee4269292fa3b41f1198f4bb7bbaaabde30ad29d4a/fsspec-2024.6.1-py3-none-any.whl.metadata
DEBUG Adding transitive dependency for sympy==1.12.1: mpmath>=1.1.0, <1.4.0
DEBUG Searching for a compatible version of networkx (*)
DEBUG Selecting: networkx==3.3 (networkx-3.3-py3-none-any.whl)
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/38/e9/5f72929373e1a0e8d142a130f3f97e6ff920070f87f91c4e13e40e0fba5a/networkx-3.3-py3-none-any.whl.metadata
DEBUG Searching for a compatible version of jinja2 (*)
DEBUG Selecting: jinja2==3.1.4 (jinja2-3.1.4-py3-none-any.whl)
DEBUG Adding transitive dependency for jinja2==3.1.4: markupsafe>=2.0
DEBUG Searching for a compatible version of fsspec (*)
DEBUG Selecting: fsspec==2024.6.1 (fsspec-2024.6.1-py3-none-any.whl)
DEBUG Searching for a compatible version of mkl{platform_system == 'Windows'} (>=2021.1.1, <=2021.4.0)
DEBUG Selecting: mkl==2021.4.0 (mkl-2021.4.0-py2.py3-none-win_amd64.whl)
DEBUG Adding transitive dependency for mkl==2021.4.0: mkl==2021.4.0
DEBUG Adding transitive dependency for mkl==2021.4.0: mkl{platform_system == 'Windows'}==2021.4.0
DEBUG Searching for a compatible version of mkl{platform_system == 'Windows'} (==2021.4.0)
DEBUG Selecting: mkl==2021.4.0 (mkl-2021.4.0-py2.py3-none-win_amd64.whl)
DEBUG Found fresh response for: https://pypi.org/simple/mpmath/
DEBUG Found fresh response for: https://download.pytorch.org/whl/mkl-2021.4.0-py2.py3-none-win_amd64.whl
DEBUG Adding transitive dependency for mkl==2021.4.0: intel-openmp>=2021.dev0, <2022.dev0
DEBUG Adding transitive dependency for mkl==2021.4.0: tbb>=2021.dev0, <2022.dev0
DEBUG Searching for a compatible version of mkl (==2021.4.0)
DEBUG Selecting: mkl==2021.4.0 (mkl-2021.4.0-py2.py3-none-win_amd64.whl)
DEBUG Adding transitive dependency for mkl==2021.4.0: intel-openmp>=2021.dev0, <2022.dev0
DEBUG Adding transitive dependency for mkl==2021.4.0: tbb>=2021.dev0, <2022.dev0
DEBUG Searching for a compatible version of mpmath (>=1.1.0, <1.4.0)
DEBUG Selecting: mpmath==1.3.0 (mpmath-1.3.0-py3-none-any.whl)
DEBUG Found fresh response for: https://download.pytorch.org/whl/mpmath-1.3.0-py3-none-any.whl
DEBUG Found fresh response for: https://pypi.org/simple/tbb/
DEBUG Found fresh response for: https://pypi.org/simple/intel-openmp/
DEBUG Found fresh response for: https://pypi.org/simple/markupsafe/
DEBUG Searching for a compatible version of markupsafe (>=2.0)
DEBUG Selecting: markupsafe==2.1.5 (MarkupSafe-2.1.5-cp310-cp310-macosx_10_9_universal2.whl)
DEBUG Found fresh response for: https://download.pytorch.org/whl/intel_openmp-2021.4.0-py2.py3-none-win_amd64.whl
DEBUG Found fresh response for: https://files.pythonhosted.org/packages/c9/32/9c88009a9feb81f0525bf1c4729a7b39a4523d626a40682c792c6a53e323/tbb-2021.13.0-py2.py3-none-manylinux1_i686.whl.metadata
DEBUG Found fresh response for: https://download.pytorch.org/whl/MarkupSafe-2.1.5-cp310-cp310-macosx_10_9_universal2.whl
DEBUG Searching for a compatible version of intel-openmp (>=2021.dev0, <2022.dev0)
DEBUG Selecting: intel-openmp==2021.4.0 (intel_openmp-2021.4.0-py2.py3-none-win_amd64.whl)
DEBUG Searching for a compatible version of tbb (>=2021.dev0, <2022.dev0)
DEBUG Selecting: tbb==2021.13.0 (tbb-2021.13.0-py2.py3-none-manylinux1_i686.whl)
DEBUG Tried 12 versions: filelock 1, fsspec 1, intel-openmp 1, jinja2 1, markupsafe 1, mkl 1, mpmath 1, networkx 1, sympy 1, tbb 1, torch 1, typing-extensions 1
DEBUG Split platform_machine == 'x86_64' took 0.024s
DEBUG Solving split platform_machine != 'x86_64' (python_version >= '3.12' and python_full_version >= '3.12.4')
DEBUG Searching for a compatible version of torch{platform_machine != 'x86_64'} (==2.3.1+cpu)
DEBUG Selecting: torch==2.3.1+cpu (torch-2.3.1+cpu-cp310-cp310-linux_x86_64.whl)
DEBUG Adding transitive dependency for torch==2.3.1+cpu: torch==2.3.1+cpu
DEBUG Adding transitive dependency for torch==2.3.1+cpu: torch{platform_machine != 'x86_64'}==2.3.1+cpu
DEBUG Searching for a compatible version of torch{platform_machine != 'x86_64'} (==2.3.1+cpu)
DEBUG Selecting: torch==2.3.1+cpu (torch-2.3.1+cpu-cp310-cp310-linux_x86_64.whl)
DEBUG Adding transitive dependency for torch==2.3.1+cpu: filelock*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: typing-extensions>=4.8.0
DEBUG Adding transitive dependency for torch==2.3.1+cpu: sympy*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: networkx*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: jinja2*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: fsspec*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: mkl{platform_system == 'Windows'}>=2021.1.1, <=2021.4.0
DEBUG Searching for a compatible version of torch (==2.3.1+cpu)
DEBUG Selecting: torch==2.3.1+cpu (torch-2.3.1+cpu-cp310-cp310-linux_x86_64.whl)
DEBUG Adding transitive dependency for torch==2.3.1+cpu: filelock*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: typing-extensions>=4.8.0
DEBUG Adding transitive dependency for torch==2.3.1+cpu: sympy*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: networkx*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: jinja2*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: fsspec*
DEBUG Adding transitive dependency for torch==2.3.1+cpu: mkl{platform_system == 'Windows'}>=2021.1.1, <=2021.4.0
DEBUG Searching for a compatible version of filelock (*)
DEBUG Selecting: filelock==3.15.4 (filelock-3.15.4-py3-none-any.whl)
DEBUG Searching for a compatible version of typing-extensions (>=4.8.0)
DEBUG Selecting: typing-extensions==4.12.2 (typing_extensions-4.12.2-py3-none-any.whl)
DEBUG Searching for a compatible version of sympy (*)
DEBUG Selecting: sympy==1.12.1 (sympy-1.12.1-py3-none-any.whl)
DEBUG Adding transitive dependency for sympy==1.12.1: mpmath>=1.1.0, <1.4.0
DEBUG Searching for a compatible version of networkx (*)
DEBUG Selecting: networkx==3.3 (networkx-3.3-py3-none-any.whl)
DEBUG Searching for a compatible version of jinja2 (*)
DEBUG Selecting: jinja2==3.1.4 (jinja2-3.1.4-py3-none-any.whl)
DEBUG Adding transitive dependency for jinja2==3.1.4: markupsafe>=2.0
DEBUG Searching for a compatible version of fsspec (*)
DEBUG Selecting: fsspec==2024.6.1 (fsspec-2024.6.1-py3-none-any.whl)
DEBUG Searching for a compatible version of mkl{platform_system == 'Windows'} (>=2021.1.1, <=2021.4.0)
DEBUG Selecting: mkl==2021.4.0 (mkl-2021.4.0-py2.py3-none-win_amd64.whl)
DEBUG Adding transitive dependency for mkl==2021.4.0: mkl==2021.4.0
DEBUG Adding transitive dependency for mkl==2021.4.0: mkl{platform_system == 'Windows'}==2021.4.0
DEBUG Searching for a compatible version of mkl{platform_system == 'Windows'} (==2021.4.0)
DEBUG Selecting: mkl==2021.4.0 (mkl-2021.4.0-py2.py3-none-win_amd64.whl)
DEBUG Adding transitive dependency for mkl==2021.4.0: intel-openmp>=2021.dev0, <2022.dev0
DEBUG Adding transitive dependency for mkl==2021.4.0: tbb>=2021.dev0, <2022.dev0
DEBUG Searching for a compatible version of mkl (==2021.4.0)
DEBUG Selecting: mkl==2021.4.0 (mkl-2021.4.0-py2.py3-none-win_amd64.whl)
DEBUG Adding transitive dependency for mkl==2021.4.0: intel-openmp>=2021.dev0, <2022.dev0
DEBUG Adding transitive dependency for mkl==2021.4.0: tbb>=2021.dev0, <2022.dev0
DEBUG Searching for a compatible version of mpmath (>=1.1.0, <1.4.0)
DEBUG Selecting: mpmath==1.3.0 (mpmath-1.3.0-py3-none-any.whl)
DEBUG Searching for a compatible version of markupsafe (>=2.0)
DEBUG Selecting: markupsafe==2.1.5 (MarkupSafe-2.1.5-cp310-cp310-macosx_10_9_universal2.whl)
DEBUG Searching for a compatible version of intel-openmp (>=2021.dev0, <2022.dev0)
DEBUG Selecting: intel-openmp==2021.4.0 (intel_openmp-2021.4.0-py2.py3-none-win_amd64.whl)
DEBUG Searching for a compatible version of tbb (>=2021.dev0, <2022.dev0)
DEBUG Selecting: tbb==2021.13.0 (tbb-2021.13.0-py2.py3-none-manylinux1_i686.whl)
DEBUG Tried 24 versions: filelock 2, fsspec 2, intel-openmp 2, jinja2 2, markupsafe 2, mkl 2, mpmath 2, networkx 2, sympy 2, tbb 2, torch 2, typing-extensions 2
DEBUG Split platform_machine != 'x86_64' took 0.000s
Resolved 12 packages in 39ms
# This file was autogenerated by uv via the following command:
#    uv pip compile requirements.txt --universal
filelock==3.15.4 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
fsspec==2024.6.1 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
intel-openmp==2021.4.0 ; platform_system == 'Windows' and (platform_machine == 'x86_64' or platform_machine != 'x86_64')
    # via mkl
jinja2==3.1.4 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
markupsafe==2.1.5 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via jinja2
mkl==2021.4.0 ; platform_system == 'Windows' and (platform_machine == 'x86_64' or platform_machine != 'x86_64')
    # via torch
mpmath==1.3.0 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via sympy
networkx==3.3 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
sympy==1.12.1 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
tbb==2021.13.0 ; platform_system == 'Windows' and (platform_machine == 'x86_64' or platform_machine != 'x86_64')
    # via mkl
torch==2.3.1+cpu ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via -r requirements.txt
typing-extensions==4.12.2 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch

Using a different version such as

--find-links https://download.pytorch.org/whl/torch_stable.html

torch==2.3.1+cpu ; platform_machine == 'x86_64'
torch==2.3.0 ; platform_machine != 'x86_64'

produces the following correct output

# This file was autogenerated by uv via the following command:
#    uv pip compile requirements.txt --universal
filelock==3.15.4 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
fsspec==2024.6.1 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
intel-openmp==2021.4.0 ; (platform_machine == 'x86_64' and platform_system == 'Windows') or (platform_machine != 'x86_64' and platform_system == 'Windows')
    # via mkl
jinja2==3.1.4 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
markupsafe==2.1.5 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via jinja2
mkl==2021.4.0 ; (platform_machine == 'x86_64' and platform_system == 'Windows') or (platform_machine != 'x86_64' and platform_system == 'Windows')
    # via torch
mpmath==1.3.0 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via sympy
networkx==3.3 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
sympy==1.12.1 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
tbb==2021.13.0 ; (platform_machine == 'x86_64' and platform_system == 'Windows') or (platform_machine != 'x86_64' and platform_system == 'Windows')
    # via mkl
torch==2.3.1+cpu ; platform_machine == 'x86_64'
    # via -r requirements.txt
torch==2.3.0 ; platform_machine != 'x86_64'
    # via -r requirements.txt
typing-extensions==4.12.2 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
@charliermarsh
Copy link
Member

Thanks, we might be ignoring local tags somewhere that we shouldn't be.

@charliermarsh charliermarsh added bug Something isn't working resolver Related to the package resolver labels Jul 8, 2024
@caiquejjx
Copy link
Contributor

@charliermarsh hey, can I work on this one?

@charliermarsh
Copy link
Member

@caiquejjx -- Feel free to take a look, I don't have a sense for how hard it is.

@caiquejjx
Copy link
Contributor

@charliermarsh I took a look and was able to find the root cause of the issue and it's in this line
and as the comments suggests:

        // If the specifier is an exact version, and the user requested a local version that's
        // more precise than the specifier, use the local version instead.

Since the first version has a local tag it is overriding the one without it. I don't have an good idea of solution, the one I can think is to store the requirements markers on the Locals object as well and compare then along the versions, if the markers don't match don't override.
Let me know your ideas and if you want to implement yourself or want to wait someone more experienced at uv since I'm new to it.

@charliermarsh
Copy link
Member

Ah yeah. We need to implement handling for locals that’s similar to how we treat URLs. @konstin has context on that. (This is a known issue, but I didn’t realize it would manifest itself this way.)

@konstin
Copy link
Member

konstin commented Jul 8, 2024

Yeah this should be solved by #4580. We implemented the ground work for this for supporting different urls when the markers are disjoint, we can implement this with the same pattern as #4435.

@charliermarsh
Copy link
Member 

torch==2.3.1+cpu ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via -r requirements.txt

Is this... actually wrong? It looks right to me. 2.3.1+cpu is a valid result for torch==2.3.1.

@caiquejjx
Copy link
Contributor

it is valid but it was specified that for platform_machine != 'x86_64' it shouldn't have the +cpu tag which is not being respected here

@charliermarsh
Copy link
Member

Why not, though? 2.3.1+cpu is a valid version to select given a torch==2.3.1 requirement. Local identifiers are ignored when computing equality.

@lbowenwest
Copy link
Author

Huh this is odd, the version specifier docs say this

If the specified version identifier is a public version identifier (no local version label), then the local version label of any candidate versions MUST be ignored when matching versions.

If the specified version identifier is a local version identifier, then the local version labels of candidate versions MUST be considered when matching versions, with the public version identifier being matched as described above, and the local version label being checked for equivalence using a strict string equality comparison.

Which from how I've read it implies that if you ask for 2.3.1 then either 2.3.1+cpu or 2.3.1 is valid, but if you ask for 2.3.1+cpu then 2.3.1 is not valid.

I guess the way around that would be to point to a specific download url?

@judahrand
Copy link

judahrand commented Jul 22, 2024 •
edited
Loading

Why not, though? 2.3.1+cpu is a valid version to select given a torch==2.3.1 requirement. Local identifiers are ignored when computing equality.

2.3.1+cpu doesn't seem to be valid on all platforms though. This is an example on arm64 MacOS:

> uv pip install 'torch==2.3.1+cpu' --find-links https://download.pytorch.org/whl/torch_stable.html
  × No solution found when resolving dependencies:
  ╰─▶ Because torch==2.3.1+cpu has no wheels are available with a matching Python ABI and you require torch==2.3.1+cpu, we can conclude that the requirements are unsatisfiable.

I'm not sure if this means that the above rules are not being respected somewhere or if as @lbowenwest suggests 2.3.1 -> 2.3.1+cpu but not 2.3.1+cpu -> 2.3.1? It would make sense to me that 2.3.1+cpu -> 2.3.1 is not valid because that is effectively a relaxation of the requirement bound rather than a tightening.

@charliermarsh
Copy link
Member

But 2.3.1 is also not valid on all platforms -- for example, in has no available wheels on Windows. It's a problem that's somewhat inherent to what PyTorch is doing here. If it doesn't provide a source distribution, we're basically at the mercy of whatever level of coverage they provide in their wheels. There will always be platforms that aren't covered.

@judahrand
Copy link

judahrand commented Jul 22, 2024 •
edited
Loading

But 2.3.1 is also not valid on all platforms -- for example, in has no available wheels on Windows. It's a problem that's somewhat inherent to what PyTorch is doing here. If it doesn't provide a source distribution, we're basically at the mercy of whatever level of coverage they provide in their wheels. There will always be platforms that aren't covered.

But given that the initial requirements provided were:

--find-links https://download.pytorch.org/whl/torch_stable.html

torch==2.3.1+cpu ; platform_machine == 'x86_64'
torch==2.3.1 ; platform_machine != 'x86_64'

surely the end resolved set of dependencies could reasonably be

# This file was autogenerated by uv via the following command:
#    uv pip compile requirements.txt --universal
filelock==3.15.4 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
fsspec==2024.6.1 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
intel-openmp==2021.4.0 ; (platform_machine == 'x86_64' and platform_system == 'Windows') or (platform_machine != 'x86_64' and platform_system == 'Windows')
    # via mkl
jinja2==3.1.4 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
markupsafe==2.1.5 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via jinja2
mkl==2021.4.0 ; (platform_machine == 'x86_64' and platform_system == 'Windows') or (platform_machine != 'x86_64' and platform_system == 'Windows')
    # via torch
mpmath==1.3.0 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via sympy
networkx==3.3 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
sympy==1.12.1 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
tbb==2021.13.0 ; (platform_machine == 'x86_64' and platform_system == 'Windows') or (platform_machine != 'x86_64' and platform_system == 'Windows')
    # via mkl
torch==2.3.1+cpu ; platform_machine == 'x86_64'
    # via -r requirements.txt
torch==2.3.1 ; platform_machine != 'x86_64'
    # via -r requirements.txt
typing-extensions==4.12.2 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch

@charliermarsh
Copy link
Member

It's reasonable but I don't know that it's more correct than what we output, for two reasons... (1) On platform_machine != 'x86_64', your torch==2.3.1 version is actually unpinned, you could get any +local suffix; and (2) if you did guarantee that you get exactly torch==2.3.1 on platform_machine != 'x86_64', that resolution would also fail at install time, e.g., on ARM Windows.

@charliermarsh
Copy link
Member

Just to be totally clear, I don't think what you're suggesting is unreasonable. I'm just trying to point out that there are tradeoffs here and both solutions will be problematic in different ways, with the root cause being that Torch has scattered platform support (and no source distributions).

@judahrand
Copy link

It's reasonable but I don't know that it's more correct than what we output, for two reasons... (1) On platform_machine != 'x86_64', your torch==2.3.1 version is actually unpinned, you could get any +local suffix; and (2) if you did guarantee that you get exactly torch==2.3.1 on platform_machine != 'x86_64', that resolution would also fail at install time, e.g., on ARM Windows.

Yeah, I'm 100% onboard with the fact that the way torch specifies versions is really unhelpful (I'm not clear why the GPU packages couldn't be specified like torch[cuda] but maybe there are build time reasons!). And based on that changing uv's behaviour to deal with the specific mess caused may not be the be the obvious solution.

Unfortunately, it is also causing me issues so I'm a little biased 😉

@charliermarsh charliermarsh self-assigned this Jul 30, 2024
@BurntSushi
Copy link
Member

Do we have a solid repro for this? I can't seem to reproduce it:

[andrew@duff i4899]$ cat requirements.in
--find-links https://download.pytorch.org/whl/torch_stable.html

torch==2.3.1+cpu ; platform_machine == 'x86_64'
torch==2.3.1 ; platform_machine != 'x86_64'
[andrew@duff i4899]$ uv-0.2.36 pip compile requirements.in --universal
Resolved 13 packages in 34ms
# This file was autogenerated by uv via the following command:
#    uv pip compile requirements.in --universal
filelock==3.15.4
    # via torch
fsspec==2024.6.1
    # via torch
intel-openmp==2021.4.0 ; platform_system == 'Windows'
    # via mkl
jinja2==3.1.4
    # via torch
markupsafe==2.1.5
    # via jinja2
mkl==2021.4.0 ; platform_system == 'Windows'
    # via torch
mpmath==1.3.0
    # via sympy
networkx==3.3
    # via torch
sympy==1.13.2
    # via torch
tbb==2021.13.1 ; platform_system == 'Windows'
    # via mkl
torch==2.3.1 ; platform_machine != 'x86_64'
    # via -r requirements.in
torch==2.3.1+cpu ; platform_machine == 'x86_64'
    # via -r requirements.in
typing-extensions==4.12.2
    # via torch

Maybe this got fixed? So I checked an older version of uv:

[andrew@duff i4899]$ uv-0.2.24 pip compile requirements.in --universal
Resolved 12 packages in 40ms
# This file was autogenerated by uv via the following command:
#    uv pip compile requirements.in --universal
filelock==3.15.4 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
fsspec==2024.6.1 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
intel-openmp==2021.4.0 ; platform_system == 'Windows' and (platform_machine == 'x86_64' or platform_machine != 'x86_64')
    # via mkl
jinja2==3.1.4 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
markupsafe==2.1.5 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via jinja2
mkl==2021.4.0 ; platform_system == 'Windows' and (platform_machine == 'x86_64' or platform_machine != 'x86_64')
    # via torch
mpmath==1.3.0 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via sympy
networkx==3.3 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
sympy==1.13.2 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch
tbb==2021.13.1 ; platform_system == 'Windows' and (platform_machine == 'x86_64' or platform_machine != 'x86_64')
    # via mkl
torch==2.3.1+cpu ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via -r requirements.in
typing-extensions==4.12.2 ; platform_machine == 'x86_64' or platform_machine != 'x86_64'
    # via torch

Which I think matches the errant behavior shown in the OP of this issue. Then uv 0.2.25 did this:

[andrew@duff i4899]$ uv-0.2.25 pip compile requirements.in --universal
Resolved 12 packages in 28ms
# This file was autogenerated by uv via the following command:
#    uv pip compile requirements.in --universal
filelock==3.15.4
    # via torch
fsspec==2024.6.1
    # via torch
intel-openmp==2021.4.0 ; platform_system == 'Windows'
    # via mkl
jinja2==3.1.4
    # via torch
markupsafe==2.1.5
    # via jinja2
mkl==2021.4.0 ; platform_system == 'Windows'
    # via torch
mpmath==1.3.0
    # via sympy
networkx==3.3
    # via torch
sympy==1.13.2
    # via torch
tbb==2021.13.1 ; platform_system == 'Windows'
    # via mkl
torch==2.3.1+cpu
    # via -r requirements.in
typing-extensions==4.12.2
    # via torch

And uv 0.2.26 and all the way up to main appear unchanged. So this might be fixed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@charliermarsh charliermarsh

Labels
bug Something isn't working resolver Related to the package resolver
Projects
No open projects
uv: release-ready
Status: Done
Milestone
No milestone
Development

No branches or pull requests
6 participants
@BurntSushi @charliermarsh @konstin @lbowenwest @judahrand @caiquejjx