-
Notifications
You must be signed in to change notification settings - Fork 562
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement --index-strategy unsafe-best-match
#3138
Implement --index-strategy unsafe-best-match
#3138
Conversation
I don't mind adding this behavior, but we'll need to reach consensus as a team. Thanks for contributing! |
I suppose I'm open to the behavior itself, but I think the current behavior will be quadratic? Since for every version we select, we're now iterating over all versions to create the merged version map. |
In use-cases of my concern, this one is the last missing piece to be able to switch to uv. There's ≈10 internal indexes with pretty much every possible package contained in more than one of those, versions randomly scattered across indexes (so currently the So for a requirement of Installed 3 packages in 5ms
- other-internal-package==0.9.0.post2
+ other-internal-package==0.9.0a1
- six==1.16.0
+ six==1.11.0
- some-internal-package==0.1.22
+ some-internal-package==0.1.20 |
I need to think on how to implement this. |
I'm working on this now. |
--index-strategy unsafe-closest-match
Okay, this seems reasonable to me now. Assuming we consider the number of indexes to be a small constant factor, this should still be linear. |
Adding some other reviewers. |
crates/uv-resolver/src/iterators.rs
Outdated
|
||
Some(next) | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we could make this generic but I kind of gave up. It felt like a lot of work for minimal gain. (It might be nice to have one iterator that can take Version
or Reverse<Version>
, but that again felt like a lot of work to save ~30 lines of code. Open to it though!)
Benchmarks: Jupyter, an easy but very common use case:
Airflow with all extra, a hard case with a bunch of backtracking:
Boto3, the hardest pathological use case we have user hit:
This looks much better than i had expected |
@zanieb @BurntSushi - any feedback on the name of the option, the docs, exposing this? (As opposed to the implementation which seems ok.) |
+1 to exposing, I can review the docs too. For naming, we have...
I think this name makes sense in the context of the others. Although "any match" confuses the naming scheme a bit. |
Yeah this LGTM. Another possible name is |
What about...
|
That's an improvement. Should |
I will make these changes (and add aliases for the existing values), revisit the merge implementation, then ship it. |
c7c5a2b
to
5e9bd0e
Compare
5e9bd0e
to
84cdf26
Compare
84cdf26
to
275f150
Compare
--index-strategy unsafe-closest-match
--index-strategy unsafe-best-match
Thanks @yorickvP ! |
Dropped the `--` prefix from the values of the `--index-strategy` option mistakenly added to documentation in astral-sh#3138.
## Summary Dropped the `--` prefix from the values of the `--index-strategy` option mistakenly added to documentation in #3138. ## Test Plan Verified that actually accepted values of `--index-strategy` don't use a `--` prefix.
Looking closer, the behaviour implemented in the PR is subtly different from the behaviour I intended. I bisected it to the $ git checkout 67b8389aa7579befaa7e6cc64afba58b89d9556b
$ cargo build
$ echo 'tensorrt-llm==0.11.0' | ./target/debug/uv pip compile - --extra-index-url https://pypi.nvidia.com --python-version=3.10 --index-strategy=unsafe-best-match --annotation-style=line
[..]
tensorrt-llm==0.11.0 vs $ git checkout 275f1503721f4a04639279d6f065de86c5075c6e
$ cargo build
$ echo 'tensorrt-llm==0.11.0' | ./target/debug/uv pip compile - --extra-index-url https://pypi.nvidia.com --python-version=3.10 --index-strategy=unsafe-best-match --annotation-style=line
error: Failed to download and build `tensorrt-llm==0.11.0`
Caused by: Failed to build: `tensorrt-llm==0.11.0`
Caused by: Build backend failed to build wheel through `build_wheel()` with exit status: 1
--- stdout:
--- stderr:
Traceback (most recent call last):
File "<string>", line 11, in <module>
File "/home/yorick/.cache/uv/.tmpZbNSch/.venv/lib/python3.11/site-packages/nvidia_stub/buildapi.py", line 29, in build_wheel
return download_wheel(pathlib.Path(wheel_directory), config_settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yorick/.cache/uv/.tmpZbNSch/.venv/lib/python3.11/site-packages/nvidia_stub/wheel.py", line 175, in download_wheel
report_install_failure(distribution, version, None)
File "/home/yorick/.cache/uv/.tmpZbNSch/.venv/lib/python3.11/site-packages/nvidia_stub/error.py", line 63, in report_install_failure
raise InstallFailedError(
nvidia_stub.error.InstallFailedError:
*******************************************************************************
The installation of tensorrt-llm for version 0.11.0 failed.
This is a special placeholder package which downloads a real wheel package
from https://pypi.nvidia.com. If https://pypi.nvidia.com is not reachable, we
cannot download the real wheel file to install. |
Sounds like a problem with breaking ties / stable ordering. Want to open an issue? Interested in working on it? |
) This fixes resolving packages that publish an invalid stub to pypi, such as tensorrt-llm. ## Summary In #3138 , we implemented `unsafe-best-match`. However, it seems to not quite work as expected. When multiple indices contain the same version, it's not clear which index the current code uses. This PR fixes that to use the first index the package is in. ## Test Plan ```console $ echo 'tensorrt-llm==0.11.0' | ./target/debug/uv pip compile - --extra-index-url https://pypi.nvidia.com --python-version=3.10 --index-strategy=unsafe-best-match --annotation-style=line ```
Summary
This index strategy resolves every package to the latest possible version across indexes. If a version is in multiple indexes, the first available index is selected.
Implements #3137
This closely matches pip.
Test Plan
Good question. I'm hesitant to use my certifi example here, since that would inevitably break when torch removes this package. Please comment!