Enhance offline testing

[mgorny]

I think uv's getting quite mature at this point. Unfortunately, the test suite is still largely dependent on accessing pypi.org online. FWICS, on top of 65e2f95 there are ~1250 tests skipped with it disabled, i.e. only ~10% of the test suite is run without it, and I suppose a lot of important features aren't tested.

Other points aside, the problem with the current approach to online testing is that it's very fragile to changes to PyPI packages. In the past, I've seen more than once that the tests were already failing within 24 hours from publishing a new version. While many of these failures can be trivially resolved, this effectively that means uv keeps being a rapidly moving target. As such, we can't really mark it stable in Gentoo — our rough guideline for the Python ecosystem is to mark stable after 2 weeks with no major issues, and by then the test suite must still be passing.

Could you please consider a long-term goal of making the tests less dependent on third-party services and more offline? Some Python packages run their own "PyPI server" (i.e. pypiserver PyPI package) for that purpose — I suppose a similar solution would be the cleanest approach to offline testing.

[charliermarsh]

See also #8153.

[mgorny]

That looks kinda similar to what I've been trying to do (externally) in the past (with mitmproxy). That said, probably a well-fit pack of small test packages would still work better.

[zanieb]

Regarding

In the past, I've seen more than once that the tests were already failing within 24 hours from publishing a new version

I consider any test that fails due to changes in PyPI a bug in our test suite — there are probably a couple cases of this left but the intention is for the test suite to be stable. Feel free to report the failures and we will fix them.

Could you please consider a long-term goal of making the tests less dependent on third-party services and more offline?

This is absolutely a goal. It's just a ton of work. Unfortunately, there is not a clear specification for the behavior of Python package indexes and PyPI itself is non-trivial to run locally. I've explored using devpi and pypiserver but both have behavior that differs from PyPI and can't handle the concurrency of our test suite.

I've invested in improving them, but there are still blockers. For some context on devpi, see:

• Add upload-time support to links parsed from a JSON API devpi/devpi#1023
• Support range requests when downloading files devpi/devpi#1021
• Connection failures when downloading wheels devpi/devpi#1022
• Return HTTP 406 instead of HTTP 404 when Accept cannot be fulfilled devpi/devpi#1020
• Broken connections with many concurrent HEAD requests Pylons/waitress#427

See also, some previous work on this topic:

• Add tooling for recording and replaying PyPI interactions #609
• Add PyPI cache to test runs #5726

In the long run, I think we'll need to write our own minimal index. It's hard to find time to work on this, usage of uv is growing and we are prioritizing correctness and functionality for users over internal improvements like this. I'm sorry this makes redistribution of uv harder.

[musicinmybrain]

To piggyback on this, I find that there are about 70 tests in the -p uv --test it target that still try to access the network even when I remove pypi, git, crates-io, and python-managed from the default features of the uv crate.

I’m currently dealing with this in Fedora by patching crates/uv/tests/it/main.rs to remove a number of groups of integration tests. I should probably offer one or more PR’s to more accurately conditionalize these tests.

However, what I’m seeing suggests that disabling these features is not necessarily routinely tested upstream, so tests with “unadvertised” dependencies on various online resources are likely to reappear in the future even if I manage to fix most of them now.