Skip to content

Add script for testing uv against different registries#13615

Merged
jtfmumm merged 7 commits intomainfrom
jtfm/registry-test-script
Jun 18, 2025
Merged

Add script for testing uv against different registries#13615
jtfmumm merged 7 commits intomainfrom
jtfm/registry-test-script

Conversation

@jtfmumm
Copy link
Contributor

@jtfmumm jtfmumm commented May 23, 2025

This PR provides a script that uses environment variables to determine which registries to test. This script is being used to run automated registry tests in CI for AWS, Azure, GCP, Artifactory, GitLab, Cloudsmith, and Gemfury.

You must configure the following required env vars for each registry:

    UV_TEST_<registry_name>_URL            URL for the registry
    UV_TEST_<registry_name>_TOKEN       authentication token
    UV_TEST_<registry_name>_PKG          private package to install

The username defaults to "__token__" but can be optionally set with:

    UV_TEST_<registry_name>_USERNAME

For each configured registry, the test will attempt to install the specified package. Some registries can fall back to PyPI internally, so it's important to choose a package that only exists in the registry you are testing.

Currently, a successful test means that it finds the line “ + <package_name>” in the output. This is because in its current form we don’t know ahead of time what package it is and hence what the exact expected output would be. The advantage if that anyone can run this locally, though they would have to have access to the registries they want to test.

You can also use the --use-op command line argument to derive these test env vars from a 1Password vault (default is "RegistryTests" but can be configured with --op-vault). It will look at all items in the vault with names following the pattern UV_TEST_<registry_name> and will derive the env vars as follows:

    `UV_TEST_<registry_name>_USERNAME` from the `username` field
    `UV_TEST_<registry_name>_TOKEN` from the `password` field
    `UV_TEST_<registry_name>_URL` from a field with the label `url`
    `UV_TEST_<registry_name>_PKG` from a field with the label `pkg`

@jtfmumm jtfmumm added the testing Internal testing of behavior label May 23, 2025
@konstin
Copy link
Member

konstin commented May 23, 2025

Can we define a default list or target that checks that the tests ran against all known registries?

@jtfmumm
Copy link
Contributor Author

jtfmumm commented May 23, 2025

Can we define a default list or target that checks that the tests ran against all known registries?

Yeah, that’s a good idea.

@jtfmumm jtfmumm force-pushed the jtfm/registry-test-script branch from d9d3be5 to dfec62c Compare May 23, 2025 19:02
@jtfmumm
Copy link
Contributor Author

jtfmumm commented May 23, 2025

I've added a cli arg --all that causes the script to check if you've tested known registries. Right now the list is artifactory, azure, aws, cloudsmith, gcp, gemfury, gitlab, and nexus. Even if you don't have credentials for all of them when you run the test, it's helpful as a reminder of other registries to test.

@jtfmumm jtfmumm force-pushed the jtfm/registry-test-script branch from dfec62c to 2ca8424 Compare May 27, 2025 18:15
@jtfmumm jtfmumm force-pushed the jtfm/registry-test-script branch from ca87cc4 to d0978f7 Compare May 27, 2025 18:59
@jtfmumm jtfmumm temporarily deployed to uv-test-publish May 29, 2025 21:33 — with GitHub Actions Inactive
@jtfmumm jtfmumm force-pushed the jtfm/registry-test-script branch from 370c1f8 to a7f21d1 Compare May 30, 2025 21:45
@codspeed-hq
Copy link

codspeed-hq bot commented May 30, 2025

CodSpeed Walltime Performance Report

Merging #13615 will not alter performance

Comparing jtfm/registry-test-script (bcc0f17) with main (49b4501)

Summary

✅ 12 untouched benchmarks

@jtfmumm jtfmumm force-pushed the jtfm/registry-test-script branch 3 times, most recently from 76fc88d to d4d4172 Compare June 5, 2025 13:40
@jtfmumm jtfmumm force-pushed the jtfm/registry-test-script branch 2 times, most recently from 1f98945 to 50ae302 Compare June 9, 2025 21:37
@jtfmumm
Copy link
Contributor Author

jtfmumm commented Jun 9, 2025

I've added support for deriving registry env vars from the 1Password cli tool (op) if you pass in the --use-op flag. These values are derived from a specified vault (default is "RegistryTests" but can be configured with --op-vault). It will look at all items in the vault with names following the pattern UV_TEST_<registry_name> and will derive the env vars as follows:

    `UV_TEST_<registry_name>_USERNAME` from the `username` field
    `UV_TEST_<registry_name>_TOKEN` from the `password` field
    `UV_TEST_<registry_name>_URL` from a field with the label `url`
    `UV_TEST_<registry_name>_PKG` from a field with the label `pkg`

If there is no corresponding field for any of these and an env var has been set in the environment, then that will be used.

@jtfmumm jtfmumm force-pushed the jtfm/registry-test-script branch 2 times, most recently from 93bb57f to b10e7c1 Compare June 9, 2025 21:58
@jtfmumm
Copy link
Contributor Author

jtfmumm commented Jun 11, 2025

I've added a CI integration test that runs the registry test script using GitHub secrets to set the env vars. It is currently running against AWS, Azure, GCP, GitLab, Artifactory, Cloudsmith, and Gemfury.

@jtfmumm jtfmumm temporarily deployed to uv-test-registries June 11, 2025 22:19 — with GitHub Actions Inactive
@jtfmumm jtfmumm temporarily deployed to uv-test-publish June 11, 2025 22:19 — with GitHub Actions Inactive
@jtfmumm jtfmumm temporarily deployed to uv-test-registries June 12, 2025 16:03 — with GitHub Actions Inactive
@jtfmumm jtfmumm force-pushed the jtfm/registry-test-script branch from 3bd558e to 355c819 Compare June 12, 2025 16:07
@jtfmumm jtfmumm temporarily deployed to uv-test-registries June 12, 2025 16:09 — with GitHub Actions Inactive
@jtfmumm jtfmumm temporarily deployed to uv-test-registries June 13, 2025 11:07 — with GitHub Actions Inactive
@jtfmumm jtfmumm temporarily deployed to uv-test-publish June 13, 2025 11:07 — with GitHub Actions Inactive
@jtfmumm jtfmumm force-pushed the jtfm/registry-test-script branch from d19f593 to 9867c18 Compare June 13, 2025 15:17
@jtfmumm jtfmumm temporarily deployed to uv-test-registries June 13, 2025 15:19 — with GitHub Actions Inactive
@jtfmumm jtfmumm temporarily deployed to uv-test-registries June 13, 2025 15:52 — with GitHub Actions Inactive
@jtfmumm jtfmumm merged commit 2fc9221 into main Jun 18, 2025
86 of 87 checks passed
@jtfmumm jtfmumm deleted the jtfm/registry-test-script branch June 18, 2025 07:43
jtfmumm added a commit that referenced this pull request Jun 18, 2025
This is mostly restoring #13215. It also includes a one-line fix for
#13208 (which resulted from that PR). In particular, Azure was returning
303s which were not being correctly handled.

I have also opened another PR (#13754) that refactors and improves the
redirect handling here. It also supersedes the fix here. There are some
tests failing here but they all pass there.

This PR depends on #13615, which adds a script for testing against
registries. The test fails for Azure when running against the restored
#13215 alone and passes with the fix. It also passes for AWS
CodeArtifact, GCP Artifact Registry, JFrog Artifactory, GitLab, and
Gemfury in both cases. I also plan to test against Cloudsmith and Nexus.
jtfmumm added a commit that referenced this pull request Jun 18, 2025
This is mostly restoring #13215. It also includes a one-line fix for
#13208 (which resulted from that PR). In particular, Azure was returning
303s which were not being correctly handled.

I have also opened another PR (#13754) that refactors and improves the
redirect handling here. It also supersedes the fix here. There are some
tests failing here but they all pass there.

This PR depends on #13615, which adds a script for testing against
registries. The test fails for Azure when running against the restored
#13215 alone and passes with the fix. It also passes for AWS
CodeArtifact, GCP Artifact Registry, JFrog Artifactory, GitLab, and
Gemfury in both cases. I also plan to test against Cloudsmith and Nexus.
jtfmumm added a commit that referenced this pull request Jun 18, 2025
This is mostly restoring #13215. It also includes a one-line fix for
#13208 (which resulted from that PR). In particular, Azure was returning
303s which were not being correctly handled.

I have also opened another PR (#13754) that refactors and improves the
redirect handling here. It also supersedes the fix here. There are some
tests failing here but they all pass there.

This PR depends on #13615, which adds a script for testing against
registries. The test fails for Azure when running against the restored
#13215 alone and passes with the fix. It also passes for AWS
CodeArtifact, GCP Artifact Registry, JFrog Artifactory, GitLab, and
Gemfury in both cases. I also plan to test against Cloudsmith and Nexus.
jtfmumm added a commit that referenced this pull request Jun 19, 2025
This is mostly restoring #13215. It also includes a one-line fix for
#13208 (which resulted from that PR). In particular, Azure was returning
303s which were not being correctly handled.

I have also opened another PR (#13754) that refactors and improves the
redirect handling here. It also supersedes the fix here. There are some
tests failing here but they all pass there.

This PR depends on #13615, which adds a script for testing against
registries. The test fails for Azure when running against the restored
#13215 alone and passes with the fix. It also passes for AWS
CodeArtifact, GCP Artifact Registry, JFrog Artifactory, GitLab, and
Gemfury in both cases. I also plan to test against Cloudsmith and Nexus.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Internal testing of behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants