-
Notifications
You must be signed in to change notification settings - Fork 765
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
uv pip install
returning 403 from private pypi cloud instance backed by s3
#2025
Comments
Do you mind updating to v0.1.11? v0.1.6 is a few versions out-of-date. |
Thanks, I just updated to v0.1.11 and now the installer hangs at the same spot for a few seconds before throwing the same 403 error. |
Can you say a bit more about how the auth is intended to work? The URL is publicly available, and redirects you to S3 URLs with credentials embedded? |
Yeah that's correct. It uses s3 one time preauthed urls. I am using pypicloud with redirect_urls enabled |
This also occurs when pip compiling from gemfury links, the package lookup on gemfury works, but downloading packages fails since they are backed by s3. This is the response from S3 when opening one of those preauthed links: Code: SignatureDoesNotMatch |
Hi. I probably don't understand half of the code in this repo, but after experimenting with uv/crates/uv-client/src/registry_client.rs Lines 512 to 515 in 661787b
Here, I believe |
That should be okay though, since the subsequent requests will also go through the auth middleware and get the appropriate headers attached. Were you able to reproduce this issue? What does your setup look like? |
Yep, you're right, @charliermarsh. I just tried hard-coding in my credentials at that point in the code, and then I got a "Request already has an authorization header" error instead. My clue was not a clue after all 😢 But yes, I can reproduce. I believe I have the same issue as @amarckal, which is that when I use curl against pypi.fury.io to fetch my private package, I get a 302 redirect to a url like this:
(I put When I run uv_client::cached_client::read_and_parse_cache file=/private/var/folders/2k/by15c_cs40l9swcck47g6lpm0000gn/T/.tmplRVtUe/wheels-v0/index/5f5b51aad86993d4/sdxp/sdxp-0.7.3-py3-none-any.msgpack
uv_client::cached_client::from_path_sync path="/private/var/folders/2k/by15c_cs40l9swcck47g6lpm0000gn/T/.tmplRVtUe/wheels-v0/index/5f5b51aad86993d4/sdxp/sdxp-0.7.3-py3-none-any.msgpack"
1.665639s 0ms TRACE uv_client::cached_client No cache entry exists for /private/var/folders/2k/by15c_cs40l9swcck47g6lpm0000gn/T/.tmplRVtUe/wheels-v0/index/5f5b51aad86993d4/sdxp/sdxp-0.7.3-py3-none-any.msgpack
1.665827s 1ms DEBUG uv_client::cached_client No cache entry for: https://pypi.fury.io/oda/-/ver_Fz9wq/sdxp-0.7.3-py3-none-any.whl#sha256=869326637eef5de7d4312b82ecf9ba85fcd9e038273d54c0bbe7602d3b8529ad
uv_client::cached_client::fresh_request url="https://pypi.fury.io/oda/-/ver_Fz9wq/sdxp-0.7.3-py3-none-any.whl#sha256=869326637eef5de7d4312b82ecf9ba85fcd9e038273d54c0bbe7602d3b8529ad"
1.666044s 0ms TRACE uv_client::cached_client Sending fresh HEAD request for https://pypi.fury.io/oda/-/ver_Fz9wq/sdxp-0.7.3-py3-none-any.whl#sha256=869326637eef5de7d4312b82ecf9ba85fcd9e038273d54c0bbe7602d3b8529ad
1.666269s 0ms DEBUG uv_auth::middleware Adding authentication to already-seen URL: https://pypi.fury.io/oda/-/ver_Fz9wq/sdxp-0.7.3-py3-none-any.whl#sha256=869326637eef5de7d4312b82ecf9ba85fcd9e038273d54c0bbe7602d3b8529ad
1.895293s 229ms TRACE uv_client::httpcache cached request https://pypi.fury.io/oda/-/ver_Fz9wq/sdxp-0.7.3-py3-none-any.whl#sha256=869326637eef5de7d4312b82ecf9ba85fcd9e038273d54c0bbe7602d3b8529ad is storable because its response has a heuristically cacheable status code 200
uv_client::cached_client::new_cache file=/private/var/folders/2k/by15c_cs40l9swcck47g6lpm0000gn/T/.tmplRVtUe/wheels-v0/index/5f5b51aad86993d4/sdxp/sdxp-0.7.3-py3-none-any.msgpack
uv_client::registry_client::read_metadata_range_request wheel=sdxp-0.7.3-py3-none-any.whl
1.896366s 0ms TRACE uv_client::registry_client Getting metadata for sdxp-0.7.3-py3-none-any.whl by range request
1.897265s DEBUG uv_auth::middleware No credentials found for: https://s3.amazonaws.com/gemfury/gems/<redacted-path>/sdxp_0_7_3_py3_none_any_whl?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<redacted-cred>%2F20240404%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240404T181433Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=<redacted-sig>
error: Failed to download: sdxp==0.7.3
Caused by: Failed to unzip wheel: sdxp-0.7.3-py3-none-any.whl
Caused by: an upstream reader returned an error: io error occurred: Request error: HTTP status client error (403 Forbidden) for url (https://s3.amazonaws.com/gemfury/gems/<redacted-path>/sdxp_0_7_3_py3_none_any_whl?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<redacted-cred>%2F20240404%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240404T181433Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=<redacted-sig>)
Caused by: io error occurred: Request error: HTTP status client error (403 Forbidden) for url (<the-same-url-again>)
Caused by: Request error: HTTP status client error (403 Forbidden) for url (<the-same-url-again>)
Caused by: HTTP status client error (403 Forbidden) for url (<the-same-url-again>) If I isolate just the url in this output, there's another possible clue:
compared with the curl one from above:
The |
Wow thanks for the sleuthing. I have no idea why that would be dropped either. |
Do you know if you did anything special to get Gemfury to run against S3? My Gemfury URLs don't look like that so I've had trouble reproducing. |
I pushed a branch with an extra log if you want to give it a try: #2823 I'm trying to narrow down when that part is dropped from the URL. |
We are experiencing the same issue. I wonder if it's because we have an older Gemfury account. This option is enabled under our organization settings: I don't remember opting into that explicitly. Is that disabled in your account? EDIT: Disabling this option changed the package source to a |
I can try enabling that and then uploading a new package. |
Sadly it's still giving me URLs like |
Without knowing, I am sure that for our Gemfury account, I have at least one package that works fine (it does not redirect to S3) and then at least this one here that fails (because it does redirect to S3). I'm not sure what causes this difference in behavior. The one that fails for me was created by running |
I emailed Gemfury. |
Perhaps I can get setup with one of these S3-back indexes, or they can tell me what I'm doing wrong. |
Ok, I've said "I think I found a clue!" before and been wrong, but I persist: I think I might have found a clue! 😆 I made this test program: use std::error::Error;
use reqwest::redirect::Policy;
use reqwest::Client;
#[tokio::main]
pub async fn main() -> Result<(), Box<dyn Error>> {
let url = "https://pypi.fury.io/oda/-/ver_Fz9wq/sdxp-0.7.3-py3-none-any.whl";
let url: reqwest::Url = url.parse().unwrap();
let client = Client::builder()
// Ensure that we *don't* follow redirects for this example
.redirect(Policy::none())
// fake being curl in case it matters (i don't think so)
.user_agent("curl/8.4.0")
.build()?;
let req = client
.head(url)
.header("authorization", "Basic <redacted>")
.header("accept", "*/*")
.build()?;
println!("111 {:?}", req);
let head_response = client.execute(req).await?;
println!("222 {:?}", head_response);
let location = head_response.headers().get("location").unwrap();
println!("333 {:?}", location);
Ok(())
} And I get this output (with redactions):
So: if I But if I change the curl-line to Next, I change my Rust test program from So finally my clue becomes: The HTTP method is encoded in the signature for the S3 urls, so when you pass that S3 url to Am I right? I need all of your 🧠s to sanity check my logic 😛 |
Other people have had this issue: https://stackoverflow.com/questions/15717230/pre-signing-amazon-s3-urls-for-both-head-and-get-verbs |
I made a proof-of-concept PR to work around the issue. I do that by passing a modified |
I have the same issue with pypicloud, and @torarvid 's PR did not work. ❯ cargo install --git https://github.com/torarvid/uv.git --rev 6d89c85 uv
❯ uv pip compile pyproject.toml -o requirements.txt --index-url http://internal-pypi:8080/simple/
error: Failed to download: internal-pkg==0.1.1054928
Caused by: HTTP status client error (403 Forbidden) for url (https://bucket-name.s3.amazonaws.com/pypi10c6/internal-pkg/internal-pkg-0.1.1054928-py3-none-any.whl
?Signature=%2FhTEgX6psBoSyuCM9F4BiwpCEbw%3D
&Expires=1870939460
&AWSAccessKeyId=FEWNCEFY
&x-amz-security-token=TP//////////ARNU/OkGQV/8AwxoYm) If I manually curl the printed URL, GET works but HEAD gets 403.
|
For pypicloud workaround, |
Interesting, ok, we can add that. Do you want to submit a PR? |
@charliermarsh I re-opened as I do not think that addresses all of the cases here. |
Thanks, sorry, I didn't mean to close this. |
If anyone is willing to test #3460 I would appreciate it. |
I am using a private pypi cloud instance backed by s3 (with no auth on my end). Public packages are resolved normally, but
uv pip
cannot resolve packages hosted on the private cloud instance.pip3 install my-private-package --index-url https://my-pip-instance.example.com/
succeeded with no problems.uv pip install
with the--no-cache
option did not change the result.Relates to #1709 and #1902
Version:
0.1.6
,0.1.11
Verbose output (anonymized)
The text was updated successfully, but these errors were encountered: