502 Bad Gateway #263
Comments
|
Same issue with pyproject-metadata. I'm guessing this is due to |
|
cc @woodruffw @di ^ |
|
@danking that looks like a PyPI issue so we need to wait for the Warehouse maintainers to take a look. Meanwhile, if Henry is right, disabling the attestations might be a workaround for you. |
|
I missed #262 in my notification. Thanks for pointing it out! I'll go ahead and merge it just in case that's it.. |
|
@danking could you try the commit from William's fork? |
|
I don't see a release? https://github.com/pypa/gh-action-pypi-publish/releases |
|
I work with @danking . Just rerun it https://github.com/spiraldb/vortex/actions/runs/10962752610/job/30454969277. I see pypi-attestations~=0.0.12 but the response is an error (though not 502) |
|
Ahh, it was pushed to the release branch. Better error at least! |
@robert3005 @danking reusable workflows are still unsupported: #255 (reply in thread). That's what you're hitting. Follow the issue mentioned there to get notified when it's implemented. |
|
I've already uploaded the pyproject-metadata wheel manually, so I can't test quickly. I can see if there are any I can quickly check. |
@henryiii I currently push a signed tag (and branches updates) from my laptop, and only then I fill out the GH release semi-manually. That's why you're seeing it appearing a bit later.
Thanks! I suppose if there's a bigger problem, we'll get more reports. I also checked that pypi/warehouse#16757 was merged yesterday and since this is the first report we're seeing, my best guess is that this report is not related to that PR. |
|
At least some closure. I can refactor and inline our workflows |
|
Hmm. @webknjaz, I did anticipate that reusable workflows would complicate the situation and actually granted both workflows (the caller and callee) trusted status. Regardless, I'll inline the workflow. |
|
@danking reusable workflows sometimes work for some people in some narrow cases, but that's mostly by accident. Warehouse still needs to implement this properly. IIRC, you need to have both workflows in the same repository and use |
|
Thanks for verifying! |
|
Why does the changelog say "nah, that wasn't it?" It did fix the issue, which was affecting every user with |
|
I was under the impression that it was just a coincidence.. Perhaps I was wrong. I thought so because there were no reports for like a day after the Warehouse change. |
|
Yep, that was indeed the fix! I think the reason no other reports occurred (thankfully) is because most people haven't flipped the |
|
I'm not sure I follow why the difference in |
|
Hmm, you're right -- it should have caused solely a 400-level error, since the change in 502 suggests a transient Fastly or other edge routing failure, possibly? |
|
Yes, that is generally what I would expect to be the cause of a 502. |
|
FWIW, from 2024-09-20 14:33 UTC until around 2024-09-20 21:51 UTC, we ran the GitHub Action to submit to PyPI ten times. Each time it retries five times, each time receiving 502 Bad Gateway. I'm not sure exactly from which attempt the first log came. We likewise assumed it was transient, which is why we retried for a few hours before reporting. Seems like maybe something between us and the PyPI servers misinterpreted the attestation failure as a 502? |
|
Weird. There was a Fastly incident right around then, but it looks like it was in an unrelated service + was already being resolved by the time you began seeing the 502s here: https://www.fastlystatus.com/incident/376938 |
|
I pushed several releases during that time, and the ones with attestations all failed. The ones without all succeeded. Most packages aren't using |
|
Easy way to test: just pin the old one and try it. :) |
|
@woodruffw @di is it possible that Warehouse was responding with a 4xx and Fastly was turning that into a 502 by mistake as Dan suggested? |
|
I don't see any way that that would happen, no. |
|
Pinning 1.10.1 absolutely does still produce a 502: And unpinning fixes it: So I think the changelog should be re-edited to say this indeed fixed it. |
|
@henryiii thank you! fixed. |
|
That's surprising, @woodruffw we should probably figure out why that's happening... |
I suspect it's the base64 change that caused us to do that point release in the first place, although I'm confused as to how it got surfaced as a 502. I'll look into it! |
I looked into it a bit, and I don't have a great working theory for why this is a 502 instead of a 40x. The only thing I can really think of is that the upload endpoint is on the |
I'm not sure if I should report here or on some PyPI specific place, but every time this action is triggered, we get a 502. I am able to publish (the exact same wheels) using twine from my laptop.
This is the whole GitHub Action output with debug logs. I can get you the wheels if that's useful. Are you able to see the GitHub Action page?
The text was updated successfully, but these errors were encountered: