Protect against web/api 5xx responses

[agjohnson]

When a build is attempted, but a web/api instance throws a 5xx response, a number of strange behaviors can result:

• Builds will get stuck in triggered state
• Builds will fail randomly when updating the api fails
• 5xx responses will be returned from the web servers to users

Part of the resolution to this may be defensive protection around intermittent 5xx responses. This may take some operation changes as well though, as our load balancer should really assume a 5xx response is enough to dislodge the server from the lb group.

Raised from #2255

[humitos]

The first one from the list is solved at #3312

I'm not sure to understand the resolution for this. If the request from the builder fails when hitting the API with a 5xx, what we can do?

[ericholscher]

I believe we should be able to have the build HTTP calls do a smart retry backoff. eg:

• Try call
• If it fails, try again
• If that fails, try again in 5s
• If that fails, try again in 10s
• If that fails, raise an exception

[humitos]

If that's the way to go, I suppose we could modify our slumber client API class to do this magic. It would be IO blocking, but I suppose that would be fine.

@agjohnson labeled as Operations so he is maybe thinking in another thing.

[agjohnson]

Yeah, my original take on this is that if a server is overloaded, we should be detecting this and limiting traffic to the server. It's probably a sign that our web servers are overloaded.

Looks like this is still an issue after Azure move: https://sentry.io/read-the-docs/readthedocs-org/issues/712687013/

Retrying is probably a good first place to be defensive. 👍 Requests might even do this natively.