Improvement: Add Backoff for Rate Limits

[s4ke]

Currently it can happen that for whatever reason the requests to the Hetzner Cloud API are going a bit crazy, causing 429 errors.

Currently, once a rate limit is reached, we keep hammering the rate API and we never get out of the rate limit issue.

[s4ke]

To me it is still a bit unclear whether we want to just communicate back the error to the docker engine directly or whether we want to wrap the calling code in https://github.com/cenkalti/backoff to automatically do the backoff.

[s4ke]

Prototype here: master...neuroforgede:docker-volume-hetzner:master

Currently evaluating.

[s4ke]

@costela What are your thoughts on this?

[s4ke]

@costela ping

[costela]

sorry, I'm bit swamped, as always 😓

I understand the motivation, but I'm not sure about the particular solution. Why not something a bit more general like WithBackoffFunc?

[s4ke]

No worries.

If I read the docs for WithBackoffFunc correctly this is used for automatic retrying of requests. We don't want automatic retries though, right? I'd rather have the volume creation fail?

[costela]

ah, you mean the fact that the retries are endless? Yes, that's a bit of a problem. But I'd suggest making that improvement at the hcloud-go side? Adding a way to stop retrying?

That feels like the better way to put this?

[s4ke]

Hmm, this looks new: hetznercloud/hcloud-go#470 but depending on the call patterns this could not be enough.

[s4ke]

Theoretically, this could also be improved on the moby side:

https://github.com/moby/moby/blob/d8b7ae78c0bcaf296609d6b45bc5df9a1d48eea8/pkg/plugins/client.go#L20C2-L20C17