Backoff on reestablishing watches when Unavailable errors are encountered #9840
Conversation
|
@jpbetz has implemented new retry in interceptor layer (still in development branch for testing), but not for this watch routine. We will see if we can incorporate those. But, if we need this be backported for Kubernetes, something simple like this would suffice as a transitional solution. Defer to @jpbetz |
| @@ -849,6 +852,17 @@ func (w *watchGrpcStream) openWatchClient() (ws pb.Watch_WatchClient, err error) | |||
| if isHaltErr(w.ctx, err) { | |||
| return nil, v3rpc.Error(err) | |||
| } | |||
| if isUnavailableErr(w.ctx, err) { | |||
| // retry, but backoff | |||
| if backoff < maxBackoff { | |||
There was a problem hiding this comment.
abstract this out as a retry func with test?
There was a problem hiding this comment.
I'd probably save abstracting it for if we wanted to use this elsewhere... it sounded like there were more systemic retry improvements in progress, so it's possible this will get dropped in the future. I was mostly looking for something minimal we could pick to 3.2.x and 3.3.x streams to alleviate this specific hotloop.
this is a targeted fix to the predominant CPU consumer when an etcd server becomes unavailable to a client with established watches. this picks cleanly back to 3.3.x and 3.2.x
This took our apiserver from 700% cpu when etcd was temporarily down to 10% cpu
xref #9578 #9740