[vtadmin] non-blocking resolver

[ajm188]

Description

This PR updates the cluster discovery resolver to make ResolveNow a non-blocking call, by signaling on a channel to watcher goroutine which is continually updating the address list for a cluster component with backoff on failure. It's largely a port of grpc-go's dns-resolver with the "do a dns lookup" replaced with "make a discovery call". (See dns_resolver for reference).

The only notable changes are:

• I added extra backoff strategies, namely "linear" (which does cur + multiplier instead of exponential's cur * multiplier) and "none" (which never backs-off).
• I fixed the backoff for exponential and linear, to prevent jittering from exceeding the configured MaxDelay. This is an oversight in grpc-go's implementation as far as I can tell.
• I added a configuration option to govern the minimum wait time between successful iterations of the watcher loop. In grpc-go this is hard-coded to 30s.
• I corrected some usages of the timer. The DNS resolver leaks open timer channels in certain conditions.

Related Issue(s)

#9977

Checklist

• "Backport me!" label has been added if this change should be backported
• Tests were added or are not required
• Documentation was added or is not required

Deployment Notes

[github-actions]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has the correct release notes label. release notes none should only be used for PRs that are so trivial that they need not be included.
• If a new flag is being introduced, review whether it is really needed. The flag names should be clear and intuitive (as far as possible), and the flag's help should be descriptive.

Bug fixes

• There should be at least one unit or end-to-end test.
• The Pull Request description should either include a link to an issue that describes the bug OR an actual description of the bug and how to reproduce, along with a description of the fix.

Non-trivial changes

• There should be some code comments as to why things are implemented the way they are.

New/Existing features

• Should be documented, either by modifying the existing documentation or creating new documentation.
• New features should have a link to a feature request issue or an RFC that documents the use cases, corner cases and test cases.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• vtctl command output order should be stable and awk-able.

[doeg]

This looks good to me, though I admit I don't really use channels enough to have a good intuition for them.

I deployed this internally and it works great. ✅

[doeg]

A jitterbug?

[ajm188]

wham! that was a good one