Make throttling nmagent fetches for nodesubnet more dynamic

[santhoshmprabhu]

Reason for Change:
We previously added fetch of secondary IPs to the NMAgent client in CNS. This PR makes the fetch asynchronous, with the ability to adjust the frequency dynamically. Specifically,

1. Min and max interval between fetches can be configured
2. Frequency is cut in half (interval doubled) if no diff is seen, up to the max interval.
3. Min interval is used as soon as a diff is seen

This is implemented using a time Ticker, but we wrap it with a wrapper so as to mock it for tests. This PR also includes some additional checks for empty responses from the nmagent (this was raised in this comment). Also have UTs.

Issue Fixed:
NA

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• relevant PR labels added

Notes:

[santhoshmprabhu]

/azp run Azure Container Networking PR

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[santhoshmprabhu]

/azp run Azure Container Networking PR

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[timraymond]

I think I'm going to recant some of what I said in the last PR now that I understand why we're doing this debouncing. As I mentioned during the design review, rate limits and wireserver are a fact of life, so it now feels weird to have arbitrary logic treating the nmagent client carefully because it might blow our rate limit. The design intention of the NMAgent client was always that users can do nmagent.DoSomethingForMe() and it will block while it deals with all of the weirdness that happens with NMAgent and Wireserver. This includes async response handling... and I'm thinking it should also include respecting rate limits. If the request is going to blow the rate limit, the WireserverTransport in the NMAgent client needs to block that request until the request will succeed. Otherwise, we're just setting the user up for failure.

Doing this with tickers is definitely the right way (in particular the whole ticker resetting thing is how I've done respecting rate limits in the past). However, it should be done in https://github.com/Azure/azure-container-networking/blob/master/nmagent/internal/wireserver.go so that it's shared across the client.

[santhoshmprabhu]

/azp run Azure Container Networking PR

[santhoshmprabhu]

/azp run Azure Container Networking PR

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[santhoshmprabhu]

/azp run Azure Container Networking PR

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[santhoshmprabhu]

/azp run Azure Container Networking PR

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[santhoshmprabhu]

/azp run Azure Container Networking PR

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[santhoshmprabhu]

/azp run Azure Container Networking PR

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).