fix hot keepalives

[fspmarshall]

The original transition to control-stream based heartbeats for the multi-resource services would write the full set of resources associated with a given service in a single hot loop. With relative small resource counts this didn't matter much, but some folks have deployments with thousands of resources per service. In these scenarios the large write spikes are likely to induce throttling and/or performance issues, and may interfere with the event systems of some backends.

This PR switches over to using a jittered delay per resource in order to evenly distribute writes (effectively returning us to the old style backend write load pattern before the switch to control-stream based heartbeats, though at a potentially lower rate now that we have variable-rate heartbeats).

changelog: fixed an issue that could cause backend instability when running very large numbers of app/db/kube resources through a single agent.

[espadolini]

Should we bump the tick up from itself rather than from now?

Suggested change

	root.tick = now.Add(h.interval(false /* first */))
	root.tick = root.tick.Add(h.interval(false /* first */))

[fspmarshall]

I kind of like it this way. If perf is good, the difference is inconsequential and we might as well calculate from the variable that isn't behind a pointer. If perf is bad, this adds a bit of slack by always scheduling the N+1th firing a full duration after then Nth firing, even if the Nth firing was delayed in being observed for some reason. For the intended usecase, I think thats a desirable property, and I think its as sensible a behavior as anything else given that the API doesn't try to hide the fact that we're not resetting until Tick is called.

[espadolini]

I don't know if I agree - this is not a goroutine that's only tightly looping on the heartbeats, we might be stuck somewhere else for a bit before coming back to the timer, couldn't we?

If choosing the correct time is inconsequential, why are we bringing over the timestamp from the timer instead of just calling Now()?

[backport-bot-workflows]

@fspmarshall See the table below for backport results.

Branch	Result
branch/v17	Failed