Backport of peering: retry establishing connection more quickly on certain errors into release/1.13.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #13938 to be assessed for backporting due to the inclusion of the label backport/1.13.

The below text is copied from the body of the original PR.

---

When running with an LB in front of the Consul servers, dialers need to retry their requests until they land on a leader server. The default retry backoff was too long because it started at 2s and went exponentially from there: 4s, 8s, etc. This change adds new retry logic for specific errors to retry quickly.

I opted to treat all FailedPrecondition errors as quick-retry errors because they should all be resolved quickly.

Changes:

1. replace retryLoopBackoff with retryLoopBackoffPeering that implements the new retry mechanic
2. change logging everywhere we receive a FailedPrecondition error to not log as an error because they aren't indicative of an actual problem
3. refactor the main HandleStream() loop to add a new channel for errors. Previously, any error received from Recv() would be swallowed and then we'd close the recvCh and use that closing to trigger the loop to exit. Now we're sending that error through the new channel and handling it within the main for{} loop.