Simplify replication lag selection logic

[sayap]

To avoid surprise, the logic is now purely based on the 3 config values, i.e. low lag / high lag / min tablets, without any outlier calculation or special case.

This gets the logic to match with the unhappy description of "will serve traffic only if there are no fully healthy tablets", shown in the vttablet status page.

[sougou]

@mpawliszyn @rafael @demmer @tirsen
@sayap has deployed this in their production environment and found this approach more stable. I personally found this much easier to understand and reason. So, I asked him to submit this back to vitess.

If you don't have objections, we can review this more carefully and get this submitted.

[rafael]

@brirams was actually just looking at this part of the codebase. He can give good input on this.

I think I'm also on board in making this simpler. One high level comment, if we decide to move forward, we should make this new approach an opt-in behavior via a flag, so it's easier to rollback if we run into regressions.

[brirams]

I agree with the overall sentiment around simplifying this logic and I think this PR does just
that. It makes it a lot clearer what you're controlling when setting the two parameters in question
(discovery_low_replication_lag and discovery_high_replication_lag_minimum_serving) by strictly evicting tablets that are above that high watermark.

However, my concern is that it also changes the existing contract of FilterByReplicationLag by
potentially returning an empty set of tablets when replication is too high. In the old way, you'd
always have something to route queries to, even when replication was too high. This is more a
question for @sougou but is that the type of behavior we want? Whatever the answer, I agree with
@rafael in that we should be able to opt-in to this maybe by providing a flag that switches between
the old and new ways of filtering and maybe consider deprecating the unsupported one.

[sougou]

However, my concern is that it also changes the existing contract of FilterByReplicationLag by
potentially returning an empty set of tablets when replication is too high. In the old way, you'd
always have something to route queries to, even when replication was too high. This is more a
question for @sougou but is that the type of behavior we want?

This is definitely the behavior we want. In fact, at the two-hour replication lag mark, the vttablet has a default setting that would make it non-serving: https://github.com/vitessio/vitess/blob/master/go/vt/vttablet/tabletmanager/healthcheck.go#L219.

@sayap: will you be able to introduce the flag to allow for fallback?

[sayap]

I have added a flag to opt-out from the new simplified logic, for users who rely on the existing (legacy) algorithm.

[rafael]

I think this should initially default to true. Otherwise behavior would change for existent users without them opting in.

We can then go through a process of:

1. Opt in to new behavior.
2. New behavior becomes default.
3. Deprecate legacy behavior.

[sayap]

It is now default to true.

[sougou]

@rafael This looks ready to go. Can you do a final once-over?

[rafael]

LGTM +1