Tablet throttler: support for custom query & threshold

[shlomi-noach]

Description

This PR introduces vttablet command line flags, which allow the user to customize the tablet throttler behavior:

• -throttle_metrics_query (string)
• -throttle_metrics_threshold (float)
• -throttle_check_as_check_self (bool)

The default behavior is for the throttler to check replication lag (where the lag threshold can be customized via the already existing -throttle_threshold).

However, the user may want to run a custom query. They are allowed to run:

• Any SELECT query that returns a single row, single value, float/numeric data type
• A SHOW GLOBAL (STATUS|VARIABLES) LIKE '<variable-name>'

An example use case would be:

vttablet
  -throttle_metrics_query "show global status like 'threads_running'"
  -throttle_metrics_threshold 150
  -throttle_check_as_check_self

The flag throttle_check_as_check_self is desired because the default throttler behavior, for writes, is to consider the overall health of the shard: the default check is for replication lag, and the throttler runs the lag evaluation query on all replicas on the shard. However, if the user wants to measure e.g. Threads_running on the primary only, then the flag throttle_check_as_check_self treats any write-check as "check-self" request, which only reads metrics on the specific tablet.

Checklist

• Should this PR be backported?
• Tests were added or are not required
• Documentation was added or is not required

Impacted Areas in Vitess

Components that this PR will affect:

• Query Serving
• VReplication
• Cluster Management
• Build/CI
• VTAdmin

[shlomi-noach]

Testing TBD via endtoend tests.

[shlomi-noach]

(still draft; will take out of draft mode when tests are satisfied)

[shlomi-noach]

Added support for -throttle_check_as_check_self, updating initial comment.

[shlomi-noach]

Ready for review.

[shlomi-noach]

Kind ping, cc @aquarapid

[shlomi-noach]

ping for review

[deepthi]

Nice work! LGTM

[deepthi]

I'm going to merge this now. If there are other comments, they can be handled in a separate PR.
cc @aquarapid

[aquarapid]

Validated as described; thanks! The only additional feature request I would have is to add the metric threshold to the debugEnvHandler, so that it can be adjusted without a vttablet restart, if needed.

[shlomi-noach]

The only additional feature request I would have is to add the metric threshold to the debugEnvHandler

Creating a separate issue for that