Add SHOW VITESS_REPLICATION_STATUS Query Support

[mattlord]

Description

What

The new SHOW VITESS_REPLICATION_STATUS command/query shows the MySQL replica/replication (not vreplication) health for the vitess deployment. It returns a row for each REPLICA and RDONLY tablet in the topology -- with support for filtering by Keyspace/Shard using a LIKE clause -- providing relevant health and status information, including the current throttler status for the tablet.

Note: This is experimental for 12.0 as I would like to have additional flexibility to tweak this based on user feedback over the coming months.

Why

This allows applications -- which are generally interfacing with Vitess through a VTGate -- to alter behavior based on the replication state. They can e.g. decide to back off or delay some planned batch work if replication lag is high or if the tablet throttler is configured and shows that there are too many requests coming in.

This also offers humans and other tooling an easy and convenient way to examine the replication health and related status of the entire Vitess topology, a Keyspace, or a Shard.

Examples

mysql> show vitess_tablets;
+-------+----------+-------+------------+---------+------------------+--------------+----------------------+
| Cell  | Keyspace | Shard | TabletType | State   | Alias            | Hostname     | PrimaryTermStartTime |
+-------+----------+-------+------------+---------+------------------+--------------+----------------------+
| zone1 | commerce | 0     | PRIMARY    | SERVING | zone1-0000000100 | 95336545ea4b | 2021-10-04T16:44:14Z |
| zone1 | commerce | 0     | REPLICA    | SERVING | zone1-0000000101 | 95336545ea4b |                      |
| zone1 | commerce | 0     | RDONLY     | SERVING | zone1-0000000102 | 95336545ea4b |                      |
| zone1 | customer | -80   | PRIMARY    | SERVING | zone1-0000000300 | 95336545ea4b | 2021-10-04T18:03:40Z |
| zone1 | customer | -80   | REPLICA    | SERVING | zone1-0000000301 | 95336545ea4b |                      |
| zone1 | customer | -80   | RDONLY     | SERVING | zone1-0000000302 | 95336545ea4b |                      |
| zone1 | customer | 80-   | PRIMARY    | SERVING | zone1-0000000400 | 95336545ea4b | 2021-10-04T18:03:40Z |
| zone1 | customer | 80-   | REPLICA    | SERVING | zone1-0000000401 | 95336545ea4b |                      |
| zone1 | customer | 80-   | RDONLY     | SERVING | zone1-0000000402 | 95336545ea4b |                      |
+-------+----------+-------+------------+---------+------------------+--------------+----------------------+
9 rows in set (0.00 sec)

mysql> show vitess_replication_status;
+----------+-------+------------+------------------+--------------+--------------------+-------------------------------------------------------------------------+----------------+-----------------------------------------+
| Keyspace | Shard | TabletType | Alias            | Hostname     | ReplicationSource  | ReplicationHealth                                                       | ReplicationLag | ThrottlerStatus                         |
+----------+-------+------------+------------------+--------------+--------------------+-------------------------------------------------------------------------+----------------+-----------------------------------------+
| commerce | 0     | REPLICA    | zone1-0000000101 | 95336545ea4b | 95336545ea4b:17100 | {"EventStreamRunning":"Yes","EventApplierRunning":"Yes","LastError":""} | 0              | {"state":"OK","load":0.00,"message":""} |
| commerce | 0     | RDONLY     | zone1-0000000102 | 95336545ea4b | 95336545ea4b:17100 | {"EventStreamRunning":"Yes","EventApplierRunning":"Yes","LastError":""} | 0              | {"state":"OK","load":0.00,"message":""} |
| customer | -80   | REPLICA    | zone1-0000000301 | 95336545ea4b | 95336545ea4b:17300 | {"EventStreamRunning":"Yes","EventApplierRunning":"Yes","LastError":""} | 0              | {"state":"OK","load":0.00,"message":""} |
| customer | -80   | RDONLY     | zone1-0000000302 | 95336545ea4b | 95336545ea4b:17300 | {"EventStreamRunning":"Yes","EventApplierRunning":"Yes","LastError":""} | 0              | {"state":"OK","load":0.00,"message":""} |
| customer | 80-   | REPLICA    | zone1-0000000401 | 95336545ea4b | 95336545ea4b:17400 | {"EventStreamRunning":"Yes","EventApplierRunning":"Yes","LastError":""} | 0              | {"state":"OK","load":0.00,"message":""} |
| customer | 80-   | RDONLY     | zone1-0000000402 | 95336545ea4b | 95336545ea4b:17400 | {"EventStreamRunning":"Yes","EventApplierRunning":"Yes","LastError":""} | 0              | {"state":"OK","load":0.00,"message":""} |
+----------+-------+------------+------------------+--------------+--------------------+-------------------------------------------------------------------------+----------------+-----------------------------------------+
6 rows in set (0.01 sec)

mysql> show vitess_replication_status like "customer/-80";
+----------+-------+------------+------------------+--------------+--------------------+-------------------------------------------------------------------------+----------------+-----------------------------------------+
| Keyspace | Shard | TabletType | Alias            | Hostname     | ReplicationSource  | ReplicationHealth                                                       | ReplicationLag | ThrottlerStatus                         |
+----------+-------+------------+------------------+--------------+--------------------+-------------------------------------------------------------------------+----------------+-----------------------------------------+
| customer | -80   | REPLICA    | zone1-0000000301 | 95336545ea4b | 95336545ea4b:17300 | {"EventStreamRunning":"Yes","EventApplierRunning":"Yes","LastError":""} | 0              | {"state":"OK","load":0.00,"message":""} |
| customer | -80   | RDONLY     | zone1-0000000302 | 95336545ea4b | 95336545ea4b:17300 | {"EventStreamRunning":"Yes","EventApplierRunning":"Yes","LastError":""} | 0              | {"state":"OK","load":0.00,"message":""} |
+----------+-------+------------+------------------+--------------+--------------------+-------------------------------------------------------------------------+----------------+-----------------------------------------+
2 rows in set (0.01 sec)

mysql> show vitess_replication_status like "commerce%";
+----------+-------+------------+------------------+--------------+--------------------+-------------------------------------------------------------------------+----------------+-----------------------------------------+
| Keyspace | Shard | TabletType | Alias            | Hostname     | ReplicationSource  | ReplicationHealth                                                       | ReplicationLag | ThrottlerStatus                         |
+----------+-------+------------+------------------+--------------+--------------------+-------------------------------------------------------------------------+----------------+-----------------------------------------+
| commerce | 0     | REPLICA    | zone1-0000000101 | 95336545ea4b | 95336545ea4b:17100 | {"EventStreamRunning":"Yes","EventApplierRunning":"Yes","LastError":""} | 0              | {"state":"OK","load":0.00,"message":""} |
| commerce | 0     | RDONLY     | zone1-0000000102 | 95336545ea4b | 95336545ea4b:17100 | {"EventStreamRunning":"Yes","EventApplierRunning":"Yes","LastError":""} | 0              | {"state":"OK","load":0.00,"message":""} |
+----------+-------+------------+------------------+--------------+--------------------+-------------------------------------------------------------------------+----------------+-----------------------------------------+
2 rows in set (0.01 sec)

$ curl -s "localhost:15101/throttler/check?app=vtgate"
{"StatusCode":200,"Value":0,"Threshold":0,"Message":""}

##
# Perform a direct insert on the commerce REPLICA and then perform the same INSERT on the primary via vtgate
##

mysql> insert into commerce.customer (email) values ("mlord@planetscale.com");
Query OK, 1 row affected (0.01 sec)

mysql> show vitess_replication_status like "commerce%"\G
*************************** 1. row ***************************
         Keyspace: commerce
            Shard: 0
       TabletType: REPLICA
            Alias: zone1-0000000101
         Hostname: 95336545ea4b
ReplicationSource: 95336545ea4b:17100
ReplicationHealth: {"EventStreamRunning":"Yes","EventApplierRunning":"No","LastError":"Could not execute Write_rows event on table vt_commerce.customer; Duplicate entry '3' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log vt-0000000100-bin.000001, end_log_pos 9933"}
   ReplicationLag: 15
  ThrottlerStatus: {"state":"OK","load":0.00,"message":""}
*************************** 2. row ***************************
         Keyspace: commerce
            Shard: 0
       TabletType: RDONLY
            Alias: zone1-0000000102
         Hostname: 95336545ea4b
ReplicationSource: 95336545ea4b:17100
ReplicationHealth: {"EventStreamRunning":"Yes","EventApplierRunning":"Yes","LastError":""}
   ReplicationLag: 0
  ThrottlerStatus: {"state":"OK","load":0.00,"message":""}
2 rows in set (0.00 sec)

Signed-off-by: Matt Lord mattalord@gmail.com

Related Issue(s)

#6984

Checklist

• Should this PR be backported?
• Tests were added or are not required
• Documentation was added: Document SHOW vitess_replication_status website#839

Deployment Notes

I'd like to flag this as experimental so that we can get it into 12.0 but have some added flexibility in modifying/improving it based on user feedback.

[sougou]

I think this is a good start. We should eventually add more info, like running state, error, etc. There is some overlap with the experimental VExec command, but I think that one didn't quite take off like we expected.

Tagging @systay @harshit-gangal to look at the parser side.

Optimistically approving.

[deepthi]

This is a nice enhancement. Thank you!

[mattlord]

Bug fix in: #8950