diff --git a/content/en/docs/reference/features/vreplication.md b/content/en/docs/reference/features/vreplication.md index e295109f0..e61b6dc0a 100644 --- a/content/en/docs/reference/features/vreplication.md +++ b/content/en/docs/reference/features/vreplication.md @@ -398,6 +398,10 @@ in the replication stream from the source. The values can be as follows: If a failover is performed on the target keyspace/shard, the new master will automatically resume VReplication from where the previous master left off. +### Throttling + +VReplication throttles operation when the source or target appear to be overloaded, indicated by replication lag. See [throttling](../../vreplication/throttling) + ## Monitoring and troubleshooting ### VTTablet /debug/status diff --git a/content/en/docs/reference/vreplication/internals.md b/content/en/docs/reference/vreplication/internals.md index 988fc7752..391ec9035 100644 --- a/content/en/docs/reference/vreplication/internals.md +++ b/content/en/docs/reference/vreplication/internals.md @@ -5,7 +5,7 @@ weight: 200 ### Introduction -The diagram above outlines how a VReplication workflow is performed. VReplication can be asked to start +The diagram below outlines how a VReplication workflow is performed. VReplication can be asked to start from a specific GTID or from the start. When starting from a GTID the _replication_ mode is used where it streams events from the binlog. diff --git a/content/en/docs/reference/vreplication/throttling.md b/content/en/docs/reference/vreplication/throttling.md new file mode 100644 index 000000000..a706e0cae --- /dev/null +++ b/content/en/docs/reference/vreplication/throttling.md @@ -0,0 +1,32 @@ +--- +title: Throttling +weight: 200 +--- + +### Introduction + +VReplication moves potentially massive amounts of data from one place to another, whether within the same keyspace and shard or across keyspaces. It copies data of entire tables and follows up to apply ongoing changes on those tables by reading the binary logs (aka the changelog). + +This places load on both the source side (where VReplication reads data from) as well as on target side (where VReplication writes data to). + +On the source side, VReplication reads the full content of tables. This typically means loading pages from disk contending for disk IO, and "polluting" the MySQL buffer pool. The operation competes with normal production traffic for both IO and memory resources. If the source is a replica, the operation may lead to replication lag. If the source is a primary, this may lead to write contention. + +On the target side, VReplication writes massive amount of data. If the target server is a primary with replicas, then the replicas may incur replication lag. + +To address the above issues, VReplication uses the [tablet throttler](../../features/tablet-throttler/) mechanism to push back both reads and writes. + +### Target throttling + +On the target side, VReplication wishes to consult the overall health of the target shard (there can be multiple shards to a VReplication workflow, and here we discuss the single shard at the end of a single VReplication stream). That shard may serve production traffic unrelated to VReplication. VReplication therefore consults the internal equivalent of `/throttler/check` when writing data to the shard's primary. This checks the replication lag on relevant replicas in the shard. The throttler will push back VReplication writes of both table-copy and changelog events. + +### Source throttling + +On the source side, VReplication only affects the single MySQL server it reads from, and has no impact on the overall shard. VStreamer, the source endpoint of VReplication, consults the equivalent of `/throttler/check-self`, which looks for replication lag on the source host. + +As long as `check-self` fails, VStreamer will not read table data, nor will it pull events from the changelog. + +### Impact of throttling + +VReplication throttling is designed to give way to normal production traffic while operating in the background. Production traffic will see less contention. The downside is that VReplication can take longer to operate. Under high load in production VReplication may altogether stall, to resume when the load subsides. + +Throttling will push back VReplication on replication lag. On systems where replication lag is normally high this can bring VReplication down to a grinding halt. In such systems consider configuring `-throttle_threshold` to a value that agrees with your constraints. The default throttling threshold is at `1` second replication lag. diff --git a/content/en/docs/reference/vreplication/vreplication.md b/content/en/docs/reference/vreplication/vreplication.md index 46d8e0b5c..494209939 100644 --- a/content/en/docs/reference/vreplication/vreplication.md +++ b/content/en/docs/reference/vreplication/vreplication.md @@ -408,6 +408,10 @@ in the replication stream from the source. The values can be as follows: If a failover is performed on the target keyspace/shard, the new master will automatically resume VReplication from where the previous master left off. +### Throttling + +VReplication throttles operation when the source or target appear to be overloaded, indicated by replication lag. See [throttling](../../vreplication/throttling) + ## Monitoring and troubleshooting ### VTTablet /debug/status