Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions content/en/docs/reference/features/vreplication.md
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,10 @@ in the replication stream from the source. The values can be as follows:
If a failover is performed on the target keyspace/shard, the new master will
automatically resume VReplication from where the previous master left off.

### Throttling

VReplication throttles operation when the source or target appear to be overloaded, indicated by replication lag. See [throttling](../../vreplication/throttling)

## Monitoring and troubleshooting

### VTTablet /debug/status
Expand Down
2 changes: 1 addition & 1 deletion content/en/docs/reference/vreplication/internals.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ weight: 200

### Introduction

The diagram above outlines how a VReplication workflow is performed. VReplication can be asked to start
The diagram below outlines how a VReplication workflow is performed. VReplication can be asked to start
from a specific GTID or from the start. When starting from a GTID the _replication_ mode is used
where it streams events from the binlog.

Expand Down
32 changes: 32 additions & 0 deletions content/en/docs/reference/vreplication/throttling.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
---
title: Throttling
weight: 200
---

### Introduction

VReplication moves potentially massive amounts of data from one place to another, whether within the same keyspace and shard or across keyspaces. It copies data of entire tables and follows up to apply ongoing changes on those tables by reading the binary logs (aka the changelog).

This places load on both the source side (where VReplication reads data from) as well as on target side (where VReplication writes data to).

On the source side, VReplication reads the full content of tables. This typically means loading pages from disk contending for disk IO, and "polluting" the MySQL buffer pool. The operation competes with normal production traffic for both IO and memory resources. If the source is a replica, the operation may lead to replication lag. If the source is a primary, this may lead to write contention.

On the target side, VReplication writes massive amount of data. If the target server is a primary with replicas, then the replicas may incur replication lag.

To address the above issues, VReplication uses the [tablet throttler](../../features/tablet-throttler/) mechanism to push back both reads and writes.

### Target throttling

On the target side, VReplication wishes to consult the overall health of the target shard (there can be multiple shards to a VReplication workflow, and here we discuss the single shard at the end of a single VReplication stream). That shard may serve production traffic unrelated to VReplication. VReplication therefore consults the internal equivalent of `/throttler/check` when writing data to the shard's primary. This checks the replication lag on relevant replicas in the shard. The throttler will push back VReplication writes of both table-copy and changelog events.

### Source throttling

On the source side, VReplication only affects the single MySQL server it reads from, and has no impact on the overall shard. VStreamer, the source endpoint of VReplication, consults the equivalent of `/throttler/check-self`, which looks for replication lag on the source host.

As long as `check-self` fails, VStreamer will not read table data, nor will it pull events from the changelog.

### Impact of throttling

VReplication throttling is designed to give way to normal production traffic while operating in the background. Production traffic will see less contention. The downside is that VReplication can take longer to operate. Under high load in production VReplication may altogether stall, to resume when the load subsides.

Throttling will push back VReplication on replication lag. On systems where replication lag is normally high this can bring VReplication down to a grinding halt. In such systems consider configuring `-throttle_threshold` to a value that agrees with your constraints. The default throttling threshold is at `1` second replication lag.
4 changes: 4 additions & 0 deletions content/en/docs/reference/vreplication/vreplication.md
Original file line number Diff line number Diff line change
Expand Up @@ -408,6 +408,10 @@ in the replication stream from the source. The values can be as follows:
If a failover is performed on the target keyspace/shard, the new master will
automatically resume VReplication from where the previous master left off.

### Throttling

VReplication throttles operation when the source or target appear to be overloaded, indicated by replication lag. See [throttling](../../vreplication/throttling)

## Monitoring and troubleshooting

### VTTablet /debug/status
Expand Down