diff --git a/content/en/docs/13.0/reference/vreplication/v1/_index.md b/content/en/docs/13.0/reference/vreplication/v1/_index.md index 2e05bf309..b86470ee3 100644 --- a/content/en/docs/13.0/reference/vreplication/v1/_index.md +++ b/content/en/docs/13.0/reference/vreplication/v1/_index.md @@ -4,6 +4,6 @@ description: v1 MoveTables and Reshard vtctld commands (**Deprecated**) weight: 1000 --- -{{< info >}} -Starting with Vitess 11.0 you should use the [VReplication v2 commands](../../vreplication) -{{< /info >}} +{{< warning >}} +The v1 commands were deprecated in Vitess 11.0 and have been **removed** in Vitess 16.0. You should begin using the [VReplication v2 commands](../../vreplication) +{{< /warning >}} diff --git a/content/en/docs/14.0/reference/vreplication/v1/_index.md b/content/en/docs/14.0/reference/vreplication/v1/_index.md index 2e05bf309..b86470ee3 100644 --- a/content/en/docs/14.0/reference/vreplication/v1/_index.md +++ b/content/en/docs/14.0/reference/vreplication/v1/_index.md @@ -4,6 +4,6 @@ description: v1 MoveTables and Reshard vtctld commands (**Deprecated**) weight: 1000 --- -{{< info >}} -Starting with Vitess 11.0 you should use the [VReplication v2 commands](../../vreplication) -{{< /info >}} +{{< warning >}} +The v1 commands were deprecated in Vitess 11.0 and have been **removed** in Vitess 16.0. You should begin using the [VReplication v2 commands](../../vreplication) +{{< /warning >}} diff --git a/content/en/docs/15.0/reference/vreplication/v1/_index.md b/content/en/docs/15.0/reference/vreplication/v1/_index.md index 2e05bf309..b86470ee3 100644 --- a/content/en/docs/15.0/reference/vreplication/v1/_index.md +++ b/content/en/docs/15.0/reference/vreplication/v1/_index.md @@ -4,6 +4,6 @@ description: v1 MoveTables and Reshard vtctld commands (**Deprecated**) weight: 1000 --- -{{< info >}} -Starting with Vitess 11.0 you should use the [VReplication v2 commands](../../vreplication) -{{< /info >}} +{{< warning >}} +The v1 commands were deprecated in Vitess 11.0 and have been **removed** in Vitess 16.0. You should begin using the [VReplication v2 commands](../../vreplication) +{{< /warning >}} diff --git a/content/en/docs/16.0/reference/programs/vtctl/_index.md b/content/en/docs/16.0/reference/programs/vtctl/_index.md index 6febda755..521f41ff9 100644 --- a/content/en/docs/16.0/reference/programs/vtctl/_index.md +++ b/content/en/docs/16.0/reference/programs/vtctl/_index.md @@ -74,17 +74,12 @@ Note that wherever `vtctl` commands produced master or MASTER for tablet type, t | [GetKeyspaces](../vtctl/keyspaces#getkeyspaces) | `GetKeyspaces ` | | [RebuildKeyspaceGraph](../vtctl/keyspaces#rebuildkeyspacegraph) | `RebuildKeyspaceGraph -- [--cells=c1,c2,...] ...` | | [ValidateKeyspace](../vtctl/keyspaces#validatekeyspace) | `ValidateKeyspace -- [--ping-tablets] ` | -| [Reshard (v1)](../../vreplication/v1/reshard) | `Reshard -- --v1 [--skip_schema_copy] ` | -| [Reshard (v2)](../../vreplication/reshard) | `Reshard ` | -| [MoveTables (v1)](../../vreplication/v1/movetables) | `MoveTables -- --v1 [--cell=] [--tablet_types=] --workflow= ` | -| [MoveTables (v2)](../../vreplication/movetables) | `MoveTables ` | -| [DropSources](../../vreplication/v1/dropsources) | `DropSources -- [--dry_run] ` | +| [Reshard](../../vreplication/reshard) | `Reshard ` | +| [MoveTables](../../vreplication/movetables) | `MoveTables ` | | [CreateLookupVindex](../vtctl/keyspaces#createLookupvindex) | `CreateLookupVindex -- [--cell=] [--tablet_types=] ` | | [ExternalizeVindex](../vtctl/keyspaces#externalizevindex) | `ExternalizeVindex .` | | [Materialize](../vtctl/keyspaces#materialize) | `Materialize , example : '{"workflow": "aaa", "source_keyspace": "source", "target_keyspace": "target", "table_settings": [{"target_table": "customer", "source_expression": "select * from customer", "create_ddl": "copy"}]}'` | | [VDiff](../vtctl/keyspaces#VDiff) | `VDiff -- [--source_cell=] [--target_cell=] [--tablet_types=] [--filtered_replication_wait_time=30s] [--max_extra_rows_to_compare=1000] ` | -| [SwitchReads](../../vreplication/v1/switchreads) | `SwitchReads -- [--cells=c1,c2,...] [--reverse] -tablet_type={replica\|rdonly} [--dry-run] ` | -| [SwitchWrites](../../vreplication/v1/switchwrites) | `SwitchWrites -- [--filtered_replication_wait_time=30s] [--cancel] [--reverse_replication=false] [--dry-run] ` | | [FindAllShardsInKeyspace](../vtctl/keyspaces#findallshardsinkeyspace) | `FindAllShardsInKeyspace ` | ### Generic diff --git a/content/en/docs/16.0/reference/programs/vtctl/keyspaces.md b/content/en/docs/16.0/reference/programs/vtctl/keyspaces.md index cf9218cc4..1ee37df46 100644 --- a/content/en/docs/16.0/reference/programs/vtctl/keyspaces.md +++ b/content/en/docs/16.0/reference/programs/vtctl/keyspaces.md @@ -156,52 +156,18 @@ Validates that all nodes reachable from the specified keyspace are consistent.<keyspace name> argument is required for the <ValidateKeyspace> command This error occurs if the command is not called with exactly one argument. - -### Reshard (v1) - -```shell -Reshard -- --v1 [--skip_schema_copy] -Start a Resharding process. Example: Reshard -- --cells='zone1,alias1' --tablet_types='primary,replica,rdonly' ks.workflow001 '0' '-80,80-'. -``` -#### Notes - -* The v1 argument is required in Vitess 11.0 and later -* This command is part of the [VReplication v1 workflow](../../../vreplication/v1) -- we recommend that you use the [v2 workflow](../../../vreplication/reshard) in Vitess 11.0 and later - -### Reshard (v2) +### Reshard ```shell Reshard ``` -### MoveTables (v1) - -```shell -MoveTables -- --v1 [--cell=] [--tablet_types=] --workflow= -Move table(s) to another keyspace, table_specs is a list of tables or the tables section of the vschema for the target keyspace. Example: '{"t1":{"column_vindexes": [{"column": "id1", "name": "hash"}]}, "t2":{"column_vindexes": [{"column": "id2", "name": "hash"}]}}'. In the case of an unsharded target keyspace the vschema for each table may be empty. Example: '{"t1":{}, "t2":{}}'. -``` -#### Notes - -* The v1 argument is required in Vitess 11.0 and later -* This command is part of the [VReplication v1 workflow](../../../vreplication/v1) -- we recommend that you use the [v2 workflow](../../../vreplication/movetables) in Vitess 11.0 and later - -## MoveTables (v2) +## MoveTables ```shell MoveTables ``` -### DropSources - -```shell -DropSources -- [--dry_run] [--rename_tables] -After a MoveTables or Resharding workflow cleanup unused artifacts like source tables, source shards and blacklists. -``` - -#### Notes - -* This command is part of the [VReplication v1 workflow](../../../vreplication/v1) -- we recommend that you use the [v2 workflow](../../../vreplication/complete) in Vitess 11.0 and later - ### CreateLookupVindex ```shell @@ -233,28 +199,6 @@ VDiff -- [--source_cell=] [--target_cell=] [--tablet_types= -Switch read traffic for the specified workflow. -``` - -#### Notes - -* This command is part of the [VReplication v1 workflow](../../../vreplication/v1) -- we recommend that you use the [v2 workflow](../../../vreplication/switchtraffic) in Vitess 11.0 and later - -### SwitchWrites - -```shell -SwitchWrites -- [--timeout=30s] [--cancel] [--reverse] [--reverse_replication=false] -tablet_types={replica|rdonly} [--dry-run] -Switch write traffic for the specified workflow. -``` - -#### Notes - -* This command is part of the [VReplication v1 workflow](../../../vreplication/v1) -- we recommend that you use the [v2 workflow](../../../vreplication/switchtraffic) in Vitess 11.0 and later - ### FindAllShardsInKeyspace Displays all of the shards in the specified keyspace. diff --git a/content/en/docs/16.0/reference/vreplication/cancel.md b/content/en/docs/16.0/reference/vreplication/cancel.md deleted file mode 100644 index ca6d51c79..000000000 --- a/content/en/docs/16.0/reference/vreplication/cancel.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -title: --- Cancel -description: Tear down a workflow where no traffic has been switched -weight: 37 -aliases: ['/docs/reference/vreplication/v2/cancel/'] ---- - -{{< info >}} -This documentation is for a new (v2) set of vtctld commands that start in Vitess 11.0. See [RFC](https://github.com/vitessio/vitess/issues/7225) for more details. -{{< /info >}} - -### Command - -``` -MoveTables/Reshard -- [--keep_data] [--keep_routing_rules] Cancel -``` - -### Description - -`Cancel` can be used if a workflow was created in error or was misconfigured and you prefer to create a new workflow instead of fixing this one. Cancel can only be called if no traffic has been switched. It removes vreplication-related artifacts like rows from vreplication and copy_state tables in the side-car `_vt` database and routing rules and blacklisted tables from the topo and, by default, the target tables/shards from the target keyspace. - -### Parameters - -#### --keep_data -**optional**\ -**default** false - -
- -Usually, the target data (tables or shards) are deleted by Cancel. If this flag is used with MoveTables, target tables will not be deleted and, with Reshard, target shards will not be dropped. - -
- -#### --keep_routing_rules -**optional**\ -**default** false - -
- -Usually, any routing rules added in the target keyspace are removed by Cancel. If this flag is used the routing rules in the target keyspace will be left in place. - -
diff --git a/content/en/docs/16.0/reference/vreplication/complete.md b/content/en/docs/16.0/reference/vreplication/complete.md deleted file mode 100644 index 9eec045f0..000000000 --- a/content/en/docs/16.0/reference/vreplication/complete.md +++ /dev/null @@ -1,65 +0,0 @@ ---- -title: --- Complete -description: Tear down a workflow after switching all traffic -weight: 34 -aliases: ['/docs/reference/vreplication/v2/complete/'] ---- - -{{< info >}} -This documentation is for a new (v2) set of vtctld commands that start in Vitess 11.0. See [RFC](https://github.com/vitessio/vitess/issues/7225) for more details. -{{< /info >}} - -### Command - -``` -MoveTables/Reshard -- [--keep_data] [--keep_routing_rules] [--rename_tables] [--dry_run] - Complete -``` - -### Description -**Alert: This is a destructive command** - -`Complete` is used after all traffic has been switched. It removes vreplication-related artifacts like rows from vreplication and copy_state tables in the side-car `_vt` database and routing rules and and blacklisted tables (for MoveTables) from the topo. By default, the source tables (or source shards) are also dropped. - -### Parameters - -#### --keep_data -**optional**\ -**default** false - -
- -Usually, the source data (tables or shards) are deleted by Complete. If this flag is used, for MoveTables, source tables will not be deleted, for Reshard, source shards will not be dropped. - -
- -#### --keep_routing_rules -**optional**\ -**default** false - -
- -Usually, any routing rules added in the source keyspace are removed by Complete. If this flag is used the routing rules in the source keyspace will be left in place. - -
- -#### --rename_tables -**optional**\ -**default** false - -
- -The rename_tables flag is applicable only for MoveTables. Tables are renamed instead of being deleted. Currently the new name is _<table_name>_old. - -We use the same renaming logic used by `pt-online-schema-change`. Such tables are automatically skipped by vreplication if they exist on the source. - -
- -#### --dry-run -**optional**\ -**default** false - -
-You can do a dry run where no actual action is taken but the command logs all the actions that would be taken -by Complete. -
diff --git a/content/en/docs/16.0/reference/vreplication/create.md b/content/en/docs/16.0/reference/vreplication/create.md deleted file mode 100644 index cdbff0e87..000000000 --- a/content/en/docs/16.0/reference/vreplication/create.md +++ /dev/null @@ -1,183 +0,0 @@ ---- -title: "--- Create" -description: Initiate a workflow -weight: 31 -aliases: ['/docs/reference/vreplication/v2/create/'] ---- - -{{< info >}} -This documentation is for a new (v2) set of vtctld commands that start in Vitess 11.0. See [RFC](https://github.com/vitessio/vitess/issues/7225) for more details. -{{< /info >}} - -### Command - -``` -MoveTables -- [--source=] [--tables=] [--cells=] - [--tablet_types=] [--all] [--exclude=] - [--auto_start] [--stop_after_copy] - Create - -Reshard -- [--source_shards=] [--target_shards=] - [--cells=] [--tablet_types=] [--skip_schema_copy] - Create - -``` - -### Description - -`MoveTables/Reshard Create` sets up and creates a new workflow. The workflow name should not conflict with that of an existing workflow. - -### Parameters - -#### --source -**mandatory** - -**MoveTables only** -
-Name of existing keyspace (the source keyspace) that contains the tables to be moved. -
- -#### table_specs -**mandatory** Either `table_specs` or `--all` needs to be specified - -**MoveTables only** -
-_Either_ - -* a comma-separated list of tables - * if target keyspace is unsharded OR - * if target keyspace is sharded AND the tables being moved are already defined in the target's vschema - - Example: `MoveTables -- --source=commerce --tables=customer,corder Create customer.commerce2customer` - -_Or_ - -* the JSON table section of the vschema for associated tables - * if target keyspace is sharded AND - * tables being moved are not yet present in the target's vschema - - Example: `MoveTables -- --source=commerce --tables='{"t1":{"column_vindexes": [{"column": "id", "name": "hash"}]}}}' Create customer.commerce2customer` - -
- -#### --cells -**optional**\ -**default** local cell - -
-A comma-separated list of cell names or cell aliases. This list is used by VReplication to determine which -cells should be used to pick a tablet for selecting data from the source keyspace.

- -###### Uses - -* Improve performance by picking a tablet in cells in network proximity with the target -* Reduce bandwidth costs by skipping cells that are in different availability zones -* Select cells where replica lags are lower -
- -#### --tablet_types -**optional**\ -**default** replica - -
-A comma-separated list of tablet types that are used while picking a tablet for sourcing data. One or more from PRIMARY, REPLICA, RDONLY.

- -###### Uses - -* To reduce the load on PRIMARY tablets by using REPLICAs or RDONLYs -* Reducing lag by pointing to PRIMARY -
- - -#### --auto_start - -**optional** -**default** true - -
- -Normally the workflow starts immediately after it is created. If this flag is set -to false then the workflow is in a Stopped state until you explicitly start it. - -
- -###### Uses - -* Allows updating the rows in `_vt.vreplication` after MoveTables has setup the -streams. For example, you can add some filters to specific tables or change the -projection clause to modify the values on the target. This -provides an easier way to create simpler Materialize workflows by first using -MoveTables with auto_start false, updating the BinlogSource as required by your -Materialize and then starting the workflow. -* Changing the `copy_state` and/or `pos` values to restart a broken MoveTables workflow -from a specific point of time. - -#### --stop_after_copy - -**optional** -**default** false - -
- -If set, the workflow will stop once the Copy phase has been completed i.e. once -all tables have been copied and VReplication decides that the lag -is small enough to start replicating, the workflow state will be set to Stopped. - -###### Uses -* If you just want a consistent snapshot of all the tables you can set this flag. The workflow -will stop once the copy is done and you can then mark the workflow as `Complete`d - -
- -#### --all -**optional** cannot specify `table_specs` if `--all` is specified - -**MoveTables only** -
- -Move all tables from the source keyspace. - -
- -#### --exclude -**optional** only applies if `--all` is specified - -**MoveTables only** -
- -If moving all tables, specifies tables to be skipped. - -
- - -#### source_shards -**mandatory** - -**Reshard only** - -
-Comma-separated shard names to reshard from. - -Example: `Reshard -- --source_shards=0 --target_shards=-80,80- Create customer.reshard1to2` - -
- -#### target_shards -**mandatory** - -**Reshard only** - -
-Comma-separated shard names to reshard to. -
- -#### --skip_schema_copy -**optional**\ -**default** false - -**Reshard only** - -
-If true the source schema is copied to the target shards. If false, you need to create the tables -before calling `Reshard`. -
diff --git a/content/en/docs/16.0/reference/vreplication/faq.md b/content/en/docs/16.0/reference/vreplication/faq.md index c1d100e8c..d41262d00 100644 --- a/content/en/docs/16.0/reference/vreplication/faq.md +++ b/content/en/docs/16.0/reference/vreplication/faq.md @@ -16,13 +16,14 @@ GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, RELOAD, PROCESS, FILE, {{< expand `Why am I seeing io.EOF errors in my workflow?`>}}

- io.EOF errors can be difficult to track down. These are usually caused by an issue at the mysql server. Here are some possible reasons: + io.EOF errors can be difficult to track down. These are usually caused by an issue at the mysql server layer. You will need to consult + the source and target vttablet logs in order to know for sure in each case. Here are some possible reasons:

  • GTID is not enabled on the server. VReplication requires GTID=on (permissible is not supported)
    • -
  • Permissions are not setup correctly for the vreplication mysql user
    • +
  • Permissions are not setup correctly for the vreplication related mysql users (in particular the `vt_filtered` user by defualt).
  • Row-based replication (RBR) binlog_format=row is not enabled. Statement-based replication (SBR) is not supported by VReplication
  • The mysql server is down or not reachable
@@ -36,8 +37,8 @@ binlog_row_image=full {{< /expand >}} - diff --git a/content/en/docs/16.0/reference/vreplication/flags.md b/content/en/docs/16.0/reference/vreplication/flags.md index 4260fb6dc..d0423da7d 100644 --- a/content/en/docs/16.0/reference/vreplication/flags.md +++ b/content/en/docs/16.0/reference/vreplication/flags.md @@ -93,18 +93,18 @@ When copying the contents of a table we go through 1+ cycles of copy,catchup,fas **Maximum** 60 (one minute)\ **Applicable on** target -For an idle source shard, the source vstreamer sends a heartbeat. Currently, that is once per second. On receiving the heartbeat the target VReplication module updates the time_updated column of the relevant row of `\_vt.vreplication`. For some setups this is a problem, for example: +For an idle source shard, the source vstreamer sends a heartbeat. Currently, that is once per second. On receiving the heartbeat the target VReplication module updates the time_updated column of the relevant row of `_vt.vreplication`. For some setups this is a problem, for example: -* if there are too many streams the extra write QPS or CPU load due to these updates are unacceptable -* if there are too many streams and/or a large source field (lot of participating tables) which generates unacceptable increase in the binlog size -* even for a single stream, if the server is of a lower configuration, then the resulting increase in the QPS or binlog increase may become significant as a percentage of resources +* If there are too many streams the extra write QPS or CPU load due to these updates are unacceptable +* If there are too many streams and/or a large source field (lot of participating tables) which generates unacceptable increase in the binlog size +* Even for a single stream, if the server is of a lower configuration, then the resulting increase in the QPS or binlog increase may become significant as a percentage of resources -**vreplicationHeartbeatUpdateInterval** determines how often the time_updated column is updated if there is no activity on the source and the source vstream is only sending heartbeats. Use a low value if you expect a high QPS or you are monitoring this column to alert about potential outages. Keep this high if: +_vreplication_heartbeat_update_interval_ determines how often the time_updated column is updated if there is no activity on the source and the source vstream is only sending heartbeats. Use a low value if you expect a high QPS or you are monitoring this column to alert about potential outages. Keep this high if: -* you have too many streams and the extra write QPS or CPU load due to these updates is unacceptable OR -* you have too many streams and/or a large binlogsource field (i.e., there are a lot of participating tables) which generates unacceptable increase in your binlog size +* You have too many streams and the extra write QPS or CPU load due to these updates is unacceptable OR +* You have too many streams and/or a large binlogsource field (i.e., there are a lot of participating tables) which generates unacceptable increase in your binlog size -Some internal processes (like online ddl) depend on the heartbeat updates for operating properly. Hence there is an upper limit on this interval, which is 60 seconds. +Some internal processes (like OnlineDDL) depend on the heartbeat updates for operating properly. Hence there is an upper limit on this interval, which is 60 seconds. #### vstream-binlog-rotation-threshold @@ -124,9 +124,9 @@ When starting a vstream which executes a query based on a [GTID](https://dev.mys **Default** 250000\ **Applicable on** source -On the source, events are buffered where applicable, to minimize network overhead. For example, multiple row events in a transaction or the set of begin/dml/commit event sets are buffered and sent together. Commits, DDLs, and synthetic events generated by VReplication like heartbeats, resharding journals cause the events buffered on the source to be sent immediately. +On the source, events are buffered and batched where applicable, to minimize network overhead. For example, multiple row events in a transaction or the set of begin/dml/commit event sets are buffered and sent together. Commits, DDLs, and synthetic events generated by VReplication like heartbeats and resharding journals cause the events buffered on the source to be sent immediately. -**vstream_packet_size** specifies the suggested packet size for VReplication streamer. This is used only as a recommendation. The actual packet size may be more or less than this amount depending on the number and type of events yet to be sent on the source. +**vstream_packet_size** specifies the suggested packet size for VReplication vstreamer. This is used only as a recommendation. The actual packet size may be more or less than this amount depending on the number and type of events yet to be sent on the source. #### watch_replication_stream @@ -134,7 +134,7 @@ On the source, events are buffered where applicable, to minimize network overhea **Default** false\ **Applicable on** source -By default vttablets reload their schema every `--queryserver-config-schema-reload-time` seconds (default 30 minutes). This can cause a problem while streaming events if DDLs are applied on the source and streaming is started _after_ the DDL was applied but _before_ vttablet refreshed its schema. This is alleviated by the _watcher_. +By default vttablets reload their schema every `--queryserver-config-schema-reload-time` seconds (default 30 minutes). This can cause a problem while streaming events if DDLs are applied on the source and streaming is started _after_ the DDL was applied but _before_ vttablet refreshed its schema. This is alleviated by enabling the _watcher_. When enabled, vttablet will start the _watcher_ which streams the MySQL replication stream from the local database, and uses it to proactively update its schema when it encounters a DDL. @@ -172,7 +172,7 @@ Stop automatically retrying when we've had consecutive failures with the same er **Default** in_order:REPLICA,PRIMARY\ **Applicable on** target -This parameter specifies the default tablet_types that will be used by the tablet picker to find sources for a VReplication stream. It can be overridden, per workflow, by passing a different list to the workflow commands like MoveTables and Reshard. +This parameter specifies the default tablet_types that will be used by the tablet picker to find sources for a VReplication stream. It can be overridden, per workflow, by passing a different list to the workflow commands like `MoveTables` and `Reshard`. #### vreplication_experimental_flags diff --git a/content/en/docs/16.0/reference/vreplication/materialize.md b/content/en/docs/16.0/reference/vreplication/materialize.md index 3593d3988..61fd74e4c 100644 --- a/content/en/docs/16.0/reference/vreplication/materialize.md +++ b/content/en/docs/16.0/reference/vreplication/materialize.md @@ -12,10 +12,10 @@ Materialize -- [--cells=] [--tablet_types=] }} Be careful to avoid using the `INSTANT ADD COLUMN` feature in [MySQL 8.0+](https://mysqlserverteam.com/mysql-8-0-innodb-now-supports-instant-add-column/) with materialization source tables as this can cause the vreplication based materialization workflow to break. @@ -40,19 +40,22 @@ cells should be used to pick a tablet for selecting data from the source keyspac * Select cells where replica lags are lower -#### --tablet_types +#### --tablet_types **optional**\ -**default** replica +**default** `--vreplication_tablet_type` parameter value for the tablet. `--vreplication_tablet_type` has the default value of "in_order:REPLICA,PRIMARY".\ +**string**
-A comma-separated list of tablet types that are used while picking a tablet for sourcing data. -One or more from PRIMARY, REPLICA, RDONLY.

+Source tablet types to replicate from (e.g. PRIMARY, REPLICA, RDONLY). + +
###### Uses * To reduce the load on PRIMARY tablets by using REPLICAs or RDONLYs * Reducing lag by pointing to PRIMARY + #### JSON spec details @@ -91,14 +94,14 @@ Materialize '{"workflow": "product_sales", "source_keyspace": "commerce", "targe Once you decide on your materialization requirements, you need to initiate a VReplication workflow as follows: -1. Initiate the migration using Materialize -2. Monitor the workflow using [Workflow](../workflow) or [VExec](../vexec) +1. Initiate the migration using `Materialize` +2. Monitor the workflow using [Workflow](../workflow) 3. Start accessing your views once the workflow has started Replicating ### Notes There are special commands to perform common materialization tasks and you should prefer them -to using Materialize directly. +to using `Materialize` directly. -* If you just want to copy tables to a different keyspace use [MoveTables](../movetables). +* If you just want to copy tables to a different keyspace use [MoveTables](../movetables) * If you want to change sharding strategies use [Reshard](../reshard) instead diff --git a/content/en/docs/16.0/reference/vreplication/metrics.md b/content/en/docs/16.0/reference/vreplication/metrics.md index 1506c8413..12b8448e8 100644 --- a/content/en/docs/16.0/reference/vreplication/metrics.md +++ b/content/en/docs/16.0/reference/vreplication/metrics.md @@ -4,13 +4,13 @@ description: Metrics related to vreplication functionality weight: 85 --- -VReplication exports several metrics using the expvars interface. These are available at the `debug/vars` endpoint of vttablet's http status pages. [More details here](../../features/monitoring/#3-push-based-metrics-system#3-push-based-metrics-system) +VReplication exports several metrics using the expvars interface. These are available at the `/debug/vars` endpoint of vttablet's http status pages. [More details can be found here](../../features/monitoring/#3-push-based-metrics-system). -## Target Metrics +## Target Tablet Metrics #### VReplicationCopyLoopCount, VReplicationCopyLoopCountTotal -During the copy phase we run one loop of bulk copy for approximately an hour at a time before running catchup. _VReplicationCopyLoopCount_ counts the number of times this loop has run for each stream and _VReplicationCopyLoopCountTotal_ the total across all streams. +During the copy phase we run one loop of bulk copy for approximately an hour at a time (by default) before running catchup. _VReplicationCopyLoopCount_ counts the number of times this loop has run for each stream and _VReplicationCopyLoopCountTotal_ the total across all streams. #### VReplicationCopyRowCount, VReplicationCopyRowCountTotal @@ -19,7 +19,7 @@ _VReplicationCopyRowCount_ counts the number of rows copied during the copy phas #### VReplicationErrors _VReplicationErrors_ counts the number of times errors occurred during vreplication. Errors are keyed -with the type of error. +by the type of error. #### VReplicationHeartbeat @@ -29,10 +29,6 @@ _VReplicationHeartbeat_ records, for each stream, the timestamp sent by the last _VReplicationMessages_ contains a stack of the last N (currently 3) messages of a vreplication stream. -#### VReplicationMessages - -_VReplicationMessages_ contains a stack of the last N (currently 3) messages of a vreplication stream. - #### VReplicationPhaseTimings, VReplicationPhaseTimingsCounts, VReplicationPhaseTimingsTotal This metric relates to the times each phase is run during the lifetime of a stream. @@ -56,10 +52,6 @@ _VReplicationQPS_ is a list of QPS values for each loop of each phase of the wor _VReplicationQueryCount_ is the total number of queries in each phase of a workflow. _VReplicationQueryCountTotal_ is the total queries across all phases and workflows. -#### VReplicationSecondsBehindMaster, VReplicationSecondsBehindMasterMax, VReplicationSecondsBehindMasterTotal - -Deprecated. See next section. - #### VReplicationLagSeconds, VReplicationLagSecondsMax, VReplicationLagSecondsTotal These metrics show the replication lag of the target stream with respect to the source stream. _VReplicationLagSeconds_ shows the current replication lag and _VReplicationLagSecondsMax_ has the maximum lag in this stream. Note that these values are only valid during the replication phase of a workflow. @@ -80,7 +72,7 @@ The number of streams running on this target This shows the state of each stream. -## Source Metrics +## Source Tablet Metrics #### VStreamPacketSize @@ -108,7 +100,7 @@ The total number of packets sent by this vttablet across all workflows #### VStreamersCreated -The total number of vstreamer created during the lifetime of this tablet +The total number of vstreamers created during the lifetime of this tablet
diff --git a/content/en/docs/16.0/reference/vreplication/migrate.md b/content/en/docs/16.0/reference/vreplication/migrate.md index d89938516..b7ed3cb6a 100644 --- a/content/en/docs/16.0/reference/vreplication/migrate.md +++ b/content/en/docs/16.0/reference/vreplication/migrate.md @@ -4,10 +4,6 @@ description: Move tables from an external cluster weight: 85 --- -{{< info >}} -This documentation is for a new (v2) set of vtctld commands that start in Vitess 11.0. See [RFC](https://github.com/vitessio/vitess/issues/7225) for more details. -{{< /info >}} - ### Command ``` @@ -17,8 +13,8 @@ Migrate -- ### Description -Migrate is used to start and manage vReplication workflows for copying keyspaces and/or tables from a source Vitess cluster, to a target Vitess cluster. -This command is built off of [MoveTables](../movetables) but has been extended to work with source and target topology services. It should be +Migrate is used to start and manage VReplication workflows for copying keyspaces and/or tables from a source Vitess cluster, to a target Vitess cluster. +This command is built off of [MoveTables](../movetables) but has been extended to work with independent source and target topology services. It should be utilized when moving Keyspaces or Tables between two separate Vitess environments. Migrate is an advantageous strategy for large sharded environments for a few reasons: @@ -30,20 +26,20 @@ for a few reasons: * Could be used for configuring lower environments with production data. Please note the Migrate command works with an externally mounted source cluster. See the related [Mount command](../mount) for more information -on external Vitess clusters. +on working with external Vitess clusters. -#### Differences between Migrate and MoveTables +#### Differences Between Migrate and MoveTables -Migrate has separate semantics and behaviors from MoveTables: +`Migrate` has separate semantics and behaviors from `MoveTables`: -* MoveTables migrates data from one keyspace to another, within the same Vitess cluster; Migrate functions between two separated Vitess clusters. -* MoveTables erases the source data upon completion by default; Migrate keeps the source data intact. +* `MoveTables` migrates data from one keyspace to another, within the same Vitess cluster; `Migrate` functions between two separated Vitess clusters. +* `MoveTables` erases the source data upon completion by default; Migrate keeps the source data intact. * There are flags available in MoveTables to change the default behavior in regards to the source data. -* MoveTables sets up routing rules and reverse replication, allowing for rollback prior to completion. - * Switching read/write traffic is not meaningful in the case of Migrate, as the Source is in a different cluster. - * Switching traffic requires the Target to have the ability to create vreplication streams (in the _vt database) on the Source; +* `MoveTables` sets up routing rules and reverse replication, allowing for rollback prior to completion. + * Switching read/write traffic is not meaningful in the case of `Migrate`, as the Source is in a different cluster. + * Switching traffic requires the Target to have the ability to create vreplication streams (in the `_vt` database) on the Source; this may not always be possible on production systems. -* Not all MoveTables options work with Migrate; for example [Progress](../progress) is unavailable with Migrate. +* Not all `MoveTables` options work with `Migrate`; for example [`Progress`](../progress) is unavailable with `Migrate`. ### Parameters @@ -66,7 +62,9 @@ If needed, you can rename the keyspace while migrating, simply provide a differe Each `action` has additional options/parameters that can be used to modify its behavior. -The options for the supported commands are the same as [MoveTables](../movetables), with the exception of `reverse_replication`. +The options for the supported commands are the same as [MoveTables](../movetables), with the exception of `--reverse_replication` as setting +up the reverse vreplication streams requires modifying the source cluster's `_vt` sidecar database which we cannot do as that database is +specific to a single Vitess cluster and these streams belong to a different one (the target cluster). A common option to give if migrating all of the tables from a source keyspace is the `--all` option. @@ -80,23 +78,23 @@ All workflows are identified by `targetKeyspace.workflow` where `targetKeyspace` ### A Migrate Workflow lifecycle {{< info >}} -NOTE: there is no reverse replication flow with Migrate. After the `Migrate Complete` command is given; no writes will be replicated between the Source and Target Vitess clusters. They are essentially two identical Vitess clusters running in two different environments. Once writing resumes on one of the clusters they will begin to drift apart. +NOTE: there is no reverse vreplication flow with `Migrate`. After the `Migrate Complete` command is given; no writes will be replicated between the Source and Target Vitess clusters. They are essentially two identical Vitess clusters running in two different environments. Once writing resumes on one of the clusters they will begin to drift apart. {{< /info >}} 1. Mount the source Vitess cluster using [Mount](../mount).
`Mount -- --type vitess --topo_type etcd2 --topo_server localhost:12379 --topo_root /vitess/global ext1` 1. Apply source vSchema to the Target's Keyspace.
`ApplyVSchema -- --vschema_file commerceVschema.json commerce` -1. Initiate the migration using [Create](../create).
+1. Initiate the migration using `Create`.
`Migrate -- --all --source ext1.commerce Create commerce.wf` -1. Monitor the workflow using [Show](../show).
+1. Monitor the workflow using `Show`.
`Workflow commerce.wf Show` 1. Confirm that data has been copied over correctly using [VDiff](../vdiff).
`VDiff commerce.wf` 1. Stop the application from writing to the source Vitess cluster.
1. Confirm again the data has been copied over correctly using [VDiff](../vdiff).
`VDiff commerce.wf` -1. Cleanup vreplication artifacts and source tables with [Complete](../complete).
+1. Cleanup vreplication artifacts and source tables with `Complete`.
`Migrate Complete commerce.wf` 1. Start the application pointed to the target Vitess Cluster. 1. Unmount the source cluster.
@@ -114,29 +112,29 @@ For Migrate to function properly, you will need to ensure communication is possi If you're migrating a keyspace from a production system, you may want to target a replica to reduce your load on the primary vttablets. This will also assist you in reducing the number of network considerations you need to make. ``` -Migrate -- --all --tablet_types "REPLICA" --source . Create +Migrate -- --all --tablet_types REPLICA --source . Create ``` To verify the Migration you can also perform VDiff with the `--tablet_types` option: ``` -VDiff -- --tablet_types "REPLICA" . +VDiff -- --tablet_types REPLICA . ``` ### Troubleshooting Errors -Migrate fails right away with error: +`Migrate` fails right away with error: ```sh E0224 23:51:45.312536 138 main.go:76] remote error: rpc error: code = Unknown desc = table table1 not found in vschema for keyspace sharded ```
Solution: -* The target table has a vSchema which does not match the source vSchema -* Upload the source vSchema to the target vSchema and try the migrate again +* The target table has a VSchema which does not match the source VSchema +* Upload the source VSchema to the target VSchema and try the `Migrate` again --- -Migrate fails right away with error: +`Migrate` fails right away with error: ```sh E0224 18:55:29.275019 578 main.go:76] remote error: rpc error: code = Unknown desc = node doesn't exist @@ -148,7 +146,7 @@ E0224 18:55:29.275019 578 main.go:76] remote error: rpc error: code = Unknow --- -After issuing Migrate command everything is stuck at 0% progress +After issuing `Migrate` command everything is stuck at 0% progress with errors found in target vttablet logs: ```sh diff --git a/content/en/docs/16.0/reference/vreplication/mount.md b/content/en/docs/16.0/reference/vreplication/mount.md index 94551330a..86426aaf5 100644 --- a/content/en/docs/16.0/reference/vreplication/mount.md +++ b/content/en/docs/16.0/reference/vreplication/mount.md @@ -4,10 +4,6 @@ description: Link an external cluster to the current one weight: 90 --- -{{< info >}} -This documentation is for a new (v2) set of vtctld commands that start in Vitess 11.0. See [RFC](https://github.com/vitessio/vitess/issues/7225) for more details. -{{< /info >}} - ### Command ``` @@ -17,12 +13,12 @@ Mount -- [--type vitess] [--topo_type=etcd2|consul|zookeeper] [--topo_server=top ### Description -Mount is used to link external Vitess clusters to the current cluster. (In the future we will also support mounting external MySQL servers.) +Mount is used to link external Vitess clusters to the current cluster. -Mounting Vitess clusters requires the topology information of the external cluster to be specified. Used in conjunction with [the Migrate command](../migrate). +Mounting Vitess clusters requires the topology information of the external cluster to be specified. Used in conjunction with [the `Migrate` command](../migrate). {{< info >}} -No validation is performed when using the Mount command. You must ensure your values are correct, or you may get errors when initializing a migration. +No validation is performed when using the `Mount` command. You must ensure your values are correct, or you may get errors when initializing a migration. {{< /info >}} @@ -36,18 +32,18 @@ The name that will be used in VReplication workflows to refer to the mounted clu Unmount an already mounted cluster. Requires `cluster_name` to be specified. -#### show +#### --show Show details of an already mounted cluster. Requires `cluster_name` to be specified. -#### list +#### --list -List all mounted clusters +List all mounted clusters. -### Topo parameters +### Topo Parameters -##### topo_type=[etcd2|consul|zookeeper] -##### topo_server= -##### topo_root= +##### --topo_type=[etcd2|consul|zookeeper] +##### --topo_server= +##### --topo_root= Mandatory (and only specified) while mounting a Vitess cluster. These should specify the topology parameters of the cluster being mounted. diff --git a/content/en/docs/16.0/reference/vreplication/movetables.md b/content/en/docs/16.0/reference/vreplication/movetables.md index ca33ec3d5..9a87969bc 100644 --- a/content/en/docs/16.0/reference/vreplication/movetables.md +++ b/content/en/docs/16.0/reference/vreplication/movetables.md @@ -5,10 +5,6 @@ weight: 10 aliases: ['/docs/reference/vreplication/v2/movetables/'] --- -{{< info >}} -This documentation is for a new (v2) set of vtctld commands that start in Vitess 11.0. See [RFC](https://github.com/vitessio/vitess/issues/7225) for more details. -{{< /info >}} - {{< warning >}} These workflows can have a significant impact on the source tablets (which are often in production) — especially when a PRIMARY tablet is used as a source. You can limit the impact on the source tablets using the [`--vreplication_copy_phase_max_*` vttablet flags](../flags/#vreplication_copy_phase_max_innodb_history_list_length) {{< /warning >}} @@ -16,7 +12,7 @@ These workflows can have a significant impact on the source tablets (which are o ## Command ``` -MoveTables +MoveTables -- ``` or @@ -30,96 +26,91 @@ MoveTables -- [--source=] [--tables=] [--cells=] ## Description -MoveTables is used to start and manage workflows to move one or more tables from an external database or an existing Vitess keyspace into a new Vitess keyspace. The target keyspace can be unsharded or sharded. +`MoveTables` is used to start and manage workflows to move one or more tables from an external database or an existing Vitess keyspace into a new Vitess keyspace. The target keyspace can be unsharded or sharded. -MoveTables is typically used for migrating data into Vitess or to implement vertical sharding. You might use the former when you first start using Vitess and the latter if you want to distribute your load across servers without sharding tables. +`MoveTables` is typically used for migrating data into Vitess or to implement vertical sharding. You might use the former when you first start using Vitess and the latter if you want to distribute your load across servers without sharding tables. ## Parameters ### action -MoveTables is an "umbrella" command. The `action` sub-command defines the operation on the workflow. -Action must be one of the following: Create, Complete, Cancel, SwitchTraffic, ReverseTrafffic, Show, or Progress. +`MoveTables` is an "umbrella" command. The `action` sub-command defines the operation on the workflow. +Action must be one of the following: `Create`, `Show`, `Progress`, `SwitchTraffic`, `ReverseTrafffic`, `Cancel`, or `Complete`. -### options +#### Create +
-Each `action` has additional options/parameters that can be used to modify its behavior. +`Create` sets up and creates a new workflow. The workflow name should not conflict with that of an existing workflow. -`actions` are common to both MoveTables and Reshard v2 workflows. Only the `create` action has different parameters, all other actions have common options and similar semantics. These actions are documented separately. +
-#### --source -**mandatory** +#### Show
-Name of existing keyspace that contains the tables to be moved +`Show` displays useful information about a workflow. (At this time the [Workflow](../workflow) Show command gives more information. This will be improved over time.)
-#### --tables -**optional** one of `table_specs` or `--all` needs to be specified +#### Progress
-_Either_ - -* a comma-separated list of tables - * if target keyspace is unsharded OR ' - * if target keyspace is sharded AND the tables being moved are already defined in the target's vschema +`Progress` reports the progress of a workflow by showing the percentage of data copied across targets, if workflow is in copy state, and the replication lag between the target and the source once the copy phase is completed. - Example: `MoveTables -- --source commerce --tables 'customer,corder' Create customer.commerce2customer` +It is too expensive to get real-time row counts of tables, using _count(*)_, say. So we use the statistics available in the `information_schema` to approximate copy progress. This data can be significantly off (up to 50-60%) depending on the utilization of the underlying mysql server resources. You can manually run `analyze table` to update the statistics if so desired. -_Or_ +
-* the JSON table section of the vschema for associated tables - * if target keyspace is sharded AND - * tables being moved are not yet present in the target's vschema +#### SwitchTraffic +
- Example: `MoveTables -- --source commerce --tables '{"t1":{"column_vindexes": [{"column": "id1", "name": "hash"}]}, "t2":{"column_vindexes": [{"column": "id2", "name": "hash"}]}}' Create customer.commerce2customer` +`SwitchTraffic` switches traffic forward for the `tablet_types` specified. This replaces the previous `SwitchReads` and `SwitchWrites` commands with a single one. It is now possible to switch all traffic with just one command, and this is the default behavior. Also, you can now switch replica, rdonly and primary traffic in any order: earlier you needed to first `SwitchReads` (for replicas and rdonly tablets) first before `SwitchWrites`.
-#### --cells -**optional**\ -**default** local cell\ -**string** - +#### ReverseTraffic
-Cell(s) or CellAlias(es) (comma-separated) to replicate from. +`ReverseTraffic` switches traffic in the reverse direction for the `tablet_types` specified. The traffic should have been previously switched forward using `SwitchTraffic` for the `cells` and `tablet_types` specified.
-#### --tablet_types -**optional**\ -**default** `--vreplication_tablet_type` parameter value for the tablet. `--vreplication_tablet_type` has the default value of "in_order:REPLICA,PRIMARY".\ -**string** - +#### Cancel
-Source tablet types to replicate from (e.g. PRIMARY, REPLICA, RDONLY). Defaults to --vreplication_tablet_type parameter value for the tablet, which has the default value of "in_order:REPLICA,PRIMARY". +`Cancel` can be used if a workflow was created in error or was misconfigured and you prefer to create a new workflow instead of fixing this one. `Cancel` can only be called if no traffic has been switched. It removes vreplication-related artifacts like rows from the vreplication and copy_state tables in the sidecar `_vt` database along with routing rules and blacklisted tables from the topo and, by default, the target tables on the target keyspace +(see [`--keep_data`](./#--keep_data) and [`--rename_tables`](#--rename_tables)).
-#### --all - -**optional** cannot specify `table_specs` if `--all` is specified +#### Complete
-Move all tables from the source keyspace. +{{< warning >}} +This is a destructive command +{{< /warning >}} + +`Complete` is used after all traffic has been switched. It removes vreplication-related artifacts like rows from vreplication and copy_state tables in the sidecar `_vt` database along with routing rules and and blacklisted tables from the topo. By default, the source tables are also dropped on the target keyspace +(see [`--keep_data`](./#--keep_data) and [`--rename_tables`](#--rename_tables)).
-#### --exclude +### options -**optional** only applies if `--all` is specified +Each `action` has additional options/parameters that can be used to modify its behavior. + +`actions` are common to both `MoveTables` and `Reshard` workflows. Only the `create` action has different parameters, all other actions have common options and similar semantics. + +#### --all + +**optional** cannot specify `table_specs` if `--all` is specified
-If moving all tables, specifies tables to be skipped. +Move all tables from the source keyspace.
#### --auto_start - -**optional** +**optional**\ **default** true
@@ -130,53 +121,60 @@ to false then the workflow is in a Stopped state until you explicitly start it.
###### Uses -* allows updating the rows in `_vt.vreplication` after MoveTables has setup the + +* Allows updating the rows in `_vt.vreplication` after `MoveTables` has setup the streams. For example, you can add some filters to specific tables or change the projection clause to modify the values on the target. This provides an easier way to create simpler Materialize workflows by first using -MoveTables with auto_start false, updating the BinlogSource as required by your -Materialize and then start the workflow. -* changing the `copy_state` and/or `pos` values to restart a broken MoveTables workflow -from a specific point of time. +`MoveTables` with auto_start false, updating the BinlogSource as required by your +`Materialize` and then start the workflow. +* Changing the `copy_state` and/or `pos` values to restart a broken `MoveTables` workflow +from a specific point of time -#### --stop_after_copy +#### --cells +**optional**\ +**default** local cell (of source tablet)\ +**string** + +
+ +Comma seperated list of Cell(s) and/or CellAlias(es) to replicate from. +
+ +###### Uses + +* Improve performance by picking a tablet in cells in network proximity with the target +* Reduce bandwidth costs by skipping cells that are in different availability zones +* Select cells where replica lags are lower + +#### --drop_foreign_keys **optional** **default** false
-If set, the workflow will stop once the Copy phase has been completed i.e. once -all tables have been copied and VReplication decides that the lag -is small enough to start replicating, the workflow state will be set to Stopped. - -###### Uses -* If you just want a consistent snapshot of all the tables you can set this flag. The workflow -will stop once the copy is done and you can then mark the workflow as `Complete`d +If true, tables in the target keyspace will be created without any foreign keys that exist on the source.
-#### --timeout +#### --dry_run **optional**\ -**default** 30s +**default** false
-For primary tablets, SwitchTraffic first stops writes on the source primary and waits for the replication to the target to -catchup with the point where the writes were stopped. If the wait time is longer than timeout -the command will error out. For setups with high write qps you may need to increase this value. +For the `SwitchTraffic`, `ReverseTraffic`, and `Complete` actions, you can do a dry run where no actual steps are taken +but the command logs all the steps that would be taken.
-#### --reverse_replication -**optional**\ -**default** true +#### --exclude +**optional** only applies if `--all` is specified
-SwitchTraffic for primary tablet types, by default, starts a reverse replication stream with the current target as the source, replicating back to the original source. This enables a quick and simple rollback using ReverseTraffic. This reverse workflow name is that of the original workflow concatenated with \_reverse. - -If set to false these reverse replication streams will not be created and you will not be able to rollback once you have switched write traffic over to the target. +If moving all tables, specifies tables to be skipped.
@@ -186,7 +184,7 @@ If set to false these reverse replication streams will not be created and you wi
-Usually, any data created by the workflow in the source and target (tables or shards) are deleted by Complete or Cancel. If this flag is used the data will be left in place. +Usually, the target tables are deleted by `Cancel`. If this flag is used the target tables will not be deleted.
@@ -196,29 +194,21 @@ Usually, any data created by the workflow in the source and target (tables or sh
-Usually, any routing rules created by the workflow in the source and target keyspace are removed by Complete or Cancel. If this flag is used the routing rules will be left in place. +Usually, any routing rules created by the workflow in the source and target keyspace are removed by `Complete` or `Cancel`. If this flag is used the routing rules will be left in place.
-#### --source_time_zone +#### --max_replication_lag_allowed **optional**\ -**default** "" +**default** the value used for `--timeout`
-Specifying this flag causes all `DATETIME` fields to be converted from the given time zone into `UTC`. It is expected that the application has -stored *all* `DATETIME` fields, in all tables being moved, in the specified time zone. On the target these `DATETIME` values will be stored in `UTC`. +While executing `SwitchTraffic` we ensure that the VReplication lag for the workflow is less than this duration, otherwise report an error and don't attempt the switch. The calculated VReplication lag is the estimated maximum lag across workflow streams between the last event seen at the source and the last event processed by the target (which would be a heartbeat event if we're fully caught up). Usually, when VReplication has caught up, this lag should be very small (under a second). -As a best practice, Vitess expects users to run their MySQL servers in `UTC`. So we do not specify a target time zone for the conversion. -It is expected that the [time zone tables have been pre-populated](https://dev.mysql.com/doc/refman/en/time-zone-support.html#time-zone-installation) on the target mysql servers. +While switching write traffic, we temporarily make the source databases read-only, and wait for the targets to catchup. This means that the application can effectively be partially down for this cutover period as writes will pause or error out. While switching write traffic this flag can ensure that you only switch traffic if the current lag is low, thus limiting this period of write-unavailability and avoiding it entirely if we're not likely to catch up within the `--timeout` window. -Any reverse replication streams running after a SwitchWrites will do the reverse date conversion on the source. - -Note that selecting the `DATETIME` columns from the target will now give the times in UTC. It is expected that the application will -perform any conversions using, for example, `SET GLOBAL time_zone = 'US/Pacific'`or `convert_tz()`. - -Also note that only columns of `DATETIME` data types are converted. If you store `DATETIME` values as `VARCHAR` or `VARBINARY` strings, -setting this flag will not convert them. +While switching read traffic this can also be used to set an approximate upper bound on how stale reads will be against the replica tablets when using `@replica` shard targeting.
@@ -246,31 +236,150 @@ We caution against against using `EXEC` or `EXEC_IGNORE` for the following reaso * You may want a different schema on the target. * You may want to apply the DDL in a different way on the target. * The DDL may take a long time to apply on the target and may disrupt replication, performance, and query execution (if serving traffic from the target) while it is being applied. + +If you do use one of these two DDL actions then you should also enable the [VReplication Schema Tracking](../../../../design-docs/vreplication/vstream/tracker/). {{< /warning >}} +#### --rename_tables +**optional**\ +**default** false + +
+ +During `Complete` or `Cancel` operations, the tables are renamed instead of being deleted. Currently the new name is _<table_name>_old. + +We use the same renaming logic used by [`pt-online-schema-change`](https://docs.percona.com/percona-toolkit/pt-online-schema-change.html). +Such tables are automatically skipped by vreplication if they exist on the source. + +
+ +#### --reverse_replication +**optional**\ +**default** true + +
+ +`SwitchTraffic` for primary tablet types, by default, starts a reverse replication stream with the current target as the source, replicating back to the original source. This enables a quick and simple rollback mechanism using `ReverseTraffic`. This reverse workflow name is that of the original workflow concatenated with \_reverse. + +If set to false these reverse replication streams will not be created and you will not be able to rollback once you have switched write traffic over to the target. + +
+ +#### --source +**mandatory** +
+ +Name of existing keyspace that contains the tables to be moved. + +
+ +#### --source_time_zone +**optional**\ +**default** "" + +
+ +Specifying this flag causes all `DATETIME` fields to be converted from the given time zone into `UTC`. It is expected that the application has +stored *all* `DATETIME` fields, in all tables being moved, in the specified time zone. On the target these `DATETIME` values will be stored in `UTC`. + +As a best practice, Vitess expects users to run their MySQL servers in `UTC`. So we do not specify a target time zone for the conversion. +It is expected that the [time zone tables have been pre-populated](https://dev.mysql.com/doc/refman/en/time-zone-support.html#time-zone-installation) on the target mysql servers. + +Any reverse replication streams running after a SwitchWrites will do the reverse date conversion on the source. + +Note that selecting the `DATETIME` columns from the target will now give the times in UTC. It is expected that the application will +perform any conversions using, for example, `SET GLOBAL time_zone = 'US/Pacific'`or `convert_tz()`. + +Also note that only columns of `DATETIME` data types are converted. If you store `DATETIME` values as `VARCHAR` or `VARBINARY` strings, +setting this flag will not convert them. + +
+ +#### --stop_after_copy + +**optional** +**default** false + +
+ +If set, the workflow will stop once the Copy phase has been completed i.e. once +all tables have been copied and VReplication decides that the lag +is small enough to start replicating, the workflow state will be set to Stopped. + +
+ +###### Uses +* If you just want a consistent snapshot of all the tables you can set this flag. The workflow +will stop once the copy is done and you can then mark the workflow as `Complete`. + +#### --tables +**optional** one of `--tables` or `--all` needs to be specified +
+ +_Either_ + +* a comma-separated list of tables + * if target keyspace is unsharded OR + * if target keyspace is sharded AND the tables being moved are already defined in the target's vschema + + Example: `MoveTables -- --source commerce --tables 'customer,corder' Create customer.commerce2customer` + +_Or_ + +* the JSON table section of the vschema for associated tables + * if target keyspace is sharded AND + * tables being moved are not yet present in the target's vschema + + Example: `MoveTables -- --source commerce --tables '{"t1":{"column_vindexes": [{"column": "id1", "name": "hash"}]}, "t2":{"column_vindexes": [{"column": "id2", "name": "hash"}]}}' Create customer.commerce2customer` + +
+ +#### --tablet_types +**optional**\ +**default** `--vreplication_tablet_type` parameter value for the tablet. `--vreplication_tablet_type` has the default value of "in_order:REPLICA,PRIMARY".\ +**string** + +
+ +Source tablet types to replicate from (e.g. PRIMARY, REPLICA, RDONLY). The value +specified impacts [tablet selection](../tablet_selection/) for the workflow. + +
+ +#### --timeout +**optional**\ +**default** 30s + +
+ +For primary tablets, SwitchTraffic first stops writes on the source primary and waits for the replication to the target to +catchup with the point where the writes were stopped. If the wait time is longer than timeout +the command will error out. For setups with high write qps you may need to increase this value. + +
### workflow identifier
-All workflows are identified by `targetKeyspace.workflow` where `targetKeyspace` is the name of the keyspace to which the tables are being moved. `workflow` is a name you assign to the MoveTables workflow to identify it. +All workflows are identified by `targetKeyspace.workflow` where `targetKeyspace` is the name of the keyspace to which the tables are being moved. `workflow` is a name you assign to the `MoveTables` workflow to identify it.
## The most basic MoveTables Workflow lifecycle -1. Initiate the migration using [Create](../create)
+1. Initiate the migration using `Create`
`MoveTables -- --source= --tables= Create ` -1. Monitor the workflow using [Show](../show) or [Progress](../progress)
+1. Monitor the workflow using `Show` or `Progress`
`MoveTables Show ` _*or*_
`MoveTables Progress `
1. Confirm that data has been copied over correctly using [VDiff](../vdiff) -1. Cutover to the target keyspace with [SwitchTraffic](../switchtraffic)
+1. Cutover to the target keyspace with `SwitchTraffic`
`MoveTables SwitchTraffic ` -1. Cleanup vreplication artifacts and source tables with [Complete](../complete)
+1. Cleanup vreplication artifacts and source tables with `Complete`
`MoveTables Complete ` @@ -278,16 +387,16 @@ All workflows are identified by `targetKeyspace.workflow` where `targetKeyspace` ### Adopting Vitess -For those wanting to try out Vitess for the first time, MoveTables provides an easy way to route part of their workload to Vitess with the ability to migrate back at any time without any risk. You point a vttablet to your existing MySQL installation, spin up an unsharded Vitess cluster and use a MoveTables workflow to start serving some tables from Vitess. You can also go further and use a Reshard workflow to experiment with a sharded version of a part of your database. +For those wanting to try out Vitess for the first time, `MoveTables` provides an easy way to route part of their workload to Vitess with the ability to migrate back at any time without any risk. You point a vttablet to your existing MySQL installation, spin up an unsharded Vitess cluster and use a `MoveTables` workflow to start serving some tables from Vitess. You can also go further and use a Reshard workflow to experiment with a sharded version of a part of your database. -See this [user guide](../../../../../docs/user-guides/configuration-advanced/unmanaged-tablet/#move-legacytable-to-the-commerce-keyspace) for detailed steps. +See this [user guide](../../../user-guides/configuration-advanced/unmanaged-tablet/#move-legacytable-to-the-commerce-keyspace) for detailed steps. ### Vertical Sharding For existing Vitess users you can easily move one or more tables to another keyspace, either for balancing load or as preparation for sharding your tables. -See this [user guide](../../../../../docs/user-guides/migration/move-tables/) which describes how MoveTables works in the local example provided in the Vitess repo. +See this [user guide](../../../user-guides/migration/move-tables/) which describes how `MoveTables` works in the local example provided in the Vitess repo. ### More Reading -* [MoveTables in practice](../../../../../docs/concepts/move-tables/) +* [`MoveTables` in practice](../../../concepts/move-tables/) diff --git a/content/en/docs/16.0/reference/vreplication/progress.md b/content/en/docs/16.0/reference/vreplication/progress.md deleted file mode 100644 index d26c5b9df..000000000 --- a/content/en/docs/16.0/reference/vreplication/progress.md +++ /dev/null @@ -1,23 +0,0 @@ ---- -title: --- Progress -description: Show copy progress and replication lag of a workflow -weight: 36 -aliases: ['/docs/reference/vreplication/v2/progress/'] ---- - -{{< info >}} -This documentation is for a new (v2) set of vtctld commands that start in Vitess 11.0. See [RFC](https://github.com/vitessio/vitess/issues/7225) for more details. -{{< /info >}} - -### Command - -``` -MoveTables/Reshard Progress -``` - -### Description -Workflows start in the copy state, (details in [VReplication Life of a stream](../../../design-docs/vreplication/life-of-a-stream), doing a bulk copy of the tables involved until they reach a low replication lag, after which we stream binlog events. Tables are copied sequentially. - -`Progress` reports the progress of a workflow by showing the percentage of data copied across targets, if workflow is in copy state, and the replication lag between the target and the source once the copy phase is completed. - -It is too expensive to get real-time row counts of tables, using _count(*)_, say. So we use the statistics available in the `information_schema` to approximate copy progress. This data can be significantly off (up to 50-60%) depending on the utilization of the underlying mysql server resources. You can manually run `analyze table` to update the statistics if so desired. diff --git a/content/en/docs/16.0/reference/vreplication/reshard.md b/content/en/docs/16.0/reference/vreplication/reshard.md index 480359681..7b48a0502 100644 --- a/content/en/docs/16.0/reference/vreplication/reshard.md +++ b/content/en/docs/16.0/reference/vreplication/reshard.md @@ -5,10 +5,6 @@ weight: 20 aliases: ['/docs/reference/vreplication/v2/reshard/'] --- -{{< info >}} -This documentation is for a new (v2) set of vtctld commands that start in Vitess 11.0. See [RFC](https://github.com/vitessio/vitess/issues/7225) for more details. -{{< /info >}} - {{< warning >}} These workflows can have a significant impact on the source tablets (which are often in production) — especially when a PRIMARY tablet is used as a source. You can limit the impact on the source tablets using the [`--vreplication_copy_phase_max_*` vttablet flags](../flags/#vreplication_copy_phase_max_innodb_history_list_length) {{< /warning >}} @@ -16,7 +12,7 @@ These workflows can have a significant impact on the source tablets (which are o ## Command ``` -Reshard +Reshard -- ``` or @@ -30,70 +26,81 @@ Reshard -- [--source_shards=] [--target_shards=] [ ## Description -Reshard is used to create and manage workflows to horizontally shard an existing keyspace. The source keyspace can be unsharded or sharded. +`Reshard` is used to create and manage workflows to horizontally shard an existing keyspace. The source keyspace can be unsharded or sharded. + ## Parameters ### action +`Reshard` is an "umbrella" command. The `action` sub-command defines the operation on the workflow. +Action must be one of the following: `Create`, `Show`, `Progress`, `SwitchTraffic`, `ReverseTrafffic`, `Cancel`, or `Complete`. + +#### Create
-Reshard is an "umbrella" command. The `action` sub-command defines the operation on the workflow. -Action must be one of the following: Create, Complete, Cancel, SwitchTraffic, ReverseTrafffic, Show, or Progress +`Create` sets up and creates a new workflow. The workflow name should not conflict with that of an existing workflow.
-### options +#### Show
-Each `action` has additional options/parameters that can be used to modify its behavior. - -`actions` are common to both MoveTables and Reshard v2 workflows. Only the `create` action has different parameters, all other actions have common options and similar semantics. These actions are documented separately. +`Show` displays useful information about a workflow. (At this time the [Workflow](../workflow) Show command gives more information. This will be improved over time.)
-#### source_shards -**mandatory** - +#### Progress
-Comma separated shard names to reshard from. -
-#### target_shards -**mandatory** +`Progress` reports the progress of a workflow by showing the percentage of data copied across targets, if workflow is in copy state, and the replication lag between the target and the source once the copy phase is completed. -
-Comma separated shard names to reshard to. -
+It is too expensive to get real-time row counts of tables, using _count(*)_, say. So we use the statistics available in the `information_schema` to approximate copy progress. This data can be significantly off (up to 50-60%) depending on the utilization of the underlying mysql server resources. You can manually run `analyze table` to update the statistics if so desired. -#### --cells -**optional**\ -**default** local cell + +#### SwitchTraffic
-Comma separated Cell(s) or CellAlias(es) to replicate from. -
-#### --tablet_types -**optional**\ -**default** empty +`SwitchTraffic` switches traffic forward for the `tablet_types` specified. This replaces the previous `SwitchReads` and `SwitchWrites` commands with a single one. It is now possible to switch all traffic with just one command, and this is the default behavior. Also, you can now switch replica, rdonly and primary traffic in any order: earlier you needed to first `SwitchReads` (for replicas and rdonly tablets) first before `SwitchWrites`. + + +#### ReverseTraffic
-Source Vitess tablet_type, or comma separated list of tablet types, that should be used for choosing source tablet(s) for the reshard. + +`ReverseTraffic` switches traffic in the reverse direction for the `tablet_types` specified. The traffic should have been previously switched forward using `SwitchTraffic` for the `cells` and `tablet_types` specified. +
-#### --skip_schema_copy -**optional**\ -**default** false +#### Cancel +
+ +`Cancel` can be used if a workflow was created in error or was misconfigured and you prefer to create a new workflow instead of fixing this one. Cancel can only be called if no traffic has been switched. It removes vreplication-related artifacts like rows from vreplication and copy_state tables in the sidecar `_vt` database along with the new target shards from the topo and, by default, the target tables on the target keyspace +(see [`--keep_data`](./#--keep_data) and [`--rename_tables`](#--rename_tables)). + +
+#### Complete
-If false, the source schema is copied to the target shards. If true, the schema won't be copied -and you need to create the tables before calling reshard. + +{{< warning >}} +This is a destructive command +{{< /warning >}} + +`Complete` is used after all traffic has been switched. It removes vreplication-related artifacts like rows from vreplication and copy_state tables in the sidecar `_vt` database along with the original source shards from the topo. By default, the source tables are also dropped on the source shards +(see [`--keep_data`](./#--keep_data) and [`--rename_tables`](#--rename_tables)) . +
-#### --auto_start +### options -**optional** +Each `action` has additional options/parameters that can be used to modify its behavior. + +`actions` are common to both `MoveTables` and `Reshard` workflows. Only the `create` action has different parameters, all other actions have common options and similar semantics. + +#### --auto_start +**optional**\ **default** true
@@ -104,53 +111,60 @@ to false then the workflow is in a Stopped state until you explicitly start it.
###### Uses -* allows updating the rows in `_vt.vreplication` after MoveTables has setup the + +* Allows updating the rows in `_vt.vreplication` after `Reshard` has setup the streams. For example, you can add some filters to specific tables or change the projection clause to modify the values on the target. This provides an easier way to create simpler Materialize workflows by first using -MoveTables with auto_start false, updating the BinlogSource as required by your +`Reshard` with auto_start false, updating the BinlogSource as required by your Materialize and then start the workflow. -* changing the `copy_state` and/or `pos` values to restart a broken MoveTables workflow -from a specific point of time. +* Changing the `copy_state` and/or `pos` values to restart a broken `Reshard` workflow +from a specific point of time -#### --stop_after_copy +#### --cells +**optional**\ +**default** local cell (of source tablet)\ +**string** +
+ +Comma seperated list of Cell(s) and/or CellAlias(es) to replicate from. + +
+ +###### Uses + +* Improve performance by picking a tablet in cells in network proximity with the target +* Reduce bandwidth costs by skipping cells that are in different availability zones +* Select cells where replica lags are lower + +#### --drop_foreign_keys **optional** **default** false
-If set, the workflow will stop once the Copy phase has been completed i.e. once -all tables have been copied and VReplication decides that the lag -is small enough to start replicating, the workflow state will be set to Stopped. - -###### Uses -* If you just want a consistent snapshot of all the tables you can set this flag. The workflow -will stop once the copy is done and you can then mark the workflow as `Complete`d +If true, tables in the target keyspace will be created without any foreign keys that exist on the source.
-#### --timeout +#### --dry_run **optional**\ -**default** 30s +**default** false
-For primary tablets, SwitchTraffic first stops writes on the source primary and waits for the replication to the target to -catchup with the point where the writes were stopped. If the wait time is longer than timeout -the command will error out. For setups with high write qps you may need to increase this value. +For the `SwitchTraffic`, `ReverseTraffic`, and `Complete` actions, you can do a dry run where no actual steps are taken +but the command logs all the steps that would be taken.
-#### --reverse_replication -**optional**\ -**default** true +#### --exclude +**optional** only applies if `--all` is specified
-SwitchTraffic for primary tablet types, by default, starts a reverse replication stream with the current target as the source, replicating back to the original source. This enables a quick and simple rollback using ReverseTraffic. This reverse workflow name is that of the original workflow concatenated with \_reverse. - -If set to false these reverse replication streams will not be created and you will not be able to rollback once you have switched write traffic over to the target. +If moving all tables, specifies tables to be skipped.
@@ -160,7 +174,7 @@ If set to false these reverse replication streams will not be created and you wi
-Usually, the target data (tables or shards) are deleted by Cancel. If this flag is used with MoveTables, target tables will not be deleted and, with Reshard, target shards will not be dropped. +Usually, the target tables are deleted by `Cancel`. If this flag is used the target tables will not be deleted.
@@ -170,7 +184,21 @@ Usually, the target data (tables or shards) are deleted by Cancel. If this flag
-Usually, any routing rules created by the workflow in the source and target keyspace are removed by Complete or Cancel. If this flag is used the routing rules will be left in place. +Usually, any routing rules created by the workflow in the source and target keyspace are removed by `Complete` or `Cancel`. If this flag is used the routing rules will be left in place. + +
+ +#### --max_replication_lag_allowed +**optional**\ +**default** the value used for `--timeout` + +
+ +While executing `SwitchTraffic` we ensure that the VReplication lag for the workflow is less than this duration, otherwise report an error and don't attempt the switch. The calculated VReplication lag is the estimated maximum lag across workflow streams between the last event seen at the source and the last event processed by the target (which would be a heartbeat event if we're fully caught up). Usually, when VReplication has caught up, this lag should be very small (under a second). + +While switching write traffic, we temporarily make the source databases read-only, and wait for the targets to catchup. This means that the application can effectively be partially down for this cutover period as writes will pause or error out. While switching write traffic this flag can ensure that you only switch traffic if the current lag is low, thus limiting this period of write-unavailability and avoiding it entirely if we're not likely to catch up within the `--timeout` window. + +While switching read traffic this can also be used to set an approximate upper bound on how stale reads will be against the replica tablets when using `@replica` shard targeting.
@@ -198,28 +226,102 @@ We caution against against using `EXEC` or `EXEC_IGNORE` for the following reaso * You may want a different schema on the target. * You may want to apply the DDL in a different way on the target. * The DDL may take a long time to apply on the target and may disrupt replication, performance, and query execution (if serving traffic from the target) while it is being applied. + +If you do use one of these two DDL actions then you should also enable the [VReplication Schema Tracking](../../../../design-docs/vreplication/vstream/tracker/). {{< /warning >}} +#### --reverse_replication +**optional**\ +**default** true + +
+ +`SwitchTraffic` for primary tablet types, by default, starts a reverse replication stream with the current target as the source, replicating back to the original source. This enables a quick and simple rollback mechanism using `ReverseTraffic`. This reverse workflow name is that of the original workflow concatenated with \_reverse. + +If set to false these reverse replication streams will not be created and you will not be able to rollback once you have switched write traffic over to the target. + +
+ +#### --stop_after_copy + +**optional** +**default** false + +
+ +If set, the workflow will stop once the Copy phase has been completed i.e. once +all tables have been copied and VReplication decides that the lag +is small enough to start replicating, the workflow state will be set to Stopped. + +
+ +###### Uses +* If you just want a consistent snapshot of all the tables you can set this flag. The workflow +will stop once the copy is done and you can then mark the workflow as `Complete`d + +#### --source_shards +**mandatory** + +
+ +Comma separated shard names to reshard from. + +
+ +#### --tablet_types +**optional**\ +**default** `--vreplication_tablet_type` parameter value for the tablet. `--vreplication_tablet_type` has the default value of "in_order:REPLICA,PRIMARY".\ +**string** + +
+ +Source tablet types to replicate from (e.g. PRIMARY, REPLICA, RDONLY). The value +specified impacts [tablet selection](../tablet_selection/) for the workflow. + +
+ +#### --target_shards +**mandatory** + +
+ +Comma separated shard names to reshard to. + +
+ +#### --timeout +**optional**\ +**default** 30s + +
+ +For primary tablets, SwitchTraffic first stops writes on the source primary and waits for the replication to the target to +catchup with the point where the writes were stopped. If the wait time is longer than timeout +the command will error out. For setups with high write qps you may need to increase this value. + +
+ + #### workflow identifier
-All workflows are identified by `targetKeyspace.workflow` where `targetKeyspace` is the name of the keyspace to which the tables are being moved. `workflow` is a name you assign to the Reshard workflow to identify it. +All workflows are identified by `targetKeyspace.workflow` where `targetKeyspace` is the name of the keyspace to which the tables are being moved. `workflow` is a name you assign to the `Reshard` workflow to identify it.
### The most basic Reshard Workflow lifecycle -1. Initiate the migration using [Create](../create)
+1. Initiate the migration using `Create`
`Reshard -- --source_shards= --target_shards= Create ` -1. Monitor the workflow using [Show](../show) or [Progress](../progress)
+1. Monitor the workflow using `Show` or `Progress`
`Reshard Show ` _*or*_
`Reshard Progress `
1. Confirm that data has been copied over correctly using [VDiff](../vdiff) -1. Cutover to the target keyspace with [SwitchTraffic](../switchtraffic)
+1. Cutover to the target keyspace with `SwitchTraffic`
`Reshard SwitchTraffic ` -1. Cleanup vreplication artifacts and source shards with [Complete](../complete)
+1. Cleanup vreplication artifacts and source shards with `Complete`
`Reshard Complete ` diff --git a/content/en/docs/16.0/reference/vreplication/reversetraffic.md b/content/en/docs/16.0/reference/vreplication/reversetraffic.md deleted file mode 100644 index cea54a1a0..000000000 --- a/content/en/docs/16.0/reference/vreplication/reversetraffic.md +++ /dev/null @@ -1,66 +0,0 @@ ---- -title: --- ReverseTraffic -description: Reverse traffic from target to source -weight: 33 -aliases: ['/docs/reference/vreplication/v2/reversetraffic/'] ---- - -{{< info >}} -This documentation is for a new (v2) set of vtctld commands that start in Vitess 11.0. See [RFC](https://github.com/vitessio/vitess/issues/7225) for more details. -{{< /info >}} - -### Command - -``` -MoveTables/Reshard -- [--tablet_types ] [--cells ] - [--timeout=timeoutDuration] [--dry_run] - ReverseTraffic -``` - -### Description - -`ReverseTraffic` switches traffic in the reverse direction for the tablet_types specified. The traffic should have been previously switched forward using SwitchTraffic for the cells/tablet_types specified. - -### Parameters - -#### --cells -**optional**\ -**default** all cells - -
- -A comma-separated list of cell names or cell aliases. Traffic will be reversed only in these cells for the -specified tablet types. - -
- -#### --tablet_types -**optional**\ -**default** all (in_order:RDONLY,REPLICA,PRIMARY) - -
- -A comma-separated list of tablet types for which traffic is to be reversed. -One or more from primary, replica, rdonly.

- -
- - -#### --timeout -**optional**\ -**default** 30s - -
- -For primary tablets, ReverseTraffic first stops writes on the target primary and waits for the replication on the source to catchup with the gtid on the target where the writes were stopped. If the wait time is longer than timeout the command will error out. For setups with high write qps you may need to increase this value. - -
- -#### --dry-run -**optional**\ -**default** false - -
-You can do a dry run where no actual action is taken but the command logs all the actions that would be taken -by ReverseTraffic. -
diff --git a/content/en/docs/16.0/reference/vreplication/show.md b/content/en/docs/16.0/reference/vreplication/show.md deleted file mode 100644 index 288d563b2..000000000 --- a/content/en/docs/16.0/reference/vreplication/show.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: --- Show -description: Show information about a workflow -weight: 35 -aliases: ['/docs/reference/vreplication/v2/show/'] ---- - -{{< info >}} -This documentation is for a new (v2) set of vtctld commands that start in Vitess 11.0. See [RFC](https://github.com/vitessio/vitess/issues/7225) for more details. -{{< /info >}} - -### Command - -``` -MoveTables/Reshard Show -``` - -### Description - -`Show` displays useful information about a workflow. (At this time the [Workflow](../workflow) Show command gives more information. This will be improved over time.) diff --git a/content/en/docs/16.0/reference/vreplication/switchtraffic.md b/content/en/docs/16.0/reference/vreplication/switchtraffic.md deleted file mode 100644 index f6d54803a..000000000 --- a/content/en/docs/16.0/reference/vreplication/switchtraffic.md +++ /dev/null @@ -1,93 +0,0 @@ ---- -title: --- SwitchTraffic -description: Switch traffic from source to target -weight: 32 -aliases: ['/docs/reference/vreplication/v2/switchtraffic/'] ---- - -{{< info >}} -This documentation is for a new (v2) set of vtctld commands that start in Vitess 11.0. See [RFC](https://github.com/vitessio/vitess/issues/7225) for more details. -{{< /info >}} - -### Command - -``` -MoveTables/Reshard -- [--tablet_types=] [--cells=] - [--timeout=timeoutDuration] [--reverse_replication] [--dry_run] - SwitchTraffic -``` - -### Description - -`SwitchTraffic` switches traffic forward for the tablet_types specified. This replaces the previous SwitchReads and SwitchWrites commands with a single one. It is now possible to switch all traffic with just one command. Also, you can now switch replica, rdonly and primary traffic in any order: earlier you needed to first SwitchReads (for replicas and rdonly tablets) first before SwitchWrites. - -### Parameters - -#### --cells -**optional**\ -**default** all cells - -
- -A comma-separated list of cell names or cell aliases. Traffic will be switched only in these cells for the -specified tablet types. - -
- -#### --tablet_types -**optional**\ -**default** all (in_order:RDONLY,REPLICA,PRIMARY) - -
- -A comma-separated list of tablet types for which traffic is to be switched. -One or more from primary,replica,rdonly.

- -
- -#### --timeout -**optional**\ -**default** 30s - -
- -For primary tablets, SwitchTraffic first stops writes on the source primary and waits for the replication to the target to -catchup with the point where the writes were stopped. If the wait time is longer than timeout -the command will error out. For setups with high write qps you may need to increase this value. - -
- -#### --reverse_replication -**optional**\ -**default** true - -
- -SwitchTraffic for primary tablet types, by default, starts a reverse replication stream with the current target as the source, replicating back to the original source. This enables a quick and simple rollback using ReverseTraffic. This reverse workflow name is that of the original workflow concatenated with \_reverse. - -If set to false these reverse replication streams will not be created and you will not be able to rollback once you have switched write traffic over to the target. - -
- -#### --max_replication_lag_allowed -**optional**\ -**default** the value used for `--timeout` - -
- -While switching traffic ensure that the VReplication lag for the workflow is less than this duration, otherwise report an error and don't attempt the switch. The calculated VReplication lag is the estimated maximum lag across workflow streams between the last event seen at the source and the last event processed by the target (which would be a heartbeat event if we're fully caught up). Usually, when VReplication has caught up, this lag should be very small (under a second). - -While switching write traffic, we temporarily make the source databases read-only, and wait for the targets to catchup. This means that the application can effectively be partially down for this cutover period as writes will pause or error out. While switching write traffic this flag can ensure that you only switch traffic if the current lag is low, thus limiting this period of write-unavailability and avoiding it entirely if we're not likely to catch up within the `--timeout` window. - -While switching read traffic this can also be used to set an approximate upper bound on how stale reads will be against the replica tablets when using `@replica` shard targeting. - -
- -#### --dry-run -**optional**\ -**default** false - -
-You can do a dry run where no actual action is taken but the command logs all the actions that would be taken -by SwitchTraffic. -
diff --git a/content/en/docs/16.0/reference/vreplication/tablet_selection.md b/content/en/docs/16.0/reference/vreplication/tablet_selection.md index bbae7b4bd..6ef70cc68 100644 --- a/content/en/docs/16.0/reference/vreplication/tablet_selection.md +++ b/content/en/docs/16.0/reference/vreplication/tablet_selection.md @@ -1,24 +1,40 @@ --- -title: Tablet selection +title: Tablet Selection weight: 300 --- ### Introduction -For [VReplication streams](../../../concepts/vstream/), we must choose a tablet to serve the role of source (vstreamer) and target (vapplier) in the replication stream and this is done automatically. +For both [VTGate VStreams and VTTablet VReplication streams](../../../concepts/vstream/) we must choose a tablet to serve the role of *source* (vstreamer). This +tablet selection is performed by the internal `TabletPicker` component. -To select the tablets we get a set of viable -- healthy and serving -- candidates for the source and target of the stream: - * **Source**: a random tablet is selected from the viable candidates of the specified types (see [tablet types](./#tablet-types)) - * **Target**: a viable primary tablet is chosen, as we need to do writes that are then replicated within the target shard +{{< info >}} +For VReplication streams a tablet also serves the role of *target* (vapplier). These, however, will always be the primary tablets in the target keyspace as we +need to replicate the streamed writes within the target shard. +{{< /info >}} -### Cell considerations +### Cells -VReplication will only look for tablet pairings within the same cell. If you want to have cross-cell streams then you will need to [create a CellAlias](https://vitess.io/docs/reference/programs/vtctl/cell-aliases/) that contains the list of potential cells and specify that using the `--cell` flag in your VReplication workflow commands. +By default the `TabletPicker` will only look for viable (healthy and serving) source tablets of the specified tablet type(s) within the local cell of the +calling process — the `vtgate` managing the VStream or the target `vttablet` for the VReplication stream — and it will select a random one from the candidate +list. If you want to support cross-cell streams then you will need to specify the list of cells or any +[CellAlias](https://vitess.io/docs/reference/programs/vtctl/cell-aliases/) that contain a list of cells using the `--cells` flag in your VReplication +workflow commands like [`MoveTables`](../movetables/) or the +[`VStreamFlags.Cells`](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/vtgate#VStreamFlags) field in a +[`VStreamRequest`](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/vtgate#VStreamRequest). -### Tablet types +### Tablet Types -The server side default which determines the candidate types made available for potential selection in a stream is set using the [vttablet's `--vreplication_tablet_type` flag](../flags/#vreplication_tablet_type) (default value is `in_order:REPLICA,PRIMARY`). The target tablet will use this when finding the viable source tablet candidates. +#### VReplication +For VReplication, the server side default which determines the candidate types made available for potential selection as the source for a stream is set +using the [`vttablet` `--vreplication_tablet_type` flag](../flags/#vreplication_tablet_type) (default value is `in_order:REPLICA,PRIMARY`). The target tablet +will provide this value to the `TabletPicker` to determine the viable source tablet candidates. You can override this default on the client side using your +workflow command's `--tablet_types` flag. -You can override this on the client side using your workflow command's `--tablet_types` flag. +You can also specify an order of preference for the tablet types using the `in_order:` prefix in both the server and client flags. For example, +`--tablet_types "in_order:REPLICA,PRIMARY"` would cause a replica source tablet to be used whenever possible and a primary tablet would only be used as +a fallback in the event that there are no viable replica tablets available at the time. -In Vitess 12.0 and later you can specify an order of preference for the tablet types using the `in_order:` prefix in both the server and client flags. For example, `--tablet_types "in_order:REPLICA,PRIMARY"` would cause a replica source tablet to be used whenever possible and a primary would only be used as a fallback in the event that there are no viable replicas available at the time. +#### VStream +For a VStream there is no default tablet type. You must specify an individual tablet type using the +[`VStreamRequest.TabletType`](https://pkg.go.dev/vitess.io/vitess/go/vt/proto/vtgate#VStreamRequest) field. \ No newline at end of file diff --git a/content/en/docs/16.0/reference/vreplication/throttling.md b/content/en/docs/16.0/reference/vreplication/throttling.md index d0f193be1..9b352184e 100644 --- a/content/en/docs/16.0/reference/vreplication/throttling.md +++ b/content/en/docs/16.0/reference/vreplication/throttling.md @@ -5,28 +5,28 @@ weight: 300 ### Introduction -VReplication moves potentially massive amounts of data from one place to another, whether within the same keyspace and shard or across keyspaces. It copies data of entire tables and follows up to apply ongoing changes on those tables by reading the binary logs (aka the changelog). +VReplication moves potentially massive amounts of data from one place to another, whether within the same keyspace and shard or across keyspaces. It copies tables and follows up to apply ongoing changes on those tables by reading the binary logs (aka the changelog). This places load on both the source side (where VReplication reads data from) as well as on target side (where VReplication writes data to). -On the source side, VReplication reads the full content of tables. This typically means loading pages from disk contending for disk IO, and "polluting" the MySQL buffer pool. The operation competes with normal production traffic for both IO and memory resources. If the source is a replica, the operation may lead to replication lag. If the source is a primary, this may lead to write contention. +On the source side, VReplication reads the content of tables. This typically means loading pages from disk contending for disk IO, and "polluting" the MySQL buffer pool. The operation competes with normal production traffic for both IO and memory resources. If the source is a replica, the operation may lead to replication lag. If the source is a primary, this may lead to write contention. On the target side, VReplication writes massive amount of data. If the target server is a primary with replicas, then the replicas may incur replication lag. -To address the above issues, VReplication uses the [tablet throttler](../../features/tablet-throttler/) mechanism to push back both reads and writes. +To help address the above issues, VReplication uses the [tablet throttler](../../features/tablet-throttler/) mechanism to push back both reads and writes. -### Target throttling +### Target Throttling -On the target side, VReplication wishes to consult the overall health of the target shard (there can be multiple shards to a VReplication workflow, and here we discuss the single shard at the end of a single VReplication stream). That shard may serve production traffic unrelated to VReplication. VReplication therefore consults the internal equivalent of `/throttler/check` when writing data to the shard's primary. This checks the replication lag on relevant replicas in the shard. The throttler will push back VReplication writes of both table-copy and changelog events. +On the target side, VReplication wishes to consult the overall health of the target shard (there can be multiple shards to a VReplication workflow, and here we discuss the single shard at the end of a single VReplication stream). That shard may serve production traffic unrelated to VReplication. VReplication therefore consults the internal equivalent of `/throttler/check` when writing data to the shard's primary. This checks the MySQL replication lag on relevant replicas in the shard. The throttler will delay the VReplication writes of both table-copy and changelog events until the shard's replication lag is under the defined threshold (1s by default). -### Source throttling +### Source Throttling On the source side, VReplication only affects the single MySQL server it reads from, and has no impact on the overall shard. VStreamer, the source endpoint of VReplication, consults the equivalent of `/throttler/check-self`, which looks for replication lag on the source host. -As long as `check-self` fails, VStreamer will not read table data, nor will it pull events from the changelog. +As long as `check-self` fails — meaning that the replication lag is not within the defined threshold (1s by default) — VStreamer will not read table data, nor will it pull events from the changelog. -### Impact of throttling +### Impact of Throttling -VReplication throttling is designed to give way to normal production traffic while operating in the background. Production traffic will see less contention. The downside is that VReplication can take longer to operate. Under high load in production VReplication may altogether stall, to resume when the load subsides. +VReplication throttling is designed to give preference to normal production traffic while operating in the background. Production traffic will see less contention. The downside is that VReplication can take longer to operate. Under high load in production VReplication may altogether stall, to resume later when the load subsides. -Throttling will push back VReplication on replication lag. On systems where replication lag is normally high this can bring VReplication down to a grinding halt. In such systems consider configuring `--throttle_threshold` to a value that agrees with your constraints. The default throttling threshold is at `1` second replication lag. +Throttling will push back VReplication on replication lag. On systems where replication lag is normally high this can prevent VReplication from being able to operate normally. In such systems consider configuring `--throttle_threshold` to a value that agrees with your constraints. The default throttling threshold is at `1` second replication lag. diff --git a/content/en/docs/16.0/reference/vreplication/v1/_index.md b/content/en/docs/16.0/reference/vreplication/v1/_index.md deleted file mode 100644 index 2e05bf309..000000000 --- a/content/en/docs/16.0/reference/vreplication/v1/_index.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -title: VReplication V1 Commands -description: v1 MoveTables and Reshard vtctld commands (**Deprecated**) -weight: 1000 ---- - -{{< info >}} -Starting with Vitess 11.0 you should use the [VReplication v2 commands](../../vreplication) -{{< /info >}} diff --git a/content/en/docs/16.0/reference/vreplication/v1/dropsources.md b/content/en/docs/16.0/reference/vreplication/v1/dropsources.md deleted file mode 100644 index 01c896c96..000000000 --- a/content/en/docs/16.0/reference/vreplication/v1/dropsources.md +++ /dev/null @@ -1,62 +0,0 @@ ---- -title: DropSources -description: Cleans up after a MoveTables and Reshard workflow -weight: 60 ---- - -{{< info >}} -Starting with Vitess 11.0 you should use the [VReplication v2 commands](../../complete) -{{< /info >}} - -### Command - -``` -DropSources -- [--dry_run] [--rename_tables] [--keep_data] -``` - -### Description - -Once SwitchWrites has been run DropSources cleans up the source resources by deleting the -source tables for a MoveTables workflow or source shards for a Reshard workflow. It also -cleans up other artifacts of the workflow, deleting forward and reverse replication streams and -blacklisted tables. - -***Warning***: This command actually deletes data. We recommend that you run this -with the --dry_run parameter first and reads its output so that you know which actions will be performed. - -### Parameters - -#### --rename_tables -**optional**\ -**default** all - -
-Only applies for a MoveTables workflow. Instead of deleting the tables in the source it renames them -using the template _<tableName>_old, the same scheme followed by pt-osc -
- -#### --dry-run -**optional**\ -**default** false - -
-You can do a dry run where no actual action is taken but the command logs all the actions that would be taken -by SwitchReads. -
- -#### --keep_data -**optional**\ -**default** false - -
- -Usually, the source data (tables or shards) are deleted by Complete. If this flag is used, for MoveTables, source tables will not be deleted, for Reshard, source shards will not be dropped. - -
- -#### keyspace.workflow -**mandatory** - -
-Name of target keyspace and the associated workflow to run VDiff on. -
diff --git a/content/en/docs/16.0/reference/vreplication/v1/movetables.md b/content/en/docs/16.0/reference/vreplication/v1/movetables.md deleted file mode 100644 index 4bcb0d743..000000000 --- a/content/en/docs/16.0/reference/vreplication/v1/movetables.md +++ /dev/null @@ -1,224 +0,0 @@ ---- -title: MoveTables -description: Move tables between keyspaces without downtime -weight: 10 ---- - -{{< info >}} -Starting with Vitess 11.0 you should use the [VReplication v2 commands](../../movetables) -{{< /info >}} - -### Command - -``` -MoveTables -- --v1 [--cells=] [--tablet_types=] --workflow= - [--all] [--exclude=] [--auto_start] [--stop_after_copy] [--on-ddl=] - -``` - -### Description - -MoveTables is used to start a workflow to move one or more tables from an external database or an existing Vitess keyspace into a new Vitess keyspace. -The target keyspace can be unsharded or sharded. - -MoveTables is used typically for migrating data into Vitess or to implement vertical sharding. You might use the former when you -first start using Vitess and the latter if you want to distribute your load across servers. - -### Parameters - -#### --cells -**optional**\ -**default** local cell - -
- -A comma-separated list of cell names or cell aliases. This list is used by VReplication to determine which -cells should be used to pick a tablet for selecting data from the source keyspace.

- -###### Uses - -* Improve performance by using picking a tablet in cells in network proximity with the target -* To reduce bandwidth costs by skipping cells that are in different availability zones -* Select cells where replica lags are lower -
- -#### --tablet_types -**optional**\ -**default** replica - -
- -A comma-separated list of tablet types that are used while picking a tablet for sourcing data. -One or more from PRIMARY, REPLICA, RDONLY.

- -###### Uses - -* To reduce the load on PRIMARY tablets by using REPLICAs or RDONLYs -* Reducing lags by pointing to PRIMARY -
- -#### workflow -**mandatory** -
- -Unique name for the MoveTables-initiated workflow, used in later commands to refer back to this workflow - -
- -#### source_keyspace -**mandatory** -
- -Name of existing keyspace that contains the tables to be moved - -
- -#### target_keyspace -**mandatory** -
- -Name of existing keyspace to which the tables will be moved - -
- -#### table_specs -**optional** one of `table_specs` or `--all` needs to be specified -
- -_Either_ - -* a comma-separated list of tables - * if target keyspace is unsharded OR - * if target keyspace is sharded AND the tables being moved are already defined in the target's vschema - - Example: `MoveTables -workflow=commerce2customer commerce customer customer,corder` - -_Or_ - -* the JSON table section of the vschema for associated tables - * if target keyspace is sharded AND - * tables being moved are not yet present in the target's vschema - - Example: `MoveTables -- --workflow=commerce2customer commerce customer '{"t1":{"column_vindexes": [{"column": "id", "name": "hash"}]}}}'` - -
- -#### --all - -**optional** cannot specify `table_specs` if `--all` is specified -
- -Move all tables from the source keyspace. - -
- -#### --exclude - -**optional** only applies if `--all` is specified -
- -If moving all tables, specifies tables to be skipped. - -
- -#### --auto_start - -**optional** -**default** true - -
- -Normally the workflow starts immediately after it is created. If this flag is set -to false then the workflow is in a Stopped state until you explicitly start it. - -###### Uses -* allows updating the rows in `_vt.vreplication` after MoveTables has setup the -streams. For example, you can add some filters to specific tables or change the -projection clause to modify the values on the target. This -provides an easier way to create simpler Materialize workflows by first using -MoveTables with auto_start false, updating the BinlogSource as required by your -Materialize and then start the workflow. -* changing the `copy_state` and/or `pos` values to restart a broken MoveTables workflow -from a specific point of time. - -
- -#### --stop_after_copy - -**optional** -**default** false - -
- -If set, the workflow will stop once the Copy phase has been completed i.e. once -all tables have been copied and VReplication decides that the lag -is small enough to start replicating, the workflow state will be set to Stopped. - -###### Uses -* If you just want a consistent snapshot of all the tables you can set this flag. The workflow -will stop once the copy is done and you can then mark the workflow as `Complete`d - -
- -#### --on-ddl -**optional**\ -**default** IGNORE - -
- -This flag allows you to specify what to do with DDL SQL statements when they are encountered -in the replication stream from the source. The values can be as follows: - -* `IGNORE`: Ignore all DDLs (this is also the default, if a value for `on-ddl` - is not provided). -* `STOP`: Stop when DDL is encountered. This allows you to make any necessary - changes to the target. Once changes are made, updating the workflow state to - `Running` will cause VReplication to continue from just after the point where - it encountered the DDL. Alternatively you may want to `Cancel` the workflow - and create a new one to fully resync with the source. -* `EXEC`: Apply the DDL, but stop if an error is encountered while applying it. -* `EXEC_IGNORE`: Apply the DDL, but ignore any errors and continue replicating. - -{{< warning >}} -We caution against against using `EXEC` or `EXEC_IGNORE` for the following reasons: - * You may want a different schema on the target. - * You may want to apply the DDL in a different way on the target. - * The DDL may take a long time to apply on the target and may disrupt replication, performance, and query execution (if serving traffic from the target) while it is being applied. -{{< /warning >}} - -
- -### A MoveTables Workflow - -Once you select the set of tables to move from one keyspace to another you need to initiate a VReplication workflow as follows: - -1. Initiate the migration using MoveTables -2. Monitor the workflow using [Workflow](../../workflow) -3. Confirm that data has been copied over correctly using [VDiff](../../vdiff) -4. Start the cutover by routing all reads from your application to those tables using [SwitchReads](../switchreads) -5. Complete the cutover by routing all writes using [SwitchWrites](../switchwrites) -6. Optionally clean up the source tables using [DropSources](../dropsources) - - -### Common use cases for MoveTables - -#### Adopting Vitess - -For those wanting to try out Vitess for the first time, MoveTables provides an easy way to route part of their workload -to Vitess with the ability to migrate back at any time without any risk. You point a vttablet to your existing MySQL installation, -spin up an unsharded Vitess cluster and use a MoveTables workflow to start serving some tables from Vitess. You can also go -further and use a Reshard workflow to experiment with a sharded version of a part of your database. - -See [user guide](../../../../user-guides/configuration-advanced/unmanaged-tablet/#move-legacytable-to-the-commerce-keyspace) for detailed steps - -#### Vertical Sharding - -For existing Vitess users you can easily move one or more tables to another keyspace, either for balancing load or -as preparation for sharding your tables. - -See [user guide](../../../../user-guides/migration/move-tables/) which describes how MoveTables works in the local example provided -in the Vitess repo. - -#### More Reading - -* [MoveTables in practice](../../../../concepts/move-tables/) diff --git a/content/en/docs/16.0/reference/vreplication/v1/reshard.md b/content/en/docs/16.0/reference/vreplication/v1/reshard.md deleted file mode 100644 index 794960417..000000000 --- a/content/en/docs/16.0/reference/vreplication/v1/reshard.md +++ /dev/null @@ -1,124 +0,0 @@ ---- -title: Reshard -description: split or merge shards in a keyspace -weight: 60 ---- - -{{< info >}} -Starting with Vitess 11.0 you should use the [VReplication v2 commands](../../reshard) -{{< /info >}} - -### Command - -``` -Reshard -- --v1 [--cells=] [--tablet_types=] [--skip_schema_copy] - [--auto_start] [--stop_after_copy] [--on-ddl=] - -``` - -### Description - -Reshard support horizontal sharding by letting you change the sharding ranges of your existing keyspace. - -### Parameters - -#### --cells -**optional**\ - -
-Comma separated Cell(s) or CellAlias(es) to replicate from. -
- -#### --tablet_types -**optional**\ -**default** empty - -
-Source Vitess tablet_type, or comma separated list of tablet types, that should be used for choosing source tablet(s) for the reshard. -
- -#### --skip_schema_copy -**optional**\ -**default** false - -
-If true the source schema is copied to the target shards. If false, you need to create the tables -before calling reshard. -
- -#### --auto_start -**optional**\ -**default** true - -
-If false, streams will start in the Stopped state and will need to be explicitly started (default true). -
- -#### --stop_after_copy -**optional**\ -**default** false - -
-Streams will be stopped once the copy phase is completed. -
- -#### --on-ddl -**optional**\ -**default** IGNORE - -
- -This flag allows you to specify what to do with DDL SQL statements when they are encountered -in the replication stream from the source. The values can be as follows: - -* `IGNORE`: Ignore all DDLs (this is also the default, if a value for `on-ddl` - is not provided). -* `STOP`: Stop when DDL is encountered. This allows you to make any necessary - changes to the target. Once changes are made, updating the workflow state to - `Running` will cause VReplication to continue from just after the point where - it encountered the DDL. Alternatively you may want to `Cancel` the workflow - and create a new one to fully resync with the source. -* `EXEC`: Apply the DDL, but stop if an error is encountered while applying it. -* `EXEC_IGNORE`: Apply the DDL, but ignore any errors and continue replicating. - -{{< warning >}} -We caution against against using `EXEC` or `EXEC_IGNORE` for the following reasons: - * You may want a different schema on the target. - * You may want to apply the DDL in a different way on the target. - * The DDL may take a long time to apply on the target and may disrupt replication, performance, and query execution (if serving traffic from the target) while it is being applied. -{{< /warning >}} - -
- -#### keyspace.workflow -**mandatory** - -
-Name of keyspace being sharded and the associated workflow name, used in later commands to refer back to this reshard. -
- -#### source_shards -**mandatory** - -
-Comma separated shard names to reshard from. -
- -#### target_shards -**mandatory** - -
-Comma separated shard names to reshard to. -
- - -### A Reshard Workflow - -Once you decide on the new resharding strategy for a keyspace, you need to initiate a VReplication workflow as follows: - -1. Initiate the migration using Reshard -2. Monitor the workflow using [Workflow](../../workflow) -3. Confirm that data has been copied over correctly using [VDiff](../../vdiff) -4. Start the cutover by routing all reads from your application to those tables using [SwitchReads](../switchreads) -5. Complete the cutover by routing all writes using [SwitchWrites](../switchwrites) -6. Optionally cleanup the source tables using [DropSources](../dropsources) diff --git a/content/en/docs/16.0/reference/vreplication/v1/switchreads.md b/content/en/docs/16.0/reference/vreplication/v1/switchreads.md deleted file mode 100644 index 4acd51c60..000000000 --- a/content/en/docs/16.0/reference/vreplication/v1/switchreads.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -title: SwitchReads -description: Route reads to target keyspace -weight: 30 -aliases: ['/docs/reference/vreplication/switchreads/'] ---- - -{{< info >}} -Starting with Vitess 11.0 you should use the [SwitchTraffic VReplication v2 commands](../../switchtraffic) -{{< /info >}} - -### Command - -``` -SwitchReads -- [--cells=c1,c2,...] [--reverse] --tablet_types={replica|rdonly} [--dry-run] -``` - -### Description - -SwitchReads is used to switch reads for tables in a MoveTables workflow or for entire keyspace -to the target keyspace in a Reshard workflow. - -### Parameters - -#### --cells -**optional**\ -**default** all - -
-Comma separated list of cells or cell aliases in which reads should be switched in the target keyspace -
- -#### --reverse -**optional**\ -**default** false - -
-When a workflow is setup the routing rules are setup so that reads/writes to the target shards -still go to the source shard since the target is not yet setup. If SwitchReads is called without --reverse then the routing rules for the target keyspace are setup to actually use it. It is assumed -that the workflow was successful and user is ready to use the target keyspace now. - -However if, for any reason, we want to abort this workflow using the --reverse flag deletes the -rules that were setup and vtgate will route the queries to this table to the the source table. -There is no way to reverse the use of the -reverse flag other than by recreating the routing rules -again using the vtctl ApplyRoutingRules command. -
- -#### --dry-run -**optional**\ -**default** false - -
-You can do a dry run where no actual action is taken but the command logs all the actions that would be taken -by SwitchReads. -
- -#### --tablet_types -**mandatory** - -
-Tablet types to switch one or both or rdonly/replica (default "rdonly,replica") -
- -#### --tablet_type -**DEPRECATED** - -
-On which type of tablets should be reads be switched to the target keyspace. One of replica or rdonly. rdonly -tables should be switched first before replica tablets. -
- -#### keyspace.workflow -**mandatory** - -
-Name of target keyspace and the associated workflow to SwitchReads for. -
diff --git a/content/en/docs/16.0/reference/vreplication/v1/switchwrites.md b/content/en/docs/16.0/reference/vreplication/v1/switchwrites.md deleted file mode 100644 index 02ce961ed..000000000 --- a/content/en/docs/16.0/reference/vreplication/v1/switchwrites.md +++ /dev/null @@ -1,95 +0,0 @@ ---- -title: SwitchWrites -description: Route writes to target keyspace in a vreplication workflow -weight: 50 -aliases: ['/docs/reference/vreplication/switchwrites/'] ---- - -{{< info >}} -Starting with Vitess 11.0 you should use the [SwitchTraffic VReplication v2 commands](../../switchtraffic) -{{< /info >}} - -### Command - -``` -SwitchWrites -- [--timeout=30s] [--cancel] [--reverse] [--reverse_replication=false] --tablet_types=primary [--dry-run] -``` - -### Description - -SwitchWrites is used to switch writes for tables in a MoveTables workflow or for entire keyspace in the -Reshard workflow away from the primary in the source keyspace to the primary in the target keyspace - -### Parameters - -#### --timeout -**optional**\ -**default** 30s - -
-Specifies the maximum time to wait, in seconds, for vreplication to catch up on primary migrations. The migration will be cancelled on a timeout. -
- - -#### --filtered_replication_wait_time -**DEPRECATED**\ -**default** 30s - -
-SwitchWrites first stops writes on the source primary and waits for the replication to the target to -catchup with the point where the writes were stopped. If the wait time is longer than filtered_replication_wait_time -the command will error out. -For setups with high write qps you may need to increase this value. -
- -#### --cancel -**optional**\ -**default** false - -
-If a previous SwitchWrites returned with an error you can restart it by running the command again (after fixing -the issue that caused the failure) or the SwitchWrites can be canceled using this parameter. Only the SwitchWrites -is cancelled: the workflow is set to Running so that replication continues. -
- -#### --reverse -**optional**\ -**default** false - -
-Reverse a previous SwitchWrites serve from source -
- -#### --reverse_replication -**optional**\ -**default** true - -
-SwitchWrites, by default, starts a reverse replication stream with the current target as the source, replicating -back to the original source. This enables a quick and simple rollback. This reverse workflow name is that -of the original workflow concatenated with \_reverse. -
- -#### --tablet_types -**mandatory** - -
-Tablet types to switch traffic for. For writes that will only be primary tablets. -Note: For tablet type rdonly you must [SwitchReads](../switchreads) before you can SwitchWrites -
- -#### --dry-run -**optional**\ -**default** false - -
-You can do a dry run where no actual action is taken but the command logs all the actions that would be taken -by SwitchWrites. -
- -#### keyspace.workflow -**mandatory** - -
-Name of target keyspace and the associated workflow to SwitchWrites for. -
diff --git a/content/en/docs/16.0/reference/vreplication/vdiff.md b/content/en/docs/16.0/reference/vreplication/vdiff.md index 38c6e2dca..eed6b8dc3 100644 --- a/content/en/docs/16.0/reference/vreplication/vdiff.md +++ b/content/en/docs/16.0/reference/vreplication/vdiff.md @@ -47,7 +47,7 @@ VDiff will choose a tablet from this cell to diff the target tables with the sou
A comma separated list of tablet types that are used while picking a tablet for sourcing data. -One or more from PRIMARY, REPLICA, RDONLY.

+One or more from PRIMARY, REPLICA, RDONLY.
#### --filtered_replication_wait_time @@ -56,7 +56,7 @@ One or more from PRIMARY, REPLICA, RDONLY.

VDiff finds the current position of the source primary and then waits for the target replication to reach -that position for `--filtered_replication_wait_time`. If the target is much behind the source or if there is +that position for --filtered_replication_wait_time. If the target is much behind the source or if there is a high write qps on the source then this time will need to be increased.
@@ -142,9 +142,9 @@ Summary for corder: {ProcessedRows:10 MatchingRows:10 MismatchedRows:0 ExtraRows Summary for customer: {ProcessedRows:11 MatchingRows:11 MismatchedRows:0 ExtraRowsSource:0 ExtraRowsTarget:0} ``` -### Using VDiff with huge tables +### Using VDiff With Huge Tables -Currently VDiff runs within vtctd. Each VDiff will stream rows from all sources and targets and then compare them row by row after assembling the rows in order. Since there are no database transactions, VDiff will run much faster than the actual workflow. However, for huge tables (billions of rows or terabytes in size) this can take several hours or even days depending on the number of rows, row composition, server configurations and the topology of the cluster. If your sources and/or targets are across multiple cells, for example, this can slow down the VDiff considerably. +Currently VDiff runs within vtctld. Each VDiff will stream rows from all sources and targets and then compare them row by row after assembling the rows in order. Since there are no database transactions, VDiff will run much faster than the actual workflow. However, for huge tables (billions of rows or terabytes in size) this can take several hours or even days depending on the number of rows, row composition, server configurations and the topology of the cluster. If your sources and/or targets are across multiple cells, for example, this can slow down the VDiff considerably. Actual VDiff speeds are of course dependent on several factors in your cluster. But as a reference, we have seen VDiffs run as fast as 400mrph (million rows per hour) (~9B rows/day) for tables with short rows, or as slow as 60mrph (~1.5B rows/day), for tables with larger width and complex columns. diff --git a/content/en/docs/16.0/reference/vreplication/vdiff2.md b/content/en/docs/16.0/reference/vreplication/vdiff2.md index 1ffcaaceb..932a7c5e5 100644 --- a/content/en/docs/16.0/reference/vreplication/vdiff2.md +++ b/content/en/docs/16.0/reference/vreplication/vdiff2.md @@ -19,7 +19,7 @@ For additional details, please see the [RFC](https://github.com/vitessio/vitess/ VDiff2 takes different sub-commands or actions similar to how the [`MoveTables`](../movetables/)/[`Reshard`](../reshard/) commands work. The first argument is the <keyspace.workflow> followed by an <action>. The following actions are supported: -#### Start a new VDiff +#### Start a New VDiff These take the same parameters as VDiff1 and schedule VDiff to run on the primary tablet of each target shard to verify the subset of data that will live on the given shard. Please note that if you do not specify a sub-command or action then `create` is assumed (this eases the transition from VDiff1 to VDiff2). If you do not pass a specific UUID then one will be generated. @@ -38,7 +38,7 @@ $ vtctlclient --server=localhost:15999 VDiff -- --v2 customer.commerce2customer VDiff bf9dfc5f-e5e6-11ec-823d-0aa62e50dd24 scheduled on target shards, use show to view progress ``` -#### Resume a previous VDiff +#### Resume a Previous VDiff The `resume` action allows you to resume a previously completed VDiff, picking up where it left off and comparing the records where the Primary Key column(s) are greater than the last record processed — with the progress and other status information saved when the run ends. This allows you to do approximate rolling or differential VDiffs (e.g. done after MoveTables finishes the initial copy phase and then again just before SwitchTraffic). @@ -60,7 +60,7 @@ VDiff 4c664dc2-eba9-11ec-9ef7-920702940ee0 resumed on target shards, use show to We cannot guarantee accurate results for `resume` when different collations are used for a table between the source and target keyspaces (more details can be seen [here](https://github.com/vitessio/vitess/pull/10497)). {{< /warning >}} -#### Show progress/status of a VDiff +#### Show Progress/Status of a VDiff ``` VDiff -- --v2 show { | last | all} @@ -133,7 +133,7 @@ $ vtctlclient --server=localhost:15999 VDiff -- --v2 --format=json customer.com Attempting to `stop` a VDiff that is already completed is a no-op. {{< /info >}} -#### Delete VDiff results +#### Delete VDiff Results ``` VDiff -- --v2 delete { | all} diff --git a/content/en/docs/16.0/reference/vreplication/vexec.md b/content/en/docs/16.0/reference/vreplication/vexec.md index 4a7fcb621..dd874bfd0 100644 --- a/content/en/docs/16.0/reference/vreplication/vexec.md +++ b/content/en/docs/16.0/reference/vreplication/vexec.md @@ -12,22 +12,23 @@ VExec -- [--dry_run] ### Description +{{< warning >}} Deprecated in version 12.0. +{{}} VExec is a wrapper over [VReplicationExec](../vreplicationexec). Given a workflow it executes the provided query on all primary tablets in the target keyspace that participate -in the workflow. Internally it calls VReplicationExec for running the query. +in the workflow. Internally it calls `VReplicationExec` for running the query. ### Parameters -#### --dry-run +#### --dry_run **optional**\ **default** false
You can do a dry run where no actual action is taken but the command logs the queries and the tablets on which the query would be run. -by VExec.
#### keyspace.workflow diff --git a/content/en/docs/16.0/reference/vreplication/vreplicationexec.md b/content/en/docs/16.0/reference/vreplication/vreplicationexec.md index e13255309..75e7500d6 100644 --- a/content/en/docs/16.0/reference/vreplication/vreplicationexec.md +++ b/content/en/docs/16.0/reference/vreplication/vreplicationexec.md @@ -7,12 +7,12 @@ weight: 70 ### Command ``` -VReplicationExec -- [--json] +VReplicationExec -- [--json] ``` ### Description -The VReplicationExec command is used to view or manage vreplication streams. More details are [here](../vreplication#exec). You would typically use one of the higher-level commands like the [WorkFlow](../workflow) command accomplish the same task. +The `VReplicationExec` command is used to view or manage vreplication streams. You would typically use one of the higher-level commands like the [WorkFlow](../workflow) command accomplish the same task. ### Parameters @@ -23,12 +23,12 @@ The VReplicationExec command is used to view or manage vreplication streams. Mor The output of the command is json formatted: to be readable by scripts. -#### tablet alias +#### tablet_alias **mandatory**
Id of the target tablet on which to run the sql query, specified using the vitess tablet id format -cell-uid (see example below). +<cell>-<uid> (see example below).
#### query diff --git a/content/en/docs/16.0/reference/vreplication/workflow.md b/content/en/docs/16.0/reference/vreplication/workflow.md index bc5a9dfc6..0519117d7 100644 --- a/content/en/docs/16.0/reference/vreplication/workflow.md +++ b/content/en/docs/16.0/reference/vreplication/workflow.md @@ -17,7 +17,7 @@ actually specifying a query to VReplication. ### Parameters -#### --dry-run +#### --dry_run **optional**\ **default** false @@ -36,15 +36,15 @@ Name of target keyspace and the associated workflow to take action on. **mandatory**
-action is one of +The Action is one of: * **stop**: sets the state of the workflow to Stopped: no further vreplication will happen until workflow is restarted -* **start**: restarts a Stopped workflows -* **delete**: removes the entries for this workflow in \_vt.vreplication +* **start**: restarts a Stopped workflow +* **delete**: removes the entries for this workflow in the `_vt.vreplication` table * **show**: returns a JSON object with details about the associated shards and also with all the columns - from the \_vt.vreplication table + from the `_vt.vreplication` table * **listall**: returns a comma separated list of all *running* workflows in a keyspace -* **tags**: a comma-separated list of key:value pairs that are used to tag the tablet +* **tags**: a comma-separated list of key:value pairs that are used to tag the workflow
#### Example diff --git a/content/en/docs/16.0/user-guides/migration/cutover.md b/content/en/docs/16.0/user-guides/migration/cutover.md index c4c8d9c85..dc1a9ffb0 100644 --- a/content/en/docs/16.0/user-guides/migration/cutover.md +++ b/content/en/docs/16.0/user-guides/migration/cutover.md @@ -32,8 +32,6 @@ vtctldclient --server vtctld.host:15999 GetRoutingRules > /var/tmp/routingrules. ## Start MoveTables -### Using --v2 commands - To begin MoveTables run the following command: ```sh @@ -54,107 +52,6 @@ vtctlclient --server vtctld.host:15999 Workflow -- targetkeyspace.workflowname s ``` The output for `Workflow ... show` is shown below in the --v1 commands. -### Using --v1 commands - -To begin MoveTables run the following command: - -```sh -vtctlclient --server vtctld.host:15999 MoveTables --workflow=workflowname sourcekeyspace targetkeyspace table1,table2,table3 -``` - -You can then monitor the workflow status: - -```sh -vtctlclient --server vtctld.host:15999 -- Workflow targetkeyspace listall -Following workflow(s) found in keyspace targetkeyspace: workflowname -vtctlclient --server vtctld.host:15999 -- Workflow targetkeyspace.workflowname show - { - "Workflow": "workflowname", - "SourceLocation": { - "Keyspace": "sourcekeyspace", - "Shards": [ - "0" - ] - }, - "TargetLocation": { - "Keyspace": "targetkeyspace", - "Shards": [ - "0" - ] - }, - "MaxVReplicationLag": 0, - "ShardStatuses": { - "0/zone1-0000000900": { - "MasterReplicationStatuses": [ - { - "Shard": "0", - "Tablet": "zone1-0000000900", - "ID": 1, - "Bls": { - "keyspace": "sourcekeyspace", - "shard": "0", - "filter": { - "rules": [ - { - "match": "table1", - "filter": "select * from table1" - }, - { - "match": "table2", - "filter": "select * from table2" - }, - { - "match": "table3", - "filter": "select * from table3" - } - ] - } - }, - "Pos": "MySQL56/82e3ebcd-d2bc-11eb-821e-00259085bb84:1-33", - "StopPos": "", - "State": "Running", - "DBName": "vt_targetkeyspace", - "TransactionTimestamp": 0, - "TimeUpdated": 1624299797, - "Message": "", - "CopyState": null - } - ], - "TabletControls": null, - "MasterIsServing": true - } - } -} -``` - -### Potential errors or issues when starting MoveTables - -1. Syntax or other issue where MoveTables doesn’t start - -- Use `Workflow .. listall` to verify that MoveTables did not start -- Check the information in -h to see what might need to be adjusted -- Fix the syntax issue and re-run the MoveTables command - -2. MoveTables starts but does not do what was expected - -- Use `Workflow .. listall` and `Workflow .. show` as described above to determine what happened -- You may need to collect vttablet log information on the source(s) and/or target(s) to diagnose some issues - -For example, if you set your `--vreplication_tablet_type` for your vttablets to RDONLY and you are not passing an override `--tablet_types` for MoveTables. - -The result will be that the MoveTables vreplication streams will only use RDONLY tablet types as their source. -If no tablets of this type are available in the same cell as the target keyspace’s primary tablet, the vreplication streams will never start and will loop while searching for eligible tablets to copy from. -You would only be able to diagnose this problem by looking at the vttablet logs for the target keyspace primary tablet. - -- In most cases, it would be appropriate to stop and delete the workflow using the following: - -```sh -vtctlclient --server vtctld.host:15999 Workflow -- targetkeyspace.workflowname stop -vtctlclient --server vtctld.host:15999 Workflow -- targetkeyspace.workflowname delete -``` - -- Then fix underlying issue and re-run the MoveTables command - ## Confirm MoveTables has completed You can tell if the MoveTables workflow is done copying by inspecting workflow “State” and “CopyState” fields. @@ -168,9 +65,9 @@ Depending on the size of the table(s), the VDiff process may need increased time - Increase the `--filtered_replication_wait_time` parameter for VDiff as the default is 30 seconds. You many need to increase this to hours on large and/or busy tables. {{< info >}} -Note that running VDiff via vtctld can lead to vtctld consuming significantly more memory than usual. -We've found this to be around 1 GB plus, instead of the a few hundred MB that it normally uses. -If you run your vtctld in a memory limited container, you may want to take this into account. +Note that running VDiff via vtctld can lead to vtctld consuming significantly more memory than usual. +We've found this to be around 1 GB plus, instead of the a few hundred MB that it normally uses. +If you run your vtctld in a memory limited container, you may want to take this into account. Similarly, while a VDiff is in progress, vtctld will consume significantly more CPU than usual. {{< /info >}} @@ -281,9 +178,6 @@ This will result in you dropping your source/original table. ## Begin your cutover -### When using --v2 commands - -With MoveTables v2 instead of performing SwitchReads/Writes you will use MoveTables SwitchTraffic. It is recommended you first run SwitchTraffic with --dry_run so you understand what actions are going to be taken before actually taking them. `--dry_run` needs to be added before SwitchTraffic along with any other [SwitchTraffic parameters](../../../reference/vreplication/switchtraffic) you want to pass. Reads and writes no longer need to be switched in specific order, but both will need to be completed to run MoveTables Complete. @@ -330,155 +224,8 @@ Unlock keyspace sourcekeyspace After you have tried the above command(s) with `--dry_run` remove just that flag to then actually run the command. -### When using --v1 commands - -1s. Depending on what `--tablet_types` you are using, you will use one of the two following commands: - -- RDONLY: - -``` -vtctlclient --server vtctld.host:1599- SwitchReads -- --tablet_types=rdonly targetkeyspace.workflowname -``` - -- REPLICA: - -``` -vtctlclient --server vtctld.host:1599- SwitchReads -- --tablet_types=replica targetkeyspace.workflowname -``` - -The output of these commands will look similar to the following: - -```sh -$ vtctlclient --server localhost:15999 SwitchReads -- --tablet_types=rdonly --reverse --dry_run targetkeyspace.workflowname -Dry Run results for SwitchReads run at 02 Jan 06 15:04 MST -Parameters: --tablet_types=rdonly --reverse --dry_run targetkeyspace.workflowname - -Lock keyspace sourcekeyspace -Switch reads for tables [t1] to keyspace sourcekeyspace for tablet types [RDONLY] -Routing rules for tables [t1] will be updated -Unlock keyspace sourcekeyspace -``` - - After you have tried the above command(s) with `--dry_run` remove just that flag to then actually run the command. - -2. You will then need to switch your writes using the following command: - -It is recommended you first run SwitchWrites with --dry_run, so you understand what actions are going to be taken before actually taking them. -`--dry_run` needs to be added before targetkeyspace.workflowname. - -``` -$ vtctlclient --server localhost:15999 SwitchWrites -- --dry_run targetkeyspace.workflowname -``` -The output of this command will look similar to the following: - -```sh -$ vtctlclient --server localhost:15999 SwitchWrites -- --dry_run targetkeyspace.workflowname -Dry Run results for SwitchWrites run at 02 Jan 06 15:04 MST -Parameters: --dry_run targetkeyspace.workflowname - -Lock keyspace sourcekeyspace -Lock keyspace targetkeyspace -Stop writes on keyspace sourcekeyspace, tables [t1]: - Keyspace sourcekeyspace, Shard 0 at Position MySQL56/09e0a4fa-0b6f-11ec-8ef5-00259085bb84:1-62 -Wait for VReplication on stopped streams to catchup for upto 30s -Create reverse replication workflow workflowname_reverse -Create journal entries on source databases -Enable writes on keyspace targetkeyspace tables [t1] -Switch routing from keyspace sourcekeyspace to keyspace targetkeyspace -Routing rules for tables [t1] will be updated -SwitchWrites completed, freeze and delete vreplication streams on: - tablet 900 -Start reverse replication streams on: - tablet 100 -Mark vreplication streams frozen on: - Keyspace targetkeyspace, Shard 0, Tablet 900, Workflow workflowname, DbName vt_targetkeyspace -Unlock keyspace targetkeyspace -Unlock keyspace sourcekeyspace -``` - -After you have tried the above command(s) with `--dry_run` remove just that flag to then actually run the command. - -### Potential errors or issues during the cutover - -1. Attempting to cross Vitess cells - -Unless you explicitly specify it, MoveTables flows will not cross Vitess cells. -Also, if you do not have the source tablet_types for the MoveTables workflow, either implied or explicitly specified, in the local cell, the MoveTables workflow will not actually take any action. -Any subsequent SwitchReads or SwitchWrites will fail. - -You need to verify that the MoveTables workflow did complete successfully (i.e. copied all data), either manually or by using VDiff. -At a minimum, you can use the `Workflow … show` command to validate that the workflow has not errored. - -This example shows a healthy, completed MoveTables workflow: - -```sh -$ vtctlclient --server localhost:15999 Workflow -- show targetkeyspace.workflowname -{ - "Workflow": "workflowname", - "SourceLocation": { - "Keyspace": "sourcekeyspace", - "Shards": [ - "0" - ] - }, - "TargetLocation": { - "Keyspace": "targetkeyspace", - "Shards": [ - "0" - ] - }, - "MaxVReplicationLag": 1, - "ShardStatuses": { - "0/zone1-0000000900": { - "MasterReplicationStatuses": [ - { - "Shard": "0", - "Tablet": "zone1-0000000900", - "ID": 1, - "Bls": { - "keyspace": "sourcekeyspace", - "shard": "0", - "filter": { - "rules": [ - { - "match": "t1", - "filter": "select * from t1 where in_keyrange(c1, 'targetkeyspace.xxhash', '-')" - } - ] - } - }, - "Pos": "c403da7f-05ec-11ec-94bd-00259085bb84:1-58", - "StopPos": "", - "State": "Running", - "DBName": "vt_targetkeyspace", - "TransactionTimestamp": 0, - "TimeUpdated": 1629928368, - "Message": "", - "CopyState": null - } - ], - "TabletControls": null, - "MasterIsServing": true - } - } -} -``` - -2. Timeouts - -If the VReplication of changes from the sourcekeyspace to the targetkeyspace are lagging (possibly because of high write rate to the source keyspace), the SwitchWrites operation may fail. -This is because as part of SwitchWrites, traffic is paused, and Vitess then waits a short amount of time for the targetkeyspace shards to catch up to the point(s) where the sourcekeyspace shard(s) were stopped. -If this does not happen within that timeout period, the SwitchWrites will fail. -The default for this wait period is 30 seconds; and can be adjusted upwards or downwards by passing the --timeout flag to the SwitchWrites command. - -{{< info >}} -Note that the above limitation does not apply to SwitchReads, since replica/rdonly instances are assumed to lag anyway. You should accordingly manually validate that the replica lag is within acceptable limits for your purposes before you run SwitchReads. You can do this by inspecting the MaxVReplicationLag value in the Workflow … show output (cf. above). This value represents the maximum lag, in seconds, of the underlying streams. -{{< /info >}} - ## Cutover rollback -### When using --v2 commands - MoveTables v2 supports cutover rollbacks via the MoveTables --ReverseTraffic command. ReverseTraffic supports the `--dry_run` flag and we recommend using it to verify what actions ReverseTraffic will take. Then remove --dry_run when you are prepared to actually ReverseTraffic. @@ -510,119 +257,8 @@ vtctlclient --server vtctld.host:15999 MoveTables -- --tablet_types=replica --dr vtctlclient --server vtctld.host:15999 MoveTables -- --tablet_types=primary --dry_run ReverseTraffic targetkeyspace.workflowname ``` -### When using --v1 commands - -#### Writes have not been switched - -{{< warning >}} -Only follow these steps if SwitchWrites has not yet been run. -{{< /warning >}} - -1. Depending on what `--tablet_types` you are using, you will use one of the two following commands: - -- RDONLY: - -``` -vtctlclient --server localhost:15999 SwitchReads -- --tablet_types=rdonly --dry_run sourcekeyspace.workflowname_reverse -``` - -- REPLICA: - -``` -vtctlclient --server localhost:15999 SwitchReads -- --tablet_types=replica --dry_run sourcekeyspace.workflowname_reverse -``` - -#### Writes have been switched: - -{{< warning >}} -Only follow these steps if SwitchWrites has already been run. -{{< /warning >}} - -1. Depending on what `--tablet_types` you are using, you will use one of the two following commands: - -- RDONLY: - -``` -vtctlclient --server vtctld.host:15999 SwitchReads -- --tablet_types=rdonly --reverse targetkeyspace.workflowname -``` - -- REPLICA: - -``` -vtctlclient --server vtctld.host:15999 SwitchReads -- --tablet_types=replica --reverse targetkeyspace.workflowname -``` - -The output of this command will look similar to the following: - -```sh -$ vtctlclient --server localhost:15999 SwitchReads -- --tablet_types=rdonly --dry_run sourcekeyspace.workflowname_reverse -Dry Run results for SwitchReads run at 02 Jan 06 15:04 MST -Parameters: --tablet_types=rdonly --dry_run sourcekeyspace.workflowname_reverse - -Lock keyspace targetkeyspace -Switch reads for tables [t1] to keyspace sourcekeyspace for tablet types [RDONLY] -Routing rules for tables [t1] will be updated -Unlock keyspace targetkeyspace -``` - -```sh -$ vtctlclient --server localhost:15999 SwitchReads -- --tablet_types=replica --dry_run sourcekeyspace.workflowname_reverse -*** SwitchReads is deprecated. Consider using v2 commands instead, see https://vitess.io/docs/reference/vreplication/ *** -Dry Run results for SwitchReads run at 02 Jan 06 15:04 MST -Parameters: --tablet_types=replica --dry_run sourcekeyspace.workflowname_reverse - -Lock keyspace targetkeyspace -Switch reads for tables [t1] to keyspace sourcekeyspace for tablet types [REPLICA] -Routing rules for tables [t1] will be updated -Unlock keyspace targetkeyspace -``` - -2. After the reads are switched you will then need to reverse the writes: - -``` -vtctlclient --server vtctld.host:15999 SwitchWrites -- sourcekeyspace.workflowname_reverse -``` - -The output of this command will look similar to the following: - -```sh -$ vtctlclient --server localhost:15999 SwitchWrites -- --dry_run sourcekeyspace.workflowname_reverse -Dry Run results for SwitchWrites run at 02 Jan 06 15:04 MST -Parameters: --dry_run sourcekeyspace.workflowname_reverse - -Lock keyspace targetkeyspace -Lock keyspace sourcekeyspace -Stop writes on keyspace targetkeyspace, tables [t1]: - Keyspace targetkeyspace, Shard 0 at Position MySQL56/1aa1a03b-0b6f-11ec-a022-00259085bb84:1-133 -Wait for VReplication on stopped streams to catchup for upto 30s -Create reverse replication workflow workflowname -Create journal entries on source databases -Enable writes on keyspace sourcekeyspace tables [t1] -Switch routing from keyspace targetkeyspace to keyspace sourcekeyspace -Routing rules for tables [t1] will be updated -SwitchWrites completed, freeze and delete vreplication streams on: - tablet 100 -Start reverse replication streams on: - tablet 900 -Mark vreplication streams frozen on: - Keyspace sourcekeyspace, Shard 0, Tablet 100, Workflow workflowname_reverse, DbName vt_sourcekeyspace -Unlock keyspace sourcekeyspace -Unlock keyspace targetkeyspace -``` - -### Potential errors or issues during the rollback - -1. Timeouts - -If the reverse workflow cannot keep the sourcekeyspace up to date with the targetkeyspace, which is now receiving the application writes, the reverse SwitchWrites may time out. -If that occurs, you may need to temporarily stop the application writes to allow the sourcekeyspace to catch up to the target keyspace. -You would then retry the SwitchWrites and then resume the application writes. -This should not be an issue if the amount of writes to the keyspaces are similar before and after the switchover. - ## Clean up of the cutover -### When using --v2 commands - #### After a successful cutover If the cutover was successful, the MoveTables Complete command will do the following: @@ -633,7 +269,7 @@ vtctlclient --server vtctld.host:15999 MoveTables -- --dry_run Complete targetke 1. Drop the tables involved in the MoveTables in the original keyspace (sourcekeyspace) 2. Remove the workflows related to the MoveTables operation -3. Clean up the routing rules. Applications pointed to the sourcekeyspace will no longer be transparently redirected to the targetkeyspace +3. Clean up the [routing rules](../../../reference/features/schema-routing-rules/). Applications pointed to the sourcekeyspace will no longer be transparently redirected to the targetkeyspace. #### After a cutover rollback @@ -645,42 +281,4 @@ vtctlclient --server vtctld.host:15999 MoveTables -- Cancel targetkeyspace.workf 1. Drop the tables involved in the MoveTables in the new keyspace (targetkeyspace) 2. Remove the workflows related to the MoveTables operation -3. Clean up the routing rules. Applications pointed to the targetkeyspace will no longer be transparently redirected to the sourcekeyspace - -### When using --v1 commands - -#### After a successful cutover - -If the cutover was successful, the DropSources command will do the following: - -``` -vtctlclient --server vtctld.host:15999 DropSources -- targetkeyspace.workflowname -``` - -1. Drop the tables involved in the MoveTables in the original keyspace (sourcekeyspace) -2. Remove the workflows related to the MoveTables operation -3. Clean up the routing rules. Applications pointed to the sourcekeyspace will no longer be transparently redirected to the targetkeyspace - -#### After a cutover rollback - -If a cutover was rolled back, a DropSources will clean up the targetkeyspace: - -``` -vtctlclient --server vtctld.host:15999 DropSources -- sourcekeyspace.workflowname_reverse -``` - -1. Drop the tables involved in the MoveTables in the new keyspace (targetkeyspace) -2. Remove the workflows related to the MoveTables operation -3. Clean up the routing rules. Applications pointed to the targetkeyspace will no longer be transparently redirected to the sourcekeyspace - -{{< info >}} -Note that when rolling back a switchover, Vitess will attempt to revive the original workflow from sourcekeyspace to targetkeyspace. If the intention is to retry the switchover at some later date, it may not be necessary to delete the original tables/workflow/etc in this fashion. -{{< /info >}} - -### Potential errors or issues during the clean up - -In certain older Vitess versions, those prior to v10.0, there were issues when cleaning the RoutingRules as part of DropSources. -We recommend that you save the RoutingRules before and after each cutover and DropSources step. -This will ensure that they have been appropriately updated. -If not, you will need to update the RoutingRules manually by editing the RoutingRules JSON blob and then applying the new RoutingRules by using [ApplyRoutingRules](../../../reference/features/schema-routing-rules/#applyroutingrules). - +3. Clean up the [routing rules](../../../reference/features/schema-routing-rules/). Applications pointed to the targetkeyspace will no longer be transparently redirected to the sourcekeyspace. \ No newline at end of file