From 0e89324178618c3fbd8d0e01c91de8c4a7b0d550 Mon Sep 17 00:00:00 2001 From: Manan Gupta Date: Wed, 22 Feb 2023 21:37:26 +0530 Subject: [PATCH 1/8] feat: breaking changes should be the first heading Signed-off-by: Manan Gupta --- doc/releasenotes/16_0_0_summary.md | 80 +++++++++++++++--------------- 1 file changed, 40 insertions(+), 40 deletions(-) diff --git a/doc/releasenotes/16_0_0_summary.md b/doc/releasenotes/16_0_0_summary.md index 9725396b574..e1a7ebde169 100644 --- a/doc/releasenotes/16_0_0_summary.md +++ b/doc/releasenotes/16_0_0_summary.md @@ -3,12 +3,6 @@ ### Table of Contents - **[Major Changes](#major-changes)** - - **[VReplication](#vreplication)** - - [VStream Copy Resume](#vstream-copy-resume) - - [VDiff2 GA](#vdiff2-ga) - - **[Tablet throttler](#tablet-throttler)** - - **[Incremental backup and point in time recovery](#inc-backup)** - - **[Replication manager removal and VTOrc becomes mandatory](#repl-manager-removal)** - **[Breaking Changes](#breaking-changes)** - [VTGate Advertised MySQL Version](#advertised-mysql-version) - [Default MySQL version on Docker](#default-mysql-version) @@ -24,6 +18,12 @@ - [Removed flag](#removed-flag) - [`lock-timeout` and `remote_operation_timeout` Changes](#lock-timeout-introduction) - [Normalized labels in the Prometheus Exporter](#normalized-lables) + - **[VReplication](#vreplication)** + - [VStream Copy Resume](#vstream-copy-resume) + - [VDiff2 GA](#vdiff2-ga) + - **[Tablet throttler](#tablet-throttler)** + - **[Incremental backup and point in time recovery](#inc-backup)** + - **[Replication manager removal and VTOrc becomes mandatory](#repl-manager-removal)** - **[New command line flags and behavior](#new-flag)** - [VTGate: Support query timeout --query-timeout](#vtgate-query-timeout) - [VTTablet: VReplication parallel insert workers --vreplication-parallel-insert-workers](#vrepl-parallel-workers) @@ -51,40 +51,6 @@ ## Major Changes -### VReplication - -#### VStream Copy Resume - -In [PR #11103](https://github.com/vitessio/vitess/pull/11103) we introduced the ability to resume a `VTGate` [`VStream` copy operation](https://vitess.io/docs/design-docs/vreplication/vstream/vscopy/). This is useful when a [`VStream` copy operation](https://vitess.io/docs/design-docs/vreplication/vstream/vscopy/) is interrupted due to e.g. a network failure or a server restart. The `VStream` copy operation can be resumed by specifying each table's last seen primary key value in the `VStream` request. Please see the [`VStream` docs](https://vitess.io/docs/16.0/reference/vreplication/vstream/) for more details. - -#### VDiff2 GA - -We are marking [VDiff v2](https://vitess.io/docs/16.0/reference/vreplication/vdiff2/) as Generally Available or production-ready in v16. We now recommend that you use v2 rather than v1 going forward. V1 will be deprecated and eventually removed in future releases. -If you wish to use v1 for any reason, you will now need to specify the `--v1` flag. - -### Tablet throttler - -The tablet throttler can now be configured dynamically. Configuration is now found in the topo service, and applies to all tablets in all shards and cells of a given keyspace. For backwards compatibility `v16` still supports `vttablet`-based command line flags for throttler ocnfiguration. - -It is possible to enable/disable, to change throttling threshold as well as the throttler query. - -See https://github.com/vitessio/vitess/pull/11604 - -### Incremental backup and point in time recovery - -In [PR #11097](https://github.com/vitessio/vitess/pull/11097) we introduced native incremental backup and point in time recovery: - -- It is possible to take an incremental backup, starting with last known (full or incremental) backup, and up to either a specified (GTID) position, or current ("auto") position. -- The backup is done by copying binary logs. The binary logs are rotated as needed. -- It is then possible to restore a backup up to a given point in time (GTID position). This involves finding a restore path consisting of a full backup and zero or more incremental backups, applied up to the given point in time. -- A server restored to a point in time remains in `DRAINED` tablet type, and does not join the replication stream (thus, "frozen" in time). -- It is possible to take incremental backups from different tablets. It is OK to have overlaps in incremental backup contents. The restore process chooses a valid path, and is valid as long as there are no gaps in the backed up binary log content. - -### Replication manager removal and VTOrc becomes mandatory -VTOrc is now a **required** component of Vitess starting from v16. If the users want VTOrc to manage replication, then they must run VTOrc. -Replication manager is removed from vttablets since the responsibility of fixing replication lies entirely with VTOrc now. -The flag `disable-replication-manager` is deprecated and will be removed in a later release. - ### Breaking Changes #### VTGate Advertised MySQL Version @@ -190,6 +156,40 @@ After the upgrade, they should then alter their configuration to also specify `l The Prometheus metrics exporter now properly normalizes _all_ label names into their `snake_case` form, as it is idiomatic for Prometheus metrics. Previously, Vitess instances were emitting inconsistent labels for their metrics, with some of them being `CamelCase` and others being `snake_case`. +### VReplication + +#### VStream Copy Resume + +In [PR #11103](https://github.com/vitessio/vitess/pull/11103) we introduced the ability to resume a `VTGate` [`VStream` copy operation](https://vitess.io/docs/design-docs/vreplication/vstream/vscopy/). This is useful when a [`VStream` copy operation](https://vitess.io/docs/design-docs/vreplication/vstream/vscopy/) is interrupted due to e.g. a network failure or a server restart. The `VStream` copy operation can be resumed by specifying each table's last seen primary key value in the `VStream` request. Please see the [`VStream` docs](https://vitess.io/docs/16.0/reference/vreplication/vstream/) for more details. + +#### VDiff2 GA + +We are marking [VDiff v2](https://vitess.io/docs/16.0/reference/vreplication/vdiff2/) as Generally Available or production-ready in v16. We now recommend that you use v2 rather than v1 going forward. V1 will be deprecated and eventually removed in future releases. +If you wish to use v1 for any reason, you will now need to specify the `--v1` flag. + +### Tablet throttler + +The tablet throttler can now be configured dynamically. Configuration is now found in the topo service, and applies to all tablets in all shards and cells of a given keyspace. For backwards compatibility `v16` still supports `vttablet`-based command line flags for throttler ocnfiguration. + +It is possible to enable/disable, to change throttling threshold as well as the throttler query. + +See https://github.com/vitessio/vitess/pull/11604 + +### Incremental backup and point in time recovery + +In [PR #11097](https://github.com/vitessio/vitess/pull/11097) we introduced native incremental backup and point in time recovery: + +- It is possible to take an incremental backup, starting with last known (full or incremental) backup, and up to either a specified (GTID) position, or current ("auto") position. +- The backup is done by copying binary logs. The binary logs are rotated as needed. +- It is then possible to restore a backup up to a given point in time (GTID position). This involves finding a restore path consisting of a full backup and zero or more incremental backups, applied up to the given point in time. +- A server restored to a point in time remains in `DRAINED` tablet type, and does not join the replication stream (thus, "frozen" in time). +- It is possible to take incremental backups from different tablets. It is OK to have overlaps in incremental backup contents. The restore process chooses a valid path, and is valid as long as there are no gaps in the backed up binary log content. + +### Replication manager removal and VTOrc becomes mandatory +VTOrc is now a **required** component of Vitess starting from v16. If the users want VTOrc to manage replication, then they must run VTOrc. +Replication manager is removed from vttablets since the responsibility of fixing replication lies entirely with VTOrc now. +The flag `disable-replication-manager` is deprecated and will be removed in a later release. + ### New command line flags and behavior #### VTGate: Support query timeout --query-timeout From 344e3c23d7c57c0e92e358cd063f41954c5ee4ca Mon Sep 17 00:00:00 2001 From: Manan Gupta Date: Wed, 22 Feb 2023 21:40:43 +0530 Subject: [PATCH 2/8] feat: reorder the breaking changes in order of user impact Signed-off-by: Manan Gupta --- doc/releasenotes/16_0_0_summary.md | 32 +++++++++++++++--------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/doc/releasenotes/16_0_0_summary.md b/doc/releasenotes/16_0_0_summary.md index e1a7ebde169..b701e1868e0 100644 --- a/doc/releasenotes/16_0_0_summary.md +++ b/doc/releasenotes/16_0_0_summary.md @@ -7,16 +7,16 @@ - [VTGate Advertised MySQL Version](#advertised-mysql-version) - [Default MySQL version on Docker](#default-mysql-version) - [Running Vitess on the Operator](#running-vitess-on-the-operator) + - [`lock-timeout` and `remote_operation_timeout` Changes](#lock-timeout-introduction) + - [Orchestrator Integration Deletion](#orc-integration-removal) - [vtctld UI Removal](#vtcltd-ui-removal) - [vtctld Flag Deprecation & Deletions](#vtctld-flag-deprecations) - - [Orchestrator Integration Deletion](#orc-integration-removal) - [mysqlctl Flags](#mysqlctl-flags) - [Query Serving Errors](#qs-errors) - [Logstats Table and Keyspace removed](#logstats-table-keyspace) - [Removed Stats](#removed-stats) - [Deprecated Stats](#deprecated-stats) - [Removed flag](#removed-flag) - - [`lock-timeout` and `remote_operation_timeout` Changes](#lock-timeout-introduction) - [Normalized labels in the Prometheus Exporter](#normalized-lables) - **[VReplication](#vreplication)** - [VStream Copy Resume](#vstream-copy-resume) @@ -75,6 +75,20 @@ However, if you are running MySQL 8.0 on the vitess-operator, with for instance 5. Remove `innodb_fast_shutdown=0` from your extra cnf in your YAML file. 6. Apply this file. +#### `lock-timeout` and `remote_operation_timeout` Changes + +Earlier, the shard and keyspace locks used to be capped by the `remote_operation_timeout`. This is no longer the case and instead a new flag called `lock-timeout` is introduced. +For backward compatibility, if `lock-timeout` is unspecified and `remote_operation_timeout` flag is provided, then its value will also be used for `lock-timeout` as well. +The default value for `remote_operation_timeout` has also changed from 30 seconds to 15 seconds. The default for the new flag `lock-timeout` is 45 seconds. + +During upgrades, if the users want to preserve the same behaviour as previous releases, then they should provide the `remote_operation_timeout` flag explicitly before upgrading. +After the upgrade, they should then alter their configuration to also specify `lock-timeout` explicitly. + +#### Orchestrator Integration Deletion + +Orchestrator integration in `vttablet` was deprecated in the previous release and is deleted in this release. +Consider using `VTOrc` instead of `Orchestrator`. + #### vtctld web UI Removal In v13, the vtctld UI was deprecated. As of this release, the `web/vtctld2` directory is deleted and the UI will no longer be included in any Vitess images going forward. All build scripts and the Makefile have been updated to reflect this change. @@ -93,11 +107,6 @@ The following deprecated flags have also been removed: - `--workflow_manager_use_election` - `--workflow_manager_disable` -#### Orchestrator Integration Deletion - -Orchestrator integration in `vttablet` was deprecated in the previous release and is deleted in this release. -Consider using `VTOrc` instead of `Orchestrator`. - #### mysqlctl Flags The [`mysqlctl` command-line client](https://vitess.io/docs/16.0/reference/programs/mysqlctl/) had some leftover (ignored) server flags after the [v15 pflag work](https://github.com/vitessio/enhancements/blob/main/veps/vep-4.md). Those unused flags have now been removed. If you are using any of the following flags with `mysqlctl` in your scripts or other tooling, they will need to be removed prior to upgrading to v16: @@ -143,15 +152,6 @@ The stats `QueriesProcessed` and `QueriesRouted` are deprecated in v16. The same The following flag is removed in v16: - `enable_semi_sync` -#### `lock-timeout` and `remote_operation_timeout` Changes - -Earlier, the shard and keyspace locks used to be capped by the `remote_operation_timeout`. This is no longer the case and instead a new flag called `lock-timeout` is introduced. -For backward compatibility, if `lock-timeout` is unspecified and `remote_operation_timeout` flag is provided, then its value will also be used for `lock-timeout` as well. -The default value for `remote_operation_timeout` has also changed from 30 seconds to 15 seconds. The default for the new flag `lock-timeout` is 45 seconds. - -During upgrades, if the users want to preserve the same behaviour as previous releases, then they should provide the `remote_operation_timeout` flag explicitly before upgrading. -After the upgrade, they should then alter their configuration to also specify `lock-timeout` explicitly. - #### Normalized labels in the Prometheus Exporter The Prometheus metrics exporter now properly normalizes _all_ label names into their `snake_case` form, as it is idiomatic for Prometheus metrics. Previously, Vitess instances were emitting inconsistent labels for their metrics, with some of them being `CamelCase` and others being `snake_case`. From 85734b3a47a4d8452f48df55981996e96f66edc0 Mon Sep 17 00:00:00 2001 From: Manan Gupta Date: Wed, 22 Feb 2023 21:42:24 +0530 Subject: [PATCH 3/8] feat: VTOrc becoming mandatory is important Signed-off-by: Manan Gupta --- doc/releasenotes/16_0_0_summary.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/doc/releasenotes/16_0_0_summary.md b/doc/releasenotes/16_0_0_summary.md index b701e1868e0..1992b562bf7 100644 --- a/doc/releasenotes/16_0_0_summary.md +++ b/doc/releasenotes/16_0_0_summary.md @@ -18,12 +18,12 @@ - [Deprecated Stats](#deprecated-stats) - [Removed flag](#removed-flag) - [Normalized labels in the Prometheus Exporter](#normalized-lables) + - **[Replication manager removal and VTOrc becomes mandatory](#repl-manager-removal)** - **[VReplication](#vreplication)** - [VStream Copy Resume](#vstream-copy-resume) - [VDiff2 GA](#vdiff2-ga) - **[Tablet throttler](#tablet-throttler)** - **[Incremental backup and point in time recovery](#inc-backup)** - - **[Replication manager removal and VTOrc becomes mandatory](#repl-manager-removal)** - **[New command line flags and behavior](#new-flag)** - [VTGate: Support query timeout --query-timeout](#vtgate-query-timeout) - [VTTablet: VReplication parallel insert workers --vreplication-parallel-insert-workers](#vrepl-parallel-workers) @@ -156,6 +156,11 @@ The following flag is removed in v16: The Prometheus metrics exporter now properly normalizes _all_ label names into their `snake_case` form, as it is idiomatic for Prometheus metrics. Previously, Vitess instances were emitting inconsistent labels for their metrics, with some of them being `CamelCase` and others being `snake_case`. +### Replication manager removal and VTOrc becomes mandatory +VTOrc is now a **required** component of Vitess starting from v16. If the users want VTOrc to manage replication, then they must run VTOrc. +Replication manager is removed from vttablets since the responsibility of fixing replication lies entirely with VTOrc now. +The flag `disable-replication-manager` is deprecated and will be removed in a later release. + ### VReplication #### VStream Copy Resume @@ -185,11 +190,6 @@ In [PR #11097](https://github.com/vitessio/vitess/pull/11097) we introduced nati - A server restored to a point in time remains in `DRAINED` tablet type, and does not join the replication stream (thus, "frozen" in time). - It is possible to take incremental backups from different tablets. It is OK to have overlaps in incremental backup contents. The restore process chooses a valid path, and is valid as long as there are no gaps in the backed up binary log content. -### Replication manager removal and VTOrc becomes mandatory -VTOrc is now a **required** component of Vitess starting from v16. If the users want VTOrc to manage replication, then they must run VTOrc. -Replication manager is removed from vttablets since the responsibility of fixing replication lies entirely with VTOrc now. -The flag `disable-replication-manager` is deprecated and will be removed in a later release. - ### New command line flags and behavior #### VTGate: Support query timeout --query-timeout From 7bc82e6530b4a937046dc96352f2d168ea063742 Mon Sep 17 00:00:00 2001 From: Florent Poinsard Date: Thu, 23 Feb 2023 10:14:13 +0200 Subject: [PATCH 4/8] Reorganize the flag deletions and deprecation announcements Signed-off-by: Florent Poinsard --- doc/releasenotes/16_0_0_summary.md | 86 +++++++++++++++--------------- 1 file changed, 44 insertions(+), 42 deletions(-) diff --git a/doc/releasenotes/16_0_0_summary.md b/doc/releasenotes/16_0_0_summary.md index 1992b562bf7..50c950bf802 100644 --- a/doc/releasenotes/16_0_0_summary.md +++ b/doc/releasenotes/16_0_0_summary.md @@ -7,16 +7,19 @@ - [VTGate Advertised MySQL Version](#advertised-mysql-version) - [Default MySQL version on Docker](#default-mysql-version) - [Running Vitess on the Operator](#running-vitess-on-the-operator) + - [Flag Deletions and Deprecations](#flag-deletions-and-deprecations) + - [VTCtld](#vtctld-flag-deletions-deprecations) + - [MySQLCtl](#mysqlctl-flag-deletions-deprecations) + - [VTTablet](#vttablet-flag-deletions-deprecations) + - [VTBackup](#vtbackup-flag-deletions-deprecations) + - [VTOrc](#vtorc-flag-deletions-deprecations) - [`lock-timeout` and `remote_operation_timeout` Changes](#lock-timeout-introduction) - [Orchestrator Integration Deletion](#orc-integration-removal) - [vtctld UI Removal](#vtcltd-ui-removal) - - [vtctld Flag Deprecation & Deletions](#vtctld-flag-deprecations) - - [mysqlctl Flags](#mysqlctl-flags) - [Query Serving Errors](#qs-errors) - [Logstats Table and Keyspace removed](#logstats-table-keyspace) - [Removed Stats](#removed-stats) - [Deprecated Stats](#deprecated-stats) - - [Removed flag](#removed-flag) - [Normalized labels in the Prometheus Exporter](#normalized-lables) - **[Replication manager removal and VTOrc becomes mandatory](#repl-manager-removal)** - **[VReplication](#vreplication)** @@ -40,8 +43,6 @@ - [Transaction Isolation Level](#transaction-isolation-level) - [Transaction Access Mode](#transaction-access-mode) - [Support for views](#support-views) - - **[VTOrc](#vtorc)** - - [Flag Deprecations](#flag-deprecations) - **[VTTestServer](#vttestserver)** - [Performance Improvement](#perf-improvement) - **[Minor Changes](#minor-changes)** @@ -75,6 +76,44 @@ However, if you are running MySQL 8.0 on the vitess-operator, with for instance 5. Remove `innodb_fast_shutdown=0` from your extra cnf in your YAML file. 6. Apply this file. +#### Flag Deletions and Deprecations + +##### VTCtld +With the removal of the vtctld UI, the following vtctld flags have been deprecated: +- `--vtctld_show_topology_crud`: This was a flag that controlled the display of CRUD topology actions in the vtctld UI. The UI is removed, so this flag is no longer necessary. + +The following deprecated flags have also been removed: +- `--enable_realtime_stats` +- `--enable_vtctld_ui` +- `--web_dir` +- `--web_dir2` +- `--workflow_manager_init` +- `--workflow_manager_use_election` +- `--workflow_manager_disable` + +##### MySQLCtld + +The [`mysqlctl` command-line client](https://vitess.io/docs/16.0/reference/programs/mysqlctl/) had some leftover (ignored) server flags after the [v15 pflag work](https://github.com/vitessio/enhancements/blob/main/veps/vep-4.md). Those unused flags have now been removed. If you are using any of the following flags with `mysqlctl` in your scripts or other tooling, they will need to be removed prior to upgrading to v16: +`--port --grpc_auth_static_client_creds --grpc_compression --grpc_initial_conn_window_size --grpc_initial_window_size --grpc_keepalive_time --grpc_keepalive_timeout` + +##### VTTablet + +The following flags were removed in v16: +- `--enable_semi_sync` +- `--backup_storage_hook`, use one of the builtin compression algorithms or `--external-compressor` and `--external-decompressor` instead. +- `--init_populate_metadata`, since we have deleted the `local_metadata` and `shard_metadata` sidecar database tables. + +The flag `--disable-replication-manager` is deprecated and will be removed in a later release. + +##### VTBackup + +The VTBackup flag `--backup_storage_hook` has been removed, use one of the builtin compression algorithms or `--external-compressor` and `--external-decompressor` instead. + + +##### VTOrc + +The flag `--lock-shard-timeout` has been deprecated. Please use the newly introduced `--lock-timeout` flag instead. More detail [here](#lock-timeout-introduction). + #### `lock-timeout` and `remote_operation_timeout` Changes Earlier, the shard and keyspace locks used to be capped by the `remote_operation_timeout`. This is no longer the case and instead a new flag called `lock-timeout` is introduced. @@ -94,24 +133,6 @@ In v13, the vtctld UI was deprecated. As of this release, the `web/vtctld2` dire However, the vtctld HTTP API will remain at `{$vtctld_web_port}/api`. -#### vtctld Flag Deprecation & Deletions -With the removal of the vtctld UI, the following vtctld flags have been deprecated: -- `--vtctld_show_topology_crud`: This was a flag that controlled the display of CRUD topology actions in the vtctld UI. The UI is removed, so this flag is no longer necessary. - -The following deprecated flags have also been removed: -- `--enable_realtime_stats` -- `--enable_vtctld_ui` -- `--web_dir` -- `--web_dir2` -- `--workflow_manager_init` -- `--workflow_manager_use_election` -- `--workflow_manager_disable` - -#### mysqlctl Flags - -The [`mysqlctl` command-line client](https://vitess.io/docs/16.0/reference/programs/mysqlctl/) had some leftover (ignored) server flags after the [v15 pflag work](https://github.com/vitessio/enhancements/blob/main/veps/vep-4.md). Those unused flags have now been removed. If you are using any of the following flags with `mysqlctl` in your scripts or other tooling, they will need to be removed prior to upgrading to v16: -`--port --grpc_auth_static_client_creds --grpc_compression --grpc_initial_conn_window_size --grpc_initial_window_size --grpc_keepalive_time --grpc_keepalive_timeout` - #### Query Serving Errors In this release, we are introducing a new way to report errors from Vitess through the query interface. @@ -147,11 +168,6 @@ The stat `QueryRowCounts` is removed in v16. `QueryRowsAffected` and `QueryRowsR The stats `QueriesProcessed` and `QueriesRouted` are deprecated in v16. The same information can be inferred from the stats `QueriesProcessedByTable` and `QueriesRoutedByTable` respectively. These stats will be removed in the next release. -#### Removed flag - -The following flag is removed in v16: -- `enable_semi_sync` - #### Normalized labels in the Prometheus Exporter The Prometheus metrics exporter now properly normalizes _all_ label names into their `snake_case` form, as it is idiomatic for Prometheus metrics. Previously, Vitess instances were emitting inconsistent labels for their metrics, with some of them being `CamelCase` and others being `snake_case`. @@ -329,17 +345,10 @@ is now fixed. The full issue can be found [here](https://github.com/vitessio/vit - `vtctlclient OnlineDDL ... [complete|retry|cancel|cancel-all]` returns empty result on success instead of number of shard affected. -- VTTablet flag `--backup_storage_hook` has been removed, use one of the builtin compression algorithms or `--external-compressor` and `--external-decompressor` instead. - -- vtbackup flag `--backup_storage_hook` has been removed, use one of the builtin compression algorithms or `--external-compressor` and `--external-decompressor` instead. - -- The VTTablet flag `--init_populate_metadata` has been deprecated, since we have deleted the `local_metadata` and `shard_metadata` sidecar database tables. - - The dead legacy Workflow Manager related code was removed in [#12085](https://github.com/vitessio/vitess/pull/12085). This included the following `vtctl` client commands: `WorkflowAction`, `WorkflowCreate`, `WorkflowWait`, `WorkflowStart`, `WorkflowStop`, `WorkflowTree`, `WorkflowDelete`. - VTAdmin's `VTExplain` endpoint has been deprecated. Users can use the new `vexplain` query format instead. The endpoint will be deleted in a future release. - ### MySQL Compatibility #### Transaction Isolation Level @@ -388,13 +397,6 @@ Views are not enabled by default in your Vitess cluster, but they can be turned To read more on how views are implemented you can read the [Views Support RFC](https://github.com/vitessio/vitess/issues/11559). And if you want to learn more on how to use views and its current limitations, you can read the [Views Documentation](https://vitess.io/docs/16.0/reference/compatibility/mysql-compatibility/#views). - -### VTOrc - -#### Flag Deprecations - -The flag `lock-shard-timeout` has been deprecated. Please use the newly introduced `lock-timeout` instead. More detail [here](#lock-timeout-introduction). - ### VTTestServer #### Performance Improvement From 45df1a22e1fcc7f4096162149b2c9e47544e7399 Mon Sep 17 00:00:00 2001 From: Florent Poinsard Date: Thu, 23 Feb 2023 11:04:21 +0200 Subject: [PATCH 5/8] Fix various typo and add vtorc links Signed-off-by: Florent Poinsard --- doc/releasenotes/16_0_0_summary.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/doc/releasenotes/16_0_0_summary.md b/doc/releasenotes/16_0_0_summary.md index 50c950bf802..7a9a781dbb3 100644 --- a/doc/releasenotes/16_0_0_summary.md +++ b/doc/releasenotes/16_0_0_summary.md @@ -103,7 +103,7 @@ The following flags were removed in v16: - `--backup_storage_hook`, use one of the builtin compression algorithms or `--external-compressor` and `--external-decompressor` instead. - `--init_populate_metadata`, since we have deleted the `local_metadata` and `shard_metadata` sidecar database tables. -The flag `--disable-replication-manager` is deprecated and will be removed in a later release. +The flag `--disable-replication-manager` is deprecated and will be removed in a future release. ##### VTBackup @@ -126,7 +126,7 @@ After the upgrade, they should then alter their configuration to also specify `l #### Orchestrator Integration Deletion Orchestrator integration in `vttablet` was deprecated in the previous release and is deleted in this release. -Consider using `VTOrc` instead of `Orchestrator`. +`VTOrc` should be deployed instead. You can read more on [how VTOrc is designed](https://vitess.io/docs/16.0/reference/vtorc/) and on [how to run VTOrc in production](https://vitess.io/docs/16.0/user-guides/configuration-basic/vtorc/). #### vtctld web UI Removal In v13, the vtctld UI was deprecated. As of this release, the `web/vtctld2` directory is deleted and the UI will no longer be included in any Vitess images going forward. All build scripts and the Makefile have been updated to reflect this change. @@ -173,9 +173,9 @@ The stats `QueriesProcessed` and `QueriesRouted` are deprecated in v16. The same The Prometheus metrics exporter now properly normalizes _all_ label names into their `snake_case` form, as it is idiomatic for Prometheus metrics. Previously, Vitess instances were emitting inconsistent labels for their metrics, with some of them being `CamelCase` and others being `snake_case`. ### Replication manager removal and VTOrc becomes mandatory -VTOrc is now a **required** component of Vitess starting from v16. If the users want VTOrc to manage replication, then they must run VTOrc. +VTOrc is now a **required** component of Vitess starting from v16. If the users want Vitess to manage replication, then they must run VTOrc. Replication manager is removed from vttablets since the responsibility of fixing replication lies entirely with VTOrc now. -The flag `disable-replication-manager` is deprecated and will be removed in a later release. +The flag `disable-replication-manager` is deprecated and will be removed in a future release. ### VReplication From 0cda0ca637ee16d13ad052262468af645041467f Mon Sep 17 00:00:00 2001 From: Florent Poinsard Date: Thu, 23 Feb 2023 11:26:45 +0200 Subject: [PATCH 6/8] Upgrading to this release with vitess-operator Signed-off-by: Florent Poinsard --- doc/releasenotes/16_0_0_summary.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/releasenotes/16_0_0_summary.md b/doc/releasenotes/16_0_0_summary.md index 7a9a781dbb3..88479bf201e 100644 --- a/doc/releasenotes/16_0_0_summary.md +++ b/doc/releasenotes/16_0_0_summary.md @@ -6,7 +6,7 @@ - **[Breaking Changes](#breaking-changes)** - [VTGate Advertised MySQL Version](#advertised-mysql-version) - [Default MySQL version on Docker](#default-mysql-version) - - [Running Vitess on the Operator](#running-vitess-on-the-operator) + - [⚠️ Upgrading to this release with vitess-operator](#upgrading-to-this-release-with-vitess-operator) - [Flag Deletions and Deprecations](#flag-deletions-and-deprecations) - [VTCtld](#vtctld-flag-deletions-deprecations) - [MySQLCtl](#mysqlctl-flag-deletions-deprecations) @@ -63,9 +63,9 @@ The users can set the `mysql_server_version` flag to advertise the correct versi The default major MySQL version used by our `vitess/lite:latest` image is going from `5.7` to `8.0`. Additionally, the patch version of MySQL80 has been upgraded from `8.0.23` to `8.0.30`. -#### Running Vitess on the Operator +#### ⚠️Upgrading to this release with vitess-operator -If you are using the vitess-operator and want to remain on MySQL 5.7, we invite you to use the `vitess/lite:v16.0.0-mysql57` Docker Image. +If you are using the vitess-operator and want to remain on MySQL 5.7, **you are required** to use the `vitess/lite:v16.0.0-mysql57` Docker Image, otherwise the `vitess/lite:v16.0.0` image will be on MySQL 80. However, if you are running MySQL 8.0 on the vitess-operator, with for instance `vitess/lite:v15.0.2-mysql80`, considering that we are bumping the patch version of MySQL 80 from `8.0.23` to `8.0.30`, you will have to manually upgrade: From aa0aeb1f8d42bb3135c91c4c989d5098b1219c1a Mon Sep 17 00:00:00 2001 From: Florent Poinsard Date: Thu, 23 Feb 2023 11:27:38 +0200 Subject: [PATCH 7/8] Fix various typo Signed-off-by: Florent Poinsard --- doc/releasenotes/16_0_0_summary.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/releasenotes/16_0_0_summary.md b/doc/releasenotes/16_0_0_summary.md index 88479bf201e..48e82a58f57 100644 --- a/doc/releasenotes/16_0_0_summary.md +++ b/doc/releasenotes/16_0_0_summary.md @@ -117,7 +117,7 @@ The flag `--lock-shard-timeout` has been deprecated. Please use the newly introd #### `lock-timeout` and `remote_operation_timeout` Changes Earlier, the shard and keyspace locks used to be capped by the `remote_operation_timeout`. This is no longer the case and instead a new flag called `lock-timeout` is introduced. -For backward compatibility, if `lock-timeout` is unspecified and `remote_operation_timeout` flag is provided, then its value will also be used for `lock-timeout` as well. +For backward compatibility, if `lock-timeout` is unspecified and `remote_operation_timeout` flag is provided, then its value will also be used for `lock-timeout`. The default value for `remote_operation_timeout` has also changed from 30 seconds to 15 seconds. The default for the new flag `lock-timeout` is 45 seconds. During upgrades, if the users want to preserve the same behaviour as previous releases, then they should provide the `remote_operation_timeout` flag explicitly before upgrading. @@ -185,7 +185,7 @@ In [PR #11103](https://github.com/vitessio/vitess/pull/11103) we introduced the #### VDiff2 GA -We are marking [VDiff v2](https://vitess.io/docs/16.0/reference/vreplication/vdiff2/) as Generally Available or production-ready in v16. We now recommend that you use v2 rather than v1 going forward. V1 will be deprecated and eventually removed in future releases. +We are marking [VDiff v2](https://vitess.io/docs/16.0/reference/vreplication/vdiff2/) as production-ready in v16. We now recommend that you use v2 rather than v1 going forward. V1 will be deprecated and eventually removed in future releases. If you wish to use v1 for any reason, you will now need to specify the `--v1` flag. ### Tablet throttler From f9c953c6b02360bc1a2895baee7beec41f0622d2 Mon Sep 17 00:00:00 2001 From: Florent Poinsard Date: Thu, 23 Feb 2023 13:19:51 +0200 Subject: [PATCH 8/8] Link to PRs in major changes Signed-off-by: Florent Poinsard --- doc/releasenotes/16_0_0_summary.md | 30 ++++++++++++++++-------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/doc/releasenotes/16_0_0_summary.md b/doc/releasenotes/16_0_0_summary.md index 48e82a58f57..95d9c31b57d 100644 --- a/doc/releasenotes/16_0_0_summary.md +++ b/doc/releasenotes/16_0_0_summary.md @@ -56,12 +56,13 @@ #### VTGate Advertised MySQL Version -VTGate now advertises MySQL version 8.0.30. This is a breaking change for clients that rely on the VTGate advertised MySQL version and still use MySQL 5.7. +Since [Pull Request #11989](https://github.com/vitessio/vitess/pull/11989), VTGate advertises MySQL version 8.0.30. This is a breaking change for clients that rely on the VTGate advertised MySQL version and still use MySQL 5.7. The users can set the `mysql_server_version` flag to advertise the correct version. #### Default MySQL version on Docker The default major MySQL version used by our `vitess/lite:latest` image is going from `5.7` to `8.0`. Additionally, the patch version of MySQL80 has been upgraded from `8.0.23` to `8.0.30`. +This change was brought by [Pull Request #12252](https://github.com/vitessio/vitess/pull/12252). #### ⚠️Upgrading to this release with vitess-operator @@ -116,7 +117,7 @@ The flag `--lock-shard-timeout` has been deprecated. Please use the newly introd #### `lock-timeout` and `remote_operation_timeout` Changes -Earlier, the shard and keyspace locks used to be capped by the `remote_operation_timeout`. This is no longer the case and instead a new flag called `lock-timeout` is introduced. +Before the changes made in [Pull Request #11881](https://github.com/vitessio/vitess/pull/11881), the shard and keyspace locks used to be capped by the `remote_operation_timeout`. This is no longer the case and instead a new flag called `lock-timeout` is introduced. For backward compatibility, if `lock-timeout` is unspecified and `remote_operation_timeout` flag is provided, then its value will also be used for `lock-timeout`. The default value for `remote_operation_timeout` has also changed from 30 seconds to 15 seconds. The default for the new flag `lock-timeout` is 45 seconds. @@ -129,13 +130,13 @@ Orchestrator integration in `vttablet` was deprecated in the previous release an `VTOrc` should be deployed instead. You can read more on [how VTOrc is designed](https://vitess.io/docs/16.0/reference/vtorc/) and on [how to run VTOrc in production](https://vitess.io/docs/16.0/user-guides/configuration-basic/vtorc/). #### vtctld web UI Removal -In v13, the vtctld UI was deprecated. As of this release, the `web/vtctld2` directory is deleted and the UI will no longer be included in any Vitess images going forward. All build scripts and the Makefile have been updated to reflect this change. +In v13, the vtctld UI was deprecated. As of this release, the `web/vtctld2` directory is deleted and the UI will no longer be included in any Vitess images going forward. All build scripts and the Makefile have been updated to reflect this change, which was done in [Pull Request #11851](https://github.com/vitessio/vitess/pull/11851) However, the vtctld HTTP API will remain at `{$vtctld_web_port}/api`. #### Query Serving Errors -In this release, we are introducing a new way to report errors from Vitess through the query interface. +In [Pull Request #10738](https://github.com/vitessio/vitess/pull/10738) we are introducing a new way to report errors from Vitess through the query interface. Errors will now have an error code for each error, which will make it easy to search for more information on the issue. For instance, the following error: @@ -158,19 +159,20 @@ If your application is searching for specific errors, you might need to update y #### Logstats Table and Keyspace removed Information about which tables are used is now reported by the field TablesUsed added in v15, that is a string array, listing all tables and which keyspace they are in. -The Table/Keyspace fields were deprecated in v15 and are now removed in the v16 release of Vitess. +The Table/Keyspace fields were deprecated in v15 and are now removed in the v16 release, more information can be found on [Pull Request #12083](https://github.com/vitessio/vitess/pull/12083). #### Removed Stats -The stat `QueryRowCounts` is removed in v16. `QueryRowsAffected` and `QueryRowsReturned` can be used instead to gather the same information. +The stat `QueryRowCounts` is removed in v16 as part of [Pull Request #12083](https://github.com/vitessio/vitess/pull/12083). `QueryRowsAffected` and `QueryRowsReturned` can be used instead to gather the same information. #### Deprecated Stats -The stats `QueriesProcessed` and `QueriesRouted` are deprecated in v16. The same information can be inferred from the stats `QueriesProcessedByTable` and `QueriesRoutedByTable` respectively. These stats will be removed in the next release. +The stats `QueriesProcessed` and `QueriesRouted` are deprecated in v16 as part of [Pull Request #12083](https://github.com/vitessio/vitess/pull/12083). The same information can be inferred from the stats `QueriesProcessedByTable` and `QueriesRoutedByTable` respectively. These stats will be removed in the next release. #### Normalized labels in the Prometheus Exporter The Prometheus metrics exporter now properly normalizes _all_ label names into their `snake_case` form, as it is idiomatic for Prometheus metrics. Previously, Vitess instances were emitting inconsistent labels for their metrics, with some of them being `CamelCase` and others being `snake_case`. +More information about this change can be found on [Pull Request #12057](https://github.com/vitessio/vitess/pull/12057). ### Replication manager removal and VTOrc becomes mandatory VTOrc is now a **required** component of Vitess starting from v16. If the users want Vitess to manage replication, then they must run VTOrc. @@ -181,7 +183,7 @@ The flag `disable-replication-manager` is deprecated and will be removed in a fu #### VStream Copy Resume -In [PR #11103](https://github.com/vitessio/vitess/pull/11103) we introduced the ability to resume a `VTGate` [`VStream` copy operation](https://vitess.io/docs/design-docs/vreplication/vstream/vscopy/). This is useful when a [`VStream` copy operation](https://vitess.io/docs/design-docs/vreplication/vstream/vscopy/) is interrupted due to e.g. a network failure or a server restart. The `VStream` copy operation can be resumed by specifying each table's last seen primary key value in the `VStream` request. Please see the [`VStream` docs](https://vitess.io/docs/16.0/reference/vreplication/vstream/) for more details. +In [Pull Request #11103](https://github.com/vitessio/vitess/pull/11103) we introduced the ability to resume a `VTGate` [`VStream` copy operation](https://vitess.io/docs/16.0/reference/vreplication/vstream/). This is useful when a [`VStream` copy operation](https://vitess.io/docs/16.0/reference/vreplication/vstream/) is interrupted due to e.g. a network failure or a server restart. The `VStream` copy operation can be resumed by specifying each table's last seen primary key value in the `VStream` request. Please see the [`VStream` docs](https://vitess.io/docs/16.0/reference/vreplication/vstream/) for more details. #### VDiff2 GA @@ -194,11 +196,11 @@ The tablet throttler can now be configured dynamically. Configuration is now fou It is possible to enable/disable, to change throttling threshold as well as the throttler query. -See https://github.com/vitessio/vitess/pull/11604 +More information on [Pull Request #11604](https://github.com/vitessio/vitess/pull/11604). ### Incremental backup and point in time recovery -In [PR #11097](https://github.com/vitessio/vitess/pull/11097) we introduced native incremental backup and point in time recovery: +In [Pull Request #11097](https://github.com/vitessio/vitess/pull/11097) we introduced native incremental backup and point in time recovery: - It is possible to take an incremental backup, starting with last known (full or incremental) backup, and up to either a specified (GTID) position, or current ("auto") position. - The backup is done by copying binary logs. The binary logs are rotated as needed. @@ -353,7 +355,7 @@ is now fixed. The full issue can be found [here](https://github.com/vitessio/vit #### Transaction Isolation Level -Support added for `set [session] transaction isolation level ` +In [Pull Request #11704](https://github.com/vitessio/vitess/pull/11704) we are adding support for `set [session] transaction isolation level ` ```sql transaction_characteristic: { @@ -374,7 +376,7 @@ This will be applied to any shard where the session will open a transaction. #### Transaction Access Mode -Support added for `start transaction` with transaction characteristic. +In [Pull Request #11704](https://github.com/vitessio/vitess/pull/11704) we are adding support for `start transaction` with transaction characteristic. ```sql START TRANSACTION @@ -402,8 +404,8 @@ And if you want to learn more on how to use views and its current limitations, y #### Performance Improvement Creating a database with vttestserver was taking ~45 seconds. This can be problematic in test environments where testcases do a lot of `create` and `drop` database. -In an effort to minimize the database creation time, we have changed the value of `tablet_refresh_interval` to 10s while instantiating vtcombo during vttestserver initialization. We have also made this configurable so that it can be reduced further if desired. -For any production cluster the default value of this flag is still [1 minute](https://vitess.io/docs/15.0/reference/programs/vtgate/). Reducing this value might put more stress on Topo Server (since we now read from Topo server more often) but for testing purposes +In an effort to minimize the database creation time, in [Pull Request #11918](https://github.com/vitessio/vitess/pull/11918) we have changed the value of `tablet_refresh_interval` to 10s while instantiating vtcombo during vttestserver initialization. We have also made this configurable so that it can be reduced further if desired. +For any production cluster the default value of this flag is still [1 minute](https://vitess.io/docs/16.0/reference/programs/vtgate/). Reducing this value might put more stress on Topo Server (since we now read from Topo server more often) but for testing purposes this shouldn't be a concern. ## Minor changes