From fb968416d71ed4a335862ffae3a336a175152f57 Mon Sep 17 00:00:00 2001 From: Alkin Tezuysal Date: Mon, 28 Sep 2020 07:25:59 +0300 Subject: [PATCH 1/6] 8_0_0 Release Notes Signed-off-by: Alkin Tezuysal --- doc/releasenotes/8_0_0_release_notes.md | 45 +++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 doc/releasenotes/8_0_0_release_notes.md diff --git a/doc/releasenotes/8_0_0_release_notes.md b/doc/releasenotes/8_0_0_release_notes.md new file mode 100644 index 00000000000..18790aa803a --- /dev/null +++ b/doc/releasenotes/8_0_0_release_notes.md @@ -0,0 +1,45 @@ +## Incompatible Changes + +*This release includes the following changes which may result in incompatibilities when upgrading from a previous release*. *It is important that Vitess components are* _[upgraded in the recommended order](https://vitess.io/docs/user-guides/upgrading/#upgrade-order)_. *This will change in the next release as documented in* *[VEP-3](https://github.com/vitessio/enhancements/blob/master/veps/vep-3.md).* + +## Deprecations + +## Bugs Fixed + +* set workload = 'olap'; can not repeat, but set workload = 'oltp' can; #4086 +* Fix where clause in information schema with correct database name #6599 + + +### VTGate / MySQL compatibility + + +## Functionality Added or Changed + +* Emergency Reparent Refactor #6449 + * Make emergency reparents more robust. #6206 +* vttablet: create database if not present #6490 + +### VReplication + +* VReplication: _vt.vreplication source column VARBINARY->BLOB #6421 + +### Point-in-time Recovery + +### Healthcheck + +### Examples / Tutorials + +## Documentation + +## Build Environment Changes + +## Functionality Neutral Changes + +* SET planning #6487 +* Settings tweak backport #6488 +* Add more system settings #6486 + +### Other + +* add skip flag that can skip comparing source & destination schema when run splitdiff #6477 +* [java] bump java version to 8.0.0-SNAPSHOT for next release #6478 From a110c39d84abad928a790183767c150bbd4a5a5b Mon Sep 17 00:00:00 2001 From: Alkin Tezuysal Date: Mon, 26 Oct 2020 14:15:06 +0300 Subject: [PATCH 2/6] v8.0 Release Notes -final draft- Signed-off-by: Alkin Tezuysal --- doc/releasenotes/8_0_0_release_notes.md | 258 ++++++++++++++++++++++-- 1 file changed, 239 insertions(+), 19 deletions(-) diff --git a/doc/releasenotes/8_0_0_release_notes.md b/doc/releasenotes/8_0_0_release_notes.md index 18790aa803a..f6c0b77186f 100644 --- a/doc/releasenotes/8_0_0_release_notes.md +++ b/doc/releasenotes/8_0_0_release_notes.md @@ -1,45 +1,265 @@ -## Incompatible Changes +This release complies with VEP-3 which removes the upgrade order requirement. Components can be upgraded in any order. It is recommended that the upgrade order should still be followed if possible, except to canary test the new version of VTGate before upgrading the rest of the components. -*This release includes the following changes which may result in incompatibilities when upgrading from a previous release*. *It is important that Vitess components are* _[upgraded in the recommended order](https://vitess.io/docs/user-guides/upgrading/#upgrade-order)_. *This will change in the next release as documented in* *[VEP-3](https://github.com/vitessio/enhancements/blob/master/veps/vep-3.md).* +## Incompatible Changes -## Deprecations +The following PRs made changes to behavior that clients might be relying on. They should be reviewed carefully so that client code can be changed in concert with a Vitess release deployment. +* Change error code from 1203 to 1153 for three specific error conditions. #6630 + The three errors being changed: +When a GRPC message coming back to the vtgate exceeds the configured size limit (configured via grpc_max_message_size) +If a write is attempting to make a change with a payload that is larger than the configured allowable size +If a scatter query is attempting to collect and process too many rows in memory within the vtgate to construct a response +* Zero auto-increment mode: This enables inserting a 0 value (and not just NULL) into an auto-increment column to generate sequence values. This matches the default behavior of MySQL. #6749 +* Turn off schema tracker by default #6712 ## Bugs Fixed -* set workload = 'olap'; can not repeat, but set workload = 'oltp' can; #4086 +### VTGate / MySQL compatibility * Fix where clause in information schema with correct database name #6599 +* Fix DDL execution on reserved connection without in active transaction #6514 +* Fix allow passed table_schema in information_schema queries to vttablet if the keyspace is not found #6653 +* Fix information schema for non-existent schema #6667 +* Fix reserved connection in autocommit mode on DML #6748 +* Fix dbname that needs escaping that we broke in fce4cfd894 #6794 +* MySQL CLI, mysql --ssl does not work in MySQL 8.0 #5556 +* OperationalError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server during query') #6208 +* Fix support for `show tables where tables_in_db` to use keyspace name, not underlying db name #6446 +* Common empty result set query on MySQL returns a row in Vitess #6663 +* Operator precedence issue #6734 +* Table names out-of-order between replica and primary #6738 +* Close Idle reserved connections and Rollback Idle transactions #6552 +* Minor Vindex fixes: #6526 +* VTgate does not report autocommit system variable and SERVER_STATUS_IN_TRANS flag correctly #5825 +* MariaDB JDBC Connector fails to connect to vtgate with default options #5851 +* VTgate panics if there are no vttablets and vtctlds #6117 +* Remove shard session held in vtgate for reserved connection on connection failure #6522 +* Session variable does not work as expected in 7.0 #6559 +* Retry should not happen when in dedicated connection #6523 +* Correctly report AUTOCOMMIT status in network packets #6551 +* Close Idle reserved connections and Rollback Idle transactions #6552 +* Reset Session for Reserved Connection Query Failure #6673 +* Errors in multi-statement queries not fatal to statement sequence #6747 #6808 +* Lock wait timeout on DELETE from table using lookup index #6698 +* Application couldn't connect to external db giving error tabletserver.go:1522] Code: INVALID_ARGUMENT syntax error at position 33 near ':vtg2' (CallerID: gnocchi) : Sql: "select convert(:vtg1, CHAR(:vtg2)) as anon_1 from dual", #6466 +* The client got the wrong database name #6753 +* DBAPIError exception wrapped from (pymysql.err.InternalError) (1105, u'vtgate: http://gnocchi-gnocchizone-vtgate:15000/: reserved connections are not supported on old gen gateway') #6577 +* Provide support for “SELECT INTO OUTFILE S3” #6811 +* Fix error around breaking of multi statements #6824 +* Support SHOW TABLE STATUS FROM db type queries in vtgate #6354 +* SHOW FULL FIELDS doesn't follow routing rules #6803 + +### Other +* VReplication tablet_picker should keep trying to find a tablet until context expires #6546 +* VReplication improve reverse workflow: update cells/tablet_types #6556 +* VDiff: fix panic for tables with a unicode_loose_md5 vindex #6640 +* Vtctld UI: Add mysqld port display back to vtctld web UI #6598 #6597 +* Vtctld: Fix workflow CopyState fetching (and thus printing). #6657 +* Fix unsaved event regression #6710 +* Change tracker to use allprivs instead of dba #6720 +* Fixes long wait filter keyspace #6721 +* s3 ListBackups doesn’t work from root directory #6732 +* Vtctld: ExecuteFetchAsDba and ExecuteFetchAsAllPrivs do not escape db names. #6545 +* Can't switch on restore_from_backup on existing shard with no tables #4896 +* Refactor conn.go #6818 + +## Functionality Added or Changed ### VTGate / MySQL compatibility +* Add support for DELETE IGNORE #6722 +* Only take on simple dual queries in the vtgate #6666 +* Support mycli access to vtgate #4365 +* Add support for unicode_loose_xxhash #6457 +* Provide virtual information_schema #5394 +* information_schema.tables_schema comparison with dynamic value no longer works #6827 +* Mysqldump: Allow setting multiple variables in one statement #5401 +* Block lock function usage other than dual table #6470 +* Lock Session Support #6517 +* Make sure lookup vindexes are queryable inside transaction #6499 +* Add unicode_loose_xxhash Vindex type #6549 +* Support vindex update or delete when destination is provided by client in dml query #6554 +* Make sure to handle EXPLAIN on tablets #6579 +* Rewrite SHOW TABLES #6615 +* Make offset work in OLAP mode #6655 +* Support sql_calc_found_rows with limit in sharded keyspace #6680 +* Lock shard session heartbeat #6683 +* Mysql -ssl tag is deprecated in the later versions of mysql #6682 +* Use statement support in olap mode #6692 +* Delete row using consistent lookup vindex in where clause #6700 +* Add vtexplain support for non-evenly sharded keyspaces #6705 +* Make sure dual queries set found_rows correctly #6718 +* Add if not exists support to database ddl in parsing #6724 +* Allow Update of same vindex using same vindex lookup on consistent lookup vindex #6736 +* Update Vindex only on changes #6737 +* Add support for `SHOW (DATABASES|VITESS_SHARDS|VITESS_TABLETS) LIKE` #6750 +* 'CASCADE' support #6743 +* Mask database name #6791 +* Make sure to backtick schema names properly #6550 +* Transaction Limiter: fix quota leak if Begin() errors or if the tx killer runs #6731 +* Allow Load Data From S3 #6823 -## Functionality Added or Changed +### OLAP Functioanlity +* Allow switching between OLAP and OLTP #6691 +* Use statement support in olap mode #6692 -* Emergency Reparent Refactor #6449 - * Make emergency reparents more robust. #6206 -* vttablet: create database if not present #6490 -### VReplication +### Set Statement Support +Set statement support is added in Vitess. There are [some system variables](https://github.com/vitessio/vitess/blob/master/go/vt/sysvars/sysvars.go#L147,L190) which are disabled by default and can be enabled using flag `-enable_system_settings` on VTGate. These system variables are set on the mysql and they does not use connection pool and have their own dedicated connection. -* VReplication: _vt.vreplication source column VARBINARY->BLOB #6421 +* Disabled passthrough system variables by default. #6859 +* Allow switching workload between OLAP and OLTP #4086 #6691 +* Support for multiple session set statement syntax #6508 #6495 #6494 +* Allow enumeration in set system variables #6493 +* Handle boolean settings better #6501 +* Set statements refactored to use plan building #6487 +* Add more system settings #6486 +* Evaluate system variables #6708 -### Point-in-time Recovery -### Healthcheck -### Examples / Tutorials +### VReplication -## Documentation +* VReplication: _vt.vreplication source column VARBINARY->BLOB, allowing more tables to participate in a workflow #6421 +* VReplication and vstreamer performance and error metrics #6519, #6132 +* VReplication: Find collations for char primary keys and cast comparison during catchup #6568 +* Varchar primary keys with collations can cause vreplication to miss rows #6622 +* VReplication: Timestamp w/ Heartbeat #6635 +* VReplication: Improve few error/log messages #6669 +* VReplication: handle SET statements in the binlog stream #6772 +* VReplication: e2e Tests added to CI #6730 +* VDiff logs: additional logging around vdiff/vstreamer for observability #6684 +* VStream: experimental POC of mysql protocol #6670 +* Materialize: Support rollups using a literal value as a column #6733 +* Materialization: Ignore non-participating tables that are renamed/deleted #6778 +* Materialize: add ability for maintaining accurate table-level aggregates #6767 +* Schema Tracker: Turn off schema tracker by default #6712 (breaking change) +* Workflow: Add List-All Command #6533 +* Workflow: List -> Show, and Expand Metadata #6544 +* Refactor: Cleanup Noisy Workflow Logging #6744 +* Schema tracker: Change tracker to use the allprivs user instead of dba #6720 +* vtctl: Adding VExec and Workflow commands #6410 +* vtctl Workflow cmd: don't expect all shards to have workflow streams #6576 +* Reshard: Sort table definitions before comparing them #6765 #6738 +* Expose a shard range calculator as a vtctl command GenerateShardRanges #6751 #6752 +* VStream: Field event now has all column attributes #6525 +* VStream events do not populate field "Flags" on FIELD events #5914 +* RFC: VReplication based SplitClone #4604 +* Experimental: Mysql Protocol support for VStream #6675 +* VReplication: Add metrics around vstreamer and vreplication metrics #6787 +* Fix issues with missing tables in vstreams #6789 +* Add validations and logs to fix common issues faced during MoveTables/Reshard #6814 +* Allow multiple blacklists for master #6816 -## Build Environment Changes +### VTtablet +* VTtablet throttling #6668 +https://vitess.io/docs/reference/features/tablet-throttler/ +* TabletManager: publish displayState #6648 +* TabletManager: call setReadOnly inside ChangeTabletType #6762 +* TabletManager: change how SetReadOnly is called to avoid errors from externally managed tablets #6786 +* TabletManager: call SetReadOnly inside ChangeTabletType #6804 +* vttablet: Open and Close healthStreamer #6515 +* vttablet: create database if not present #6490 +* Restore: do not change tablet type to RESTORE if not actually performing a restore #6679 +* Restore: checkNoDB should not require tables to be present #6695 +* Heartbeat: Additional metrics/stats (lag histogram) #6528 #6634 +* VTtablet throttler feature flag: -enable-lag-throttler #6815 +* VTtablet two-phase commit design doc #6498 -## Functionality Neutral Changes +### VTorc - Orchestrator +* Orchestrator: initial import of Orchestrator #6582 +* Orchestrator: vitess mode #6613 +* Orchestrator: more changes #6714 +* Orchestrator: use contexts with timeout for remote operations #6780 +* Orchestrator: fixed orchestrator govet errors #6781 +* Orchestrator: Add clusters_to_watch flag to. Defaults to all clusters. #6821 -* SET planning #6487 -* Settings tweak backport #6488 -* Add more system settings #6486 +### Point in Time Recovery + +* PITR: Fix deadlock in ff code #6774 +* Add flags to allow PITR binlog client TLS support #6775 ### Other +* Add realtime health stats to vtctld /api/keyspace/ks/tablets API #6569 +* vtctld UI: Fix logic for displaying vindexes #6603 +* Online schema changes [#6547](https://vitess.io/docs/user-guides/schema-changes/) +* Table lifecycle management #6719, [docs](https://vitess.io/docs/reference/features/table-lifecycle/) +* Online-DDL: migration uses low priority throttling #6837 +* Emergency Reparent Refactor #6449 + * Make emergency reparents more robust. #6206 +* Reparent test failures #6706 +* Update user-facing terminology in the vtctld2 UI #6481 +* GetSchema: Batch/parallel access to underlying mysqld for lower latency #6491 +* Prometheus interface: vtgate_buffer_* metrics cardinality #6326 +* Show more vindex details in vtctld /app/schema UI #6450 +* GRPC: update enforcement policy on server to match the one from the client #6629 + +## Examples / Tutorials + +* region_sharding: working resharding example #6565 +* vtcompose/docker-compose: fix InitShardMaster #6584 +* [docker] fix tablet hostname #6509 +## Documentation + +* Replace more uses of slave with replica/replication #6520 +* Update some option docstrings for accuracy. #6590 +* Minor addition to Materialize help text. #6627 +* Remove auto-generation comment #6728 * add skip flag that can skip comparing source & destination schema when run splitdiff #6477 +* Fix and clarify some help options #6817 + +## Build Environment Changes + +* Address #6631 by reducing optimization level #6632 +* Add front-end build tooling + fix build error #6473 +* Replace gogo proto with golang #6571 +* Improve make proto #6580 +* Remove chromium hard-coding #6636 +* Docker - allow BUILD_NUMBER variable override #6628 +* Docker/lite: Fix package URLs for ubi images. #6704 +* Helm & docker/k8s update #6723 +* Fix build errors in ./go/vt/vtgate for go1.15 #6800 +* Fix build error on vcursor_impl tests on newer versions of go #6799 * [java] bump java version to 8.0.0-SNAPSHOT for next release #6478 +* Helm/docker: fix Docker builds + tag 7.0.2 #6773 + +## Functionality Neutral Changes + +* Routing cleanup: remove routeOptions #6531 +* PITR: testcase #6594 +* Replaced Error with a warning in case parsing of VT_SPAN_CONTEXT fails #6766 +* Reparent tests refactoring: setup/teardown for each test, reduce cut/paste, improve readability #6726 +* Decider: endtoend test infrastructure + tests #6770 +* Orchestrator: add more test cases #6801 +* Orchestrator: don't set DbName for tests that don't need it. #6812 +* Add unit test case to improve test coverage for go/sqltypes/named_result.go #6672 +* End to end: deflake sharding tests #6715 +* Workflow test: fix flaky test. remove obsolete code #6694 +* Add unit test for func Proto3ValuesEqual. #6649 +* Ensure tests for discoverygateway #6536 +* Convert syslog events to normal log events #6511 +* Add diagnostic logging to healthcheck. #6512 +* Healthcheck should receive healthcheck updates from all tablets in cells_to_watch #6852 +* Named columns and convenience SQL query interface #6543 +* Check http response code in vtctld API tests #6570 +* Reverting package-lock.json to the one pre-PR #6603 #6611 +* Removed sqlparser.preview to set logstats stmtType and use plan.type #6637 +* AST struct name rewording #6642 +* Fix: Remove SetMaster Query Expectation from ERS Test #6617 +* Fix path for vttablet query log in end to end test #6656 +* Fixes error log #6496 +* Add diagnostic logging to healthcheck. #6535 +* Try to install apt keys from list of different keyservers #6674 +* Trivial copy-pasta comment fixup #6592 +* Remove long-unused memcache and cacheservice #6596 +* Fix MoveTables docstring; was not valid JSON. #6606 +* Tablet gateway: unit tests #6608 +* Tablet gateway doesn't see all tablets in cells from cells_to_watch #6846 +* Add Sleep to ERS Unit Test #6610 +* Strings to enum #6729 +* Remove syslog dependency #3563 +* Operator precedence must take associativity into consideration #6758 +* Add optional ignore to delete statement #6802 +* Invalid trace payload causes vitess to fail the request and return an error #6759 +* Vttablet: minor logging fix #6564 +* Vttablet: add more logging to checkMastership #6618 From c0091beed8ff0c8a8ebfb36fdb0d72f477cd189e Mon Sep 17 00:00:00 2001 From: Alkin Tezuysal Date: Mon, 26 Oct 2020 16:20:35 +0300 Subject: [PATCH 3/6] Added [8.0] Fix error handling in olap mode #6940 Signed-off-by: Alkin Tezuysal --- doc/releasenotes/8_0_0_release_notes.md | 1 + 1 file changed, 1 insertion(+) diff --git a/doc/releasenotes/8_0_0_release_notes.md b/doc/releasenotes/8_0_0_release_notes.md index f6c0b77186f..48c5f670715 100644 --- a/doc/releasenotes/8_0_0_release_notes.md +++ b/doc/releasenotes/8_0_0_release_notes.md @@ -46,6 +46,7 @@ If a scatter query is attempting to collect and process too many rows in memory * Fix error around breaking of multi statements #6824 * Support SHOW TABLE STATUS FROM db type queries in vtgate #6354 * SHOW FULL FIELDS doesn't follow routing rules #6803 +* Fix error handling in olap mode #6940 ### Other * VReplication tablet_picker should keep trying to find a tablet until context expires #6546 From b434dde81e1bd89780c6796f285918ea6fb7c1c8 Mon Sep 17 00:00:00 2001 From: Alkin Tezuysal Date: Tue, 27 Oct 2020 12:43:20 +0300 Subject: [PATCH 4/6] Minor updates in this branch Signed-off-by: Alkin Tezuysal --- doc/releasenotes/8_0_0_release_notes.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/doc/releasenotes/8_0_0_release_notes.md b/doc/releasenotes/8_0_0_release_notes.md index 48c5f670715..9ade6977b32 100644 --- a/doc/releasenotes/8_0_0_release_notes.md +++ b/doc/releasenotes/8_0_0_release_notes.md @@ -46,7 +46,8 @@ If a scatter query is attempting to collect and process too many rows in memory * Fix error around breaking of multi statements #6824 * Support SHOW TABLE STATUS FROM db type queries in vtgate #6354 * SHOW FULL FIELDS doesn't follow routing rules #6803 -* Fix error handling in olap mode #6940 +* VDiff/Tablet Picker: fix picker retry logic to handle canceled context #6954 +* Healthcheck: Use isIncluded correctly to fix replica/rdonly routing bug #6922 ### Other * VReplication tablet_picker should keep trying to find a tablet until context expires #6546 @@ -102,6 +103,7 @@ If a scatter query is attempting to collect and process too many rows in memory ### OLAP Functioanlity * Allow switching between OLAP and OLTP #6691 * Use statement support in olap mode #6692 +* Fix error handling in olap mode #6940 ### Set Statement Support @@ -223,6 +225,10 @@ https://vitess.io/docs/reference/features/tablet-throttler/ * Fix build error on vcursor_impl tests on newer versions of go #6799 * [java] bump java version to 8.0.0-SNAPSHOT for next release #6478 * Helm/docker: fix Docker builds + tag 7.0.2 #6773 +* Docker: Upgrade to Debian Buster (release-8.0) #6888 +* Docker: Pin mysql 5.7 and 8.0 versions for xtrabackup compatibility +* Docker: Revert "Docker - upgrade to Debian Buster (release-8.0)" #6929 +* Zookeper: Download zookeeper 3.4.14 from archive site #6867 ## Functionality Neutral Changes @@ -240,7 +246,7 @@ https://vitess.io/docs/reference/features/tablet-throttler/ * Ensure tests for discoverygateway #6536 * Convert syslog events to normal log events #6511 * Add diagnostic logging to healthcheck. #6512 -* Healthcheck should receive healthcheck updates from all tablets in cells_to_watch #6852 +* Healthcheck should receive healthcheck updates from all tablets in cells_to_watch #6852 * Named columns and convenience SQL query interface #6543 * Check http response code in vtctld API tests #6570 * Reverting package-lock.json to the one pre-PR #6603 #6611 From 234a25c41b72a202a57b1bc984a38a70f603a894 Mon Sep 17 00:00:00 2001 From: Alkin Tezuysal Date: Tue, 27 Oct 2020 12:48:24 +0300 Subject: [PATCH 5/6] Vtorc comment by Sugu Signed-off-by: Alkin Tezuysal --- doc/releasenotes/8_0_0_release_notes.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc/releasenotes/8_0_0_release_notes.md b/doc/releasenotes/8_0_0_release_notes.md index 9ade6977b32..be216e50aa6 100644 --- a/doc/releasenotes/8_0_0_release_notes.md +++ b/doc/releasenotes/8_0_0_release_notes.md @@ -169,6 +169,8 @@ https://vitess.io/docs/reference/features/tablet-throttler/ * VTtablet two-phase commit design doc #6498 ### VTorc - Orchestrator +Following PRs are experimental version of Vitess-native Orchestrator 'vtorc' is ready for users to try out. + * Orchestrator: initial import of Orchestrator #6582 * Orchestrator: vitess mode #6613 * Orchestrator: more changes #6714 From 72d930259cfcf7167b50542feaceed65eb84b584 Mon Sep 17 00:00:00 2001 From: Alkin Tezuysal Date: Tue, 27 Oct 2020 13:52:56 +0300 Subject: [PATCH 6/6] Updated set commentary by systay Signed-off-by: Alkin Tezuysal --- doc/releasenotes/8_0_0_release_notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/releasenotes/8_0_0_release_notes.md b/doc/releasenotes/8_0_0_release_notes.md index be216e50aa6..26d50aab3dd 100644 --- a/doc/releasenotes/8_0_0_release_notes.md +++ b/doc/releasenotes/8_0_0_release_notes.md @@ -107,7 +107,7 @@ If a scatter query is attempting to collect and process too many rows in memory ### Set Statement Support -Set statement support is added in Vitess. There are [some system variables](https://github.com/vitessio/vitess/blob/master/go/vt/sysvars/sysvars.go#L147,L190) which are disabled by default and can be enabled using flag `-enable_system_settings` on VTGate. These system variables are set on the mysql and they does not use connection pool and have their own dedicated connection. +Set statement support is added in Vitess. There are [some system variables](https://github.com/vitessio/vitess/blob/master/go/vt/sysvars/sysvars.go#L147,L190) which are disabled by default and can be enabled using flag `-enable_system_settings` on VTGate.These system variables are set on the backing MySQL instance, and will force the connection to be dedicated instead of part of the connection pool. * Disabled passthrough system variables by default. #6859 * Allow switching workload between OLAP and OLTP #4086 #6691