Online DDL: support 'SHOW VITESS_MIGRATIONS' queries by shlomi-noach · Pull Request #7638 · vitessio/vitess

shlomi-noach · 2021-03-08T10:24:36Z

Description

Supporting these queries in vtgate:

SHOW VITESS_MIGRATIONS - shows all migrations for keyspace
SHOW VITESS_MIGRATIONS LIKE '...' - shows vitess migrations filtered for UUID, or context, or status. For example:
- SHOW VITESS_MIGRATIONS LIKE '6df3201f_7fda_11eb_aa27_f875a4d24e90' - shows a specific migration
- SHOW VITESS_MIGRATIONS LIKE 'failed' - show migrations with failed status
SHOW VITESS_MIGRATIONS WHERE ... - add any custom filter, e.g.:
- SHOW VITESS_MIGRATIONS WHERE started_timestamp > NOW() - INTERVAL 7 DAY
SHOW VITESS_MIGRATIONS FROM keyspace [LIKE ...|WHERE ...] - explicitly indicate keyspace

Related Issue(s)

Checklist

Should this PR be backported?
Tests were added or are not required
Documentation was added or is not required

Deployment Notes

Impacted Areas in Vitess

Components that this PR will affect:

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2021-03-08T11:09:47Z

Seems like even though we specify key.DestinationAllShards, query only runs on a single shard.

Signed-off-by: Andres Taylor <andres@planetscale.com>

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

Signed-off-by: Andres Taylor <andres@planetscale.com>

…ess into online-ddl-sql-interface Signed-off-by: Andres Taylor <andres@planetscale.com>

Signed-off-by: Andres Taylor <andres@planetscale.com>

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2021-03-09T08:03:40Z

Thank you very much to @harshit-gangal and @systay who guided and led me in the right path, and for noticing my Online DDL endtoend tests had wrong/broken shard configuration!

shlomi-noach · 2021-03-09T08:04:50Z

This is now ready to review. This PR only introduced SHOW VITESS_MIGRATIONS command(s), which is a more urgent requirement. ALTER VITESS_MIGRATIONS will be handled in a followup PR.

Documentation to follow.

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2021-03-09T08:21:40Z

Added one last commit, refactored a couple functions in endtoend tests, where I originally did a lot of copy+paste. There's more cleanup work to be done, but not in this PR.

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

go/test/endtoend/cluster/query_util.go

go/test/endtoend/cluster/vtgate_util.go

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

harshit-gangal

I would suggest to refactor the file directory as go/test/endtoend/onlineddl/<ghost/vrepl/...> and place query_util and vtgate_util under go/test/endtoend/onlineddl/ those are the only tests that are using it.

harshit-gangal · 2021-03-09T18:45:30Z

go/vt/sqlparser/ast.go

 	nodeType := strings.ToLower(node.Type)
 	if (nodeType == "tables" || nodeType == "columns" || nodeType == "fields" || nodeType == "index" || nodeType == "keys" || nodeType == "indexes" ||
-		nodeType == "databases" || nodeType == "schemas" || nodeType == "keyspaces" || nodeType == "vitess_keyspaces" || nodeType == "vitess_shards" || nodeType == "vitess_tablets") && node.ShowTablesOpt != nil {
+		nodeType == "databases" || nodeType == "schemas" || nodeType == "keyspaces" || nodeType == "vitess_keyspaces" || nodeType == "vitess_shards" || nodeType == "vitess_tablets" || nodeType == "vitess_migrations") && node.ShowTablesOpt != nil {


this is not needed as show vitess migrations is not legacy.

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach · 2021-03-10T05:34:36Z

@harshit-gangal I have restructured the tests, all nested under go/test/endtoend/onlineddl

go/test/endtoend/onlineddl # base package with shared functionality
go/test/endtoend/onlineddl/vrepl
go/test/endtoend/onlineddl/vrepl_stress
go/test/endtoend/onlineddl/ghost
go/test/endtoend/onlineddl/revert

shlomi-noach added 2 commits March 8, 2021 12:13

Online DDL: support 'SHOW VITESS_MIGRATIONS' queries

c8d4d4b

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

fulklk parse for 'show vitess_migrations'

823d1dc

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

systay and others added 12 commits March 8, 2021 14:35

parser & ast: use ShowBasic instead of ShowLegacy for vitess_migrations

0b5adde

Signed-off-by: Andres Taylor <andres@planetscale.com>

use vtgate 'SHOW VITESS_MIGRATIONS'

ae739f4

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

added FROM and build engine primitive

1d112ab

Signed-off-by: Andres Taylor <andres@planetscale.com>

Merge branch 'online-ddl-sql-interface' of github.com:planetscale/vit…

2ddaeb2

…ess into online-ddl-sql-interface Signed-off-by: Andres Taylor <andres@planetscale.com>

add executor test for show vitess_migrations

aab5701

Signed-off-by: Andres Taylor <andres@planetscale.com>

use send primitive for show vitess_migrations

7d00e70

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

fix test setup to use sharded keyspace

606807b

Signed-off-by: Harshit Gangal <harshit@planetscale.com>

stricter SHOW VITESS_MIGRATONS query; formatting improvements

a9a7ee6

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

endtoend online ddl gh-ost tests: fixed sharded keyspace

dad0e47

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

fixed unit test

237e223

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

endtoend online ddl revert tests: using SHOW VITESS_MIGRATIONS

3896ca8

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

endtoend online ddl stress tests: using SHOW VITESS_MIGRATIONS

a420c31

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach requested review from a team, harshit-gangal, rohit-nayak-ps and systay March 9, 2021 08:02

shlomi-noach marked this pull request as ready for review March 9, 2021 08:03

shlomi-noach requested a review from GuptaManan100 as a code owner March 9, 2021 08:03

refactor: consolidate utility functions

7227937

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

fixed vtgate unit test

4bffce9

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach mentioned this pull request Mar 9, 2021

Suggestions for online DDL query syntax #7604

Closed

fixed test number of shards

3d570ec

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

systay approved these changes Mar 9, 2021

View reviewed changes

systay reviewed Mar 9, 2021

View reviewed changes

go/test/endtoend/cluster/query_util.go Outdated Show resolved Hide resolved

systay reviewed Mar 9, 2021

View reviewed changes

go/test/endtoend/cluster/query_util.go Outdated Show resolved Hide resolved

systay reviewed Mar 9, 2021

View reviewed changes

go/test/endtoend/cluster/vtgate_util.go Outdated Show resolved Hide resolved

fixed per review

b2788bd

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

harshit-gangal reviewed Mar 9, 2021

View reviewed changes

shlomi-noach added 8 commits March 10, 2021 07:13

remove unneeded code

fc66891

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

added go/test/endtoend/onlineddl

b0282e3

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

general purpose testing package for online-ddl tests

a328a49

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

correct package name

b228da2

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

use new shared package

c90ec62

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

restructure directories

38bdc9c

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

rename package names

86ad32a

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

update paths

cea6361

Signed-off-by: Shlomi Noach <2607934+shlomi-noach@users.noreply.github.com>

shlomi-noach merged commit b659a7f into vitessio:master Mar 10, 2021

shlomi-noach deleted the online-ddl-sql-interface branch March 10, 2021 08:49

askdba added the Component: Query Serving label Mar 10, 2021

askdba added this to the v10.0 milestone Mar 10, 2021

shlomi-noach mentioned this pull request Mar 25, 2021

Online DDL: revertible migrations, declarative migrations, SQL commands vitessio/website#733

Merged

ajm188 mentioned this pull request Jul 15, 2021

slack vitess v10.pre tinyspeck/vitess#228

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Online DDL: support 'SHOW VITESS_MIGRATIONS' queries#7638

Online DDL: support 'SHOW VITESS_MIGRATIONS' queries#7638
shlomi-noach merged 26 commits intovitessio:masterfrom
planetscale:online-ddl-sql-interface

shlomi-noach commented Mar 8, 2021 •

edited

Loading

Uh oh!

shlomi-noach commented Mar 8, 2021

Uh oh!

shlomi-noach commented Mar 9, 2021

Uh oh!

shlomi-noach commented Mar 9, 2021

Uh oh!

shlomi-noach commented Mar 9, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harshit-gangal left a comment

Uh oh!

harshit-gangal Mar 9, 2021

Uh oh!

shlomi-noach Mar 10, 2021

Uh oh!

shlomi-noach commented Mar 10, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

shlomi-noach commented Mar 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Checklist

Deployment Notes

Impacted Areas in Vitess

Uh oh!

shlomi-noach commented Mar 8, 2021

Uh oh!

shlomi-noach commented Mar 9, 2021

Uh oh!

shlomi-noach commented Mar 9, 2021

Uh oh!

shlomi-noach commented Mar 9, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harshit-gangal left a comment

Choose a reason for hiding this comment

Uh oh!

harshit-gangal Mar 9, 2021

Choose a reason for hiding this comment

Uh oh!

shlomi-noach Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

shlomi-noach commented Mar 10, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shlomi-noach commented Mar 8, 2021 •

edited

Loading