Remove Usage of VReplicationExec For _vt.vreplication Reads by mattlord · Pull Request #14424 · vitessio/vitess

mattlord · 2023-11-02T03:30:44Z

Description

This replaces reads (i.e. SELECTs) against the _vt.vreplication sidecar table in the vtctldclient implementation (vtctl/vtctlclient are also deprecated) that were using the deprecated VReplicationExec RPC with the newer tabletmanager RPCs which do not expose the implementation (SQL against tables) and thus address upgrade/downgrade issues.

Note: the deprecated vtctlclient / wrangler implementation is left unchanged — so that it can be used as a fallback if there are any issues with the new vtctldclient implementation until it's eventually removed — by adding legacy functions in its code path as needed.

It also begins the process of unifying the creation of VReplication workflow streams in vtctldclient using the new tabletmanager RPCs. Materialize and Reshard were still using the deprecated VReplicationExec RPC and here we generalized the MoveTables usage of the new RPCs and moved Materialize over to that.

Related Issue(s)

Part of: Tracking Issue: Replace VReplicationExec Command + RPC #12086

Checklist

"Backport to:" labels have been added if this change should be back-ported
Tests were added or are not required
Did the new or modified tests pass consistently locally and on the CI
Documentation was added or is not required

Signed-off-by: Matt Lord <mattalord@gmail.com>

vitess-bot · 2023-11-02T03:30:47Z

Signed-off-by: Matt Lord <mattalord@gmail.com>

github-actions · 2023-12-04T01:29:21Z

This PR is being marked as stale because it has been open for 30 days with no activity. To rectify, you may do any of the following:

Push additional commits to the associated branch.
Remove the stale label.
Add a comment indicating why it is not stale.

If no action is taken within 7 days, this PR will be closed.

Signed-off-by: Matt Lord <mattalord@gmail.com>

go/vt/vtctl/workflow/resharder.go

go/vt/wrangler/switcher_interface.go

Signed-off-by: Matt Lord <mattalord@gmail.com>

… comments Signed-off-by: Matt Lord <mattalord@gmail.com>

Signed-off-by: Matt Lord <mattalord@gmail.com>

1. Add a new check to the e2e test that confirms all streams are running after a Reshard. 2. Address a TODO on Reshard stream migrations 3. Fix a bug in readRefStreams (was using the workflow struct rather (than workflow name in the map key) 4. Moves more spots to using proto getters as they are nil safe Signed-off-by: Matt Lord <mattalord@gmail.com>

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord · 2024-03-06T03:36:15Z

@rohit-nayak-ps I've addressed your review comments now and the tests are all green again. Please let me know what you think. Thanks!

go/test/endtoend/vreplication/helper_test.go

proto/tabletmanagerdata.proto

go/vt/vtctl/workflow/materializer_test.go

rohit-nayak-ps · 2024-03-08T11:23:19Z

go/vt/vtctl/workflow/materializer_test.go

+			sourceShards: []string{"-"},
+			targetShards: []string{"-80", "80-"},
+			wantReqs: map[uint32]*tabletmanagerdatapb.CreateVReplicationWorkflowRequest{
+				200: {


The expected requests are quite similar: only shards are different in this and the next, for example. A helper function to get the request based on the differing attributes might be useful. It will both reduce the size of the test and make it easier to see what is the difference in the expectation.

Can you show me an example of what you mean? I don't see a clear and intuitive way to do this w/o the test case NOT containing the logic or explicitly stating what each test case wants.

I can do this later when I am refactoring the tests. Let's get this PR merged for now!

go/vt/vtctl/workflow/resharder.go

go/vt/vtctl/workflow/server.go

go/vt/vttablet/tabletmanager/rpc_vreplication.go

Signed-off-by: Matt Lord <mattalord@gmail.com>

Generalize new workflow creation path

52daf18

Signed-off-by: Matt Lord <mattalord@gmail.com>

vitess-bot bot added NeedsDescriptionUpdate The description is not clear or comprehensive enough, and needs work NeedsIssue A linked issue is missing for this Pull Request NeedsWebsiteDocsUpdate What it says labels Nov 2, 2023

github-actions bot added this to the v19.0.0 milestone Nov 2, 2023

Move first VReplication usage

e9b94d8

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the rm_vrepl_exec branch from 1ce5ab3 to e9b94d8 Compare November 2, 2023 03:57

Correct generalized create flow

12e1a60

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the rm_vrepl_exec branch 2 times, most recently from 41dcece to 68c41ff Compare November 2, 2023 20:48

Get Materialize creation working w/o VReplicationExec

1045dd5

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the rm_vrepl_exec branch from 68c41ff to 1045dd5 Compare November 2, 2023 21:29

mattlord mentioned this pull request Nov 2, 2023

Bug Report: UpdateVReplicationWorkflow tablet RPC only updates one stream #14446

Closed

mattlord force-pushed the rm_vrepl_exec branch from d470f96 to 8f49c6b Compare November 3, 2023 01:10

Fix workflow update bug

fcc631d

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the rm_vrepl_exec branch from 8f49c6b to fcc631d Compare November 3, 2023 02:22

mattlord added Type: Internal Cleanup Component: VReplication and removed NeedsIssue A linked issue is missing for this Pull Request labels Nov 3, 2023

github-actions bot added the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Dec 4, 2023

mattlord removed the Stale Marks PRs as stale after a period of inactivity, which are then closed after a grace period. label Dec 4, 2023

mattlord changed the title ~~Remove Usage of VReplicationExec~~ Remove Usage of VReplicationExec For Reads Jan 3, 2024

Merge remote-tracking branch 'origin/main' into rm_vrepl_exec

c7637e7

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the rm_vrepl_exec branch from bac951f to 7bfcb8b Compare January 3, 2024 17:19

mattlord changed the title ~~Remove Usage of VReplicationExec For Reads~~ Remove Usage of VReplicationExec For _vt.vreplication Reads Jan 3, 2024

mattlord force-pushed the rm_vrepl_exec branch from 7bfcb8b to 4c34f54 Compare January 3, 2024 20:18

Add HasVReplicationWorkflows and ReadVReplicationWorkflows RPCs

4e0a542

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the rm_vrepl_exec branch from 4c34f54 to 4e0a542 Compare January 3, 2024 21:53

rohit-nayak-ps requested changes Feb 9, 2024

View reviewed changes

go/vt/vtctl/workflow/resharder.go Show resolved Hide resolved

go/vt/wrangler/switcher_interface.go Outdated Show resolved Hide resolved

Merge remote-tracking branch 'origin/main' into rm_vrepl_exec

561a154

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the rm_vrepl_exec branch from cfddc38 to 471c7a3 Compare February 12, 2024 22:10

Modify stream check to catch unmigrated streams

9162c6f

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the rm_vrepl_exec branch from 471c7a3 to 9162c6f Compare February 12, 2024 23:57

mattlord requested a review from rohit-nayak-ps March 5, 2024 18:56

Add RPC to update the state for multiple workflows and address review…

9831ee5

… comments Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the rm_vrepl_exec branch from c4d802a to f2de9a0 Compare March 5, 2024 20:02

Merge remote-tracking branch 'origin/main' into rm_vrepl_exec

c2ae9a4

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the rm_vrepl_exec branch from f2de9a0 to c2ae9a4 Compare March 5, 2024 20:04

mattlord added 3 commits March 5, 2024 16:04

Fix db_name usage in new RPC

c82c8e5

Signed-off-by: Matt Lord <mattalord@gmail.com>

Minor changes after self review of new code

798c509

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord mentioned this pull request Mar 6, 2024

Tracking Issue: Replace VReplicationExec Command + RPC #12086

Open

8 tasks

rohit-nayak-ps reviewed Mar 9, 2024

View reviewed changes

Address review comments

bb1219c

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the rm_vrepl_exec branch from a40bbeb to bb1219c Compare March 10, 2024 01:05

mattlord added 2 commits March 9, 2024 21:29

Address more review comments

7a1b5d9

Signed-off-by: Matt Lord <mattalord@gmail.com>

Merge remote-tracking branch 'origin/main' into rm_vrepl_exec

a97f7ae

Signed-off-by: Matt Lord <mattalord@gmail.com>

mattlord force-pushed the rm_vrepl_exec branch from 3640fda to a97f7ae Compare March 10, 2024 02:29

mattlord added 5 commits March 9, 2024 21:32

Fixup uncaught merge conflict

94d59b3

Signed-off-by: Matt Lord <mattalord@gmail.com>

Comment nit

c910193

Signed-off-by: Matt Lord <mattalord@gmail.com>

Also split out query building for ReadVReplicationWorkflows

f2d64ea

Signed-off-by: Matt Lord <mattalord@gmail.com>

Stop needlessly passing db_name from vtctld to vttablet

35492ed

Signed-off-by: Matt Lord <mattalord@gmail.com>

Minor changes after self review of new code

60f7d05

Signed-off-by: Matt Lord <mattalord@gmail.com>

rohit-nayak-ps approved these changes Mar 10, 2024

View reviewed changes

mattlord merged commit fbaed97 into vitessio:main Mar 11, 2024

mattlord deleted the rm_vrepl_exec branch March 11, 2024 00:44

mattlord mentioned this pull request Mar 19, 2024

VReplication: Fix workflow filtering in GetWorkflows RPC #15524

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove Usage of VReplicationExec For _vt.vreplication Reads#14424

Remove Usage of VReplicationExec For _vt.vreplication Reads#14424
mattlord merged 46 commits intovitessio:mainfrom
planetscale:rm_vrepl_exec

mattlord commented Nov 2, 2023 •

edited

Loading

Uh oh!

vitess-bot bot commented Nov 2, 2023

Uh oh!

github-actions bot commented Dec 4, 2023

Uh oh!

Uh oh!

Uh oh!

mattlord commented Mar 6, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rohit-nayak-ps Mar 8, 2024

Uh oh!

mattlord Mar 10, 2024 •

edited

Loading

Uh oh!

rohit-nayak-ps Mar 10, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mattlord commented Nov 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Checklist

Uh oh!

vitess-bot bot commented Nov 2, 2023

Review Checklist

General

Tests

Documentation

New flags

If a workflow is added or modified:

Backward compatibility

Uh oh!

github-actions bot commented Dec 4, 2023

Uh oh!

Uh oh!

Uh oh!

mattlord commented Mar 6, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rohit-nayak-ps Mar 8, 2024

Choose a reason for hiding this comment

Uh oh!

mattlord Mar 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohit-nayak-ps Mar 10, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mattlord commented Nov 2, 2023 •

edited

Loading

mattlord Mar 10, 2024 •

edited

Loading