vrepl: import from multiple external sources by sougou · Pull Request #6103 · vitessio/vitess

sougou · 2020-04-22T19:53:24Z

This change builds on top of the tabletserver componentization changes. With this change, you can add a section to the tablet_config yaml that configures external sources, like this:

externalConnections:
  product:
    socket: /home/sougou/dev/src/vitess.io/vitess/vtdataroot/vtroot_15201/vt_0000000622/mysql.sock
    dbName: vt_product
    app:
      user: vt_app
    dba:
      user: vt_dba
  customer:
    flavor: FilePos
    socket: /home/sougou/dev/src/vitess.io/vitess/vtdataroot/vtroot_15201/vt_0000000620/mysql.sock
    dbName: vt_customer
    app:
      user: vt_app
    dba:
      user: vt_dba

You can then create replication streams from these external sources with these inserts:

VReplicationExec: insert into _vt.vreplication (workflow, db_name, source, pos, max_tps, max_replication_lag, tablet_types, time_updated, transaction_timestamp, state) values('product', 'vt_commerce', 'filter:<rules:<match:\"product\" > > external_mysql:\"product\" ', '', 9999, 9999, 'master', 0, 0, 'Running')
VReplicationExec: insert into _vt.vreplication (workflow, db_name, source, pos, max_tps, max_replication_lag, tablet_types, time_updated, transaction_timestamp, state) values('customer', 'vt_commerce', 'filter:<rules:<match:\"customer\" > > external_mysql:\"customer\" ', '', 9999, 9999, 'master', 0, 0, 'Running')
VReplicationExec: insert into _vt.vreplication (workflow, db_name, source, pos, max_tps, max_replication_lag, tablet_types, time_updated, transaction_timestamp, state) values('orders', 'vt_commerce', 'filter:<rules:<match:\"orders\" > > external_mysql:\"customer\" ', '', 9999, 9999, 'master', 0, 0, 'Running')

harshit-gangal · 2020-04-23T05:04:45Z

Does it have to be via insert only or the workflow can be run with vtctl command as well?

sougou · 2020-04-23T05:15:26Z

Does it have to be via insert only or the workflow can be run with vtctl command as well?

It's actually a vtctl VReplicationExec command. I just pasted the insert part of it.

harshit-gangal · 2020-04-28T18:01:34Z

go/vt/vttablet/tabletserver/tx_engine.go

+	te.txPool.Open(te.env.Config().DB.AppWithDB(), te.env.Config().DB.DbaWithDB(), te.env.Config().DB.AppDebugWithDB())

 	if te.twopcEnabled && te.state == AcceptingReadAndWrite {
-		te.twoPC.Open(te.env.DBConfigs())
+		te.twoPC.Open(te.env.Config().DB)


I am not liking these kind of calls Config().DB.AppWithDB. I like it before with DBConfig().func

I can add a convenience function that shortcuts Config().DB to DBConfigs() keeping it backward compatible. Will that work?

That will work

go/vt/vttablet/tabletmanager/vreplication/engine.go

go/vt/vttablet/tabletmanager/vreplication/external_connector.go

harshit-gangal · 2020-04-28T19:15:10Z

@rohit-nayak-ps , interested in looking at vreplication changes?

sougou

@rafael will also be reviewing this.

sougou · 2020-04-28T19:48:27Z

go/vt/vttablet/tabletserver/tx_engine.go

+	te.txPool.Open(te.env.Config().DB.AppWithDB(), te.env.Config().DB.DbaWithDB(), te.env.Config().DB.AppDebugWithDB())

 	if te.twopcEnabled && te.state == AcceptingReadAndWrite {
-		te.twoPC.Open(te.env.DBConfigs())
+		te.twoPC.Open(te.env.Config().DB)


I can add a convenience function that shortcuts Config().DB to DBConfigs() keeping it backward compatible. Will that work?

go/vt/vttablet/tabletmanager/vreplication/engine.go

go/vt/vttablet/tabletmanager/vreplication/external_connector.go

rafael

@sougou this looks really good. I reviewed and tested this PR in our environment.

I only have one comment around the tests. Let me know your thoughts.

rafael · 2020-05-05T00:03:44Z

go/cmd/vttablet/vttablet.go

-	config.DB = config.DB.Init(socketFile)
+	config.DB.Init(socketFile)
+	for _, cfg := range config.ExternalConnections {
+		cfg.Init("")


nit - Maybe it would be clearer to have InitWithSocket and Init() I found it a bit confusing that this is initialized with empty string without any context on why that's the case.

go/vt/vttablet/tabletmanager/vreplication/engine.go

rafael · 2020-05-05T00:35:13Z

go/vt/vttablet/tabletmanager/vreplication/external_connector_test.go

+}
+
+func startExternalVReplication(t *testing.T, bls *binlogdatapb.BinlogSource) (cancelr func()) {
+	query := binlogplayer.CreateVReplication("test", bls, "", 9223372036854775807, 9223372036854775807, 0, vrepldb)


It seems that we are only testing vcopy in this test. I think we should also test a externalVReplication with a starting position provided. That will ensure that we go through the main methods of the interface (VStream and VStreamRows). I think this got lost when deleting the old tests.

There are upcoming use cases where we'll create new named subcomponents of tabletserver along with the global unnamed one. Specifically for vreplication external mysql. This change will tolerate such usage and we'll just reuse the vars created by the unnamed tabletserver. Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

The binlog dump was using dba privileges, but it's better to use app, which is also expected to have privileges to request binlogs. Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

rafael · 2020-05-05T23:39:01Z

LGTM

sougou requested review from harshit-gangal and rafael April 22, 2020 19:54

morgo modified the milestone: v6.0 Apr 27, 2020

harshit-gangal reviewed Apr 28, 2020

View reviewed changes

sougou commented Apr 28, 2020

View reviewed changes

rafael reviewed May 5, 2020

View reviewed changes

sougou added 16 commits May 5, 2020 10:58

vstreamer: change to use vt_app always

e12cd1c

The binlog dump was using dba privileges, but it's better to use app, which is also expected to have privileges to request binlogs. Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

vstream: fix broken test

7f28458

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

yaml: ExternalConnections for vreplication

f5f2872

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

vrepl: externalConnector: cache

c6239ca

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

vrepl: tests for externalConnector

929e0e6

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

vrepl: cleanup VStreamerClient

4c67c62

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

yaml: use TabletConfig.DB instead of dbcfgs

aeabda5

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

yaml: tabletenv.NewTestEnv -> NewEnv

1747a51

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

vrepl: reinstate VStreamClient.Open

407d845

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

vrepl: migration e2e test WIP

6c6444d

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

vrepl: customizers for cluster creation

4c836a1

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

vrepl: migration tests working

528c84b

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

vrepl: documentation for test

f5e7b5a

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

vrepl: vstreamerClient -> VStreamerClient

516d6bd

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

vrepl: review comments

4948935

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

sougou force-pushed the ss-vrepl-external branch from ce7cf75 to 4948935 Compare May 5, 2020 17:59

rafael approved these changes May 5, 2020

View reviewed changes

rafael merged commit 80e6295 into vitessio:master May 5, 2020

sougou mentioned this pull request May 11, 2020

track how many times vstreaclient is opened and closed #6097

Closed

sougou deleted the ss-vrepl-external branch May 11, 2020 22:33

deepthi added this to the v7.0 milestone Jul 27, 2020

ameetkotian mentioned this pull request Aug 20, 2020

Slack vitess 2020.08.19.r0 tinyspeck/vitess#180

Merged

Conversation

sougou commented Apr 22, 2020

Uh oh!

harshit-gangal commented Apr 23, 2020

Uh oh!

sougou commented Apr 23, 2020

Uh oh!

harshit-gangal Apr 28, 2020

Choose a reason for hiding this comment

Uh oh!

sougou Apr 28, 2020

Choose a reason for hiding this comment

Uh oh!

harshit-gangal Apr 29, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

harshit-gangal commented Apr 28, 2020

Uh oh!

sougou left a comment

Choose a reason for hiding this comment

Uh oh!

sougou Apr 28, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rafael left a comment

Choose a reason for hiding this comment

Uh oh!

rafael May 5, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rafael May 5, 2020

Choose a reason for hiding this comment

Uh oh!

rafael commented May 5, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants