VTShovel - VReplication support for external databases

[rafael]

Description

Have you ever wanted to leverage the powers of vreplication outside the environment of Vitess? Do you dream about copying bytes? The following PR will have a solution for you.

Introducing: vtshovel . A flexible tool that allows you to create vreplication streams directly from mysql instances outside of the vitess ecosystem.

To give a bit of context about the motivation for this tool, we (Slack) are in the process of migrating entire databases from our legacy mysql clusters to Vitess. We plan to leverage this tool to help us get in sync mysql instances from our legacy clusters to their Vitess counterparts.

We are thinking that other folks might find useful to have a tool like this when doing migrations.

Core Design

• The core design for this feature is to leverage all the vreplication framework. It addition to vttabletss, vreplication streams can now point to external databases. This is done by introducing an abstraction that implements vstreamer methods:

  // VStreamerClient exposes the core interface of a vstreamer
  type VStreamerClient interface {
      // Open sets up all the environment for a vstream
      Open(ctx context.Context) error
      // Close closes a vstream
      Close(ctx context.Context) error
           // VStream streams VReplication events based on the specified filter.
          VStream(ctx context.Context, startPos string, filter *binlogdatapb.Filter, send func([]*binlogdatapb.VEvent) error) error
          // VStreamRows streams rows of a table from the specified starting point.
          VStreamRows(ctx context.Context, query string, lastpk *querypb.QueryResult, send func(*binlogdatapb.VStreamRowsResponse) error) error
   }
  

• Depending on the configuration of the vreplication, a vplayer will choose between a TabletVStreamerClient and a NewMySQLVStreamerClient.
• There is some technical debt introduced in the way we are choosing credentials for the external mysql. At the moment we don't have a good way to do that. To not increase the scope of this PR we added erepl to dbconfigs.
• I added good test coverage to vstreamer_client and also an integration test for the vplayer.

Additional changes

• This PR also adds supports for statement based replication. Binlonplayer can understand both statement and row based replication. Certain types of filters won't be supported in statement based and the stream will fail in such cases. At the moment match all rules will be supported for statement based replication streams.

• It also added support to stream without using gtids. This was done cutting some corners, but it will be cleaned up soon. @sougou and I are working on that.

[sougou]

Approach very nice overall. A few minor nits.

[sougou]

I'm thinking this function can pull the dbconfigs based on the external repl user name. Then you don't have to pass it through to vreplication.Engine.

[rafael]

My thinking here is that the end goal is to be able to point to any external DB, having a this parameter here will make it more flexible.

I think we shouldn't rely heavily in the repl username as we would like to refactor that soon.

What do you think?

[sougou]

Looking good. Couple of comments.

[sougou]

This is a conflict between grpc code gen and goimports. If you re-run goimports, all these files will revert to unchanged. Or, you can just manually revert these yourself.

[rafael]

Hi @jawabuu, the way this code ended up landing, is not a separate binary. It is intended to be run as part of vttablet. The way it works is that is possible to have VReplication streams where the source is external.

[rafael]

There is some discussion about how this is used in our Slack community: https://vitess.slack.com/archives/C0PQY0PTK/p1604989571062700?thread_ts=1579649445.062400&cid=C0PQY0PTK