Allow multiple databases in one mysql instance

[deepthi]

We would like to allow multiple vttablets to run against the same MySQL server, with each one using a separate database/schema. The way this will be used is by having a separate schema for each keyspace.
Care needs to be taken in designating tablets as master/replica to match the underlying MySQL configuration.

For this to work, the following are needed:

• vreplication table should be scoped by db_name. the PK does not need to change (since it is a unique id). Most selects and deletes will remain the same with a few exceptions.
• reparent_journal does not need to change. Vitess only writes to this table and there is enough information for a human operator to recognize the reparenting events.
• local_metadata table is used for orchestrator integration. Alias in local_metadata is the tablet alias. This is used in orchestrator config to call TER for the tablet. With multiple vttablets pointing to one mysql server, in case of failover this will need to change to TER all the vttablets pointing to the new master. ClusterAlias in local_metadata is currently keyspace.shard. It is used in the Orc web UI.
• local_metadata and shard_metadata are used by schema_swap, which needs to change to work at the individual database/schema level
• To ensure feature parity, tables used by 2PC also need a new column db_name

How to use:

• Not all vttablets can manage the MySQL server. One tablet needs to be started in the managed mode, and the others in non-managed mode. Backups can then be done from the managing vttablet. See https://vitess.io/docs/user-guides/vttablet-modes/
• Non-managing vttablets will need to be reparented using TabletExternallyReparented. See https://vitess.io/docs/user-guides/reparenting/#external-reparenting
• Restore from backup will need to be triggered by a managing vttablet. Other vttablets can then be started up and pointed to the running MySQL.

Tasks:

• add db_name to _vt tables (vreplication, local_metadata, shard_metadata).
• new integration test with 2 vttablets running against each MySQL
• change orchestrator conf
• TwoPC related tables

This PR addresses the basic workflows (reparenting, resharding). It also includes the necessary changes to schema_swap code. Orchestrator and TwoPC will be addressed in forthcoming PRs.

cc @tirsen
Signed-off-by: deepthi deepthi@planetscale.com

[tirsen]

This is not going to work for us. We still need to be able to change the _vt database name through configuration. The reason is we want to host multiple apps in the same Aurora instance in staging due to cost and they will have different permissions.

[tirsen]

Actually this will work for us short and medium term since we don't currently share database clusters between services. So let's keep going down this path.

[sougou]

On my first pass, I'm not feeling comfortable about the local_metadata change. Let's see if we can avoid changing it at all.
I'm thinking we should even kill schema swap if need be, since it's the only other tool that depends on it, and needs the db_name field.

I also don't like the bloat on the vreplication alter, but it appears unavoidable. So, let's leave that part alone.

[deepthi]

On my first pass, I'm not feeling comfortable about the local_metadata change. Let's see if we can avoid changing it at all.
I'm thinking we should even kill schema swap if need be, since it's the only other tool that depends on it, and needs the db_name field.

For the record, @sougou and I discussed this at length. We decided to keep these changes so that orchestrator can be used with vitess in the multiple vttablet vs single mysql configuration.

[rafael]

I think this PR introduces a regression when you have tablets that have a build that does not use db_name. We are trying to provision new tablets in our dev environment and seeing the following error:

F0422 12:54:35.889873    3243 vttablet.go:130] NewActionAgent() failed: Field 'db_name' doesn't have a default value (errno 1364) (sqlstate HY000) during query: INSERT INTO _vt.local_metadata (name,value) VALUES ('PromotionRule','neutral') ON DUPLICATE KEY UPDATE value = 'neutral'
failed to -init_populate_metadata

[deepthi]

I think this PR introduces a regression when you have tablets that have a build that does not use db_name. We are trying to provision new tablets in our dev environment and seeing the following error:

F0422 12:54:35.889873    3243 vttablet.go:130] NewActionAgent() failed: Field 'db_name' doesn't have a default value (errno 1364) (sqlstate HY000) during query: INSERT INTO _vt.local_metadata (name,value) VALUES ('PromotionRule','neutral') ON DUPLICATE KEY UPDATE value = 'neutral'
failed to -init_populate_metadata

As it turns out, a new tablet was deployed with a new build, and a backup was taken from that.
Then tablets on hosts with old builds tried to restore from this backup, and failed.
This is really a 'forward-compatibility' type of scenario that we did not design for.
In order to avoid such problems during rolling updates that might span days or weeks, a fix has been implemented in #4830

[deepthi]

Re orchestrator integration, we will not change the config file at docker/orchestrator/orchestrator.conf.json, that will continue to be for the single-schema case. With multiple schemas, the conf file needs to change as follows:

"DetectClusterAliasQuery": "SELECT GROUP_CONCAT(value) FROM _vt.local_metadata WHERE name='ClusterAlias'",
  "DetectInstanceAliasQuery": "SELECT GROUP_CONCAT(value) FROM _vt.local_metadata WHERE name='Alias'",
  "PostMasterFailoverProcesses": [
    "echo 'Recovered from {failureType} on {failureCluster}. Failed: {failedHost}:{failedPort}; Promoted: {successorHost}:{successorPort}' >> /tmp/recovery.log",
    "IFS=','  for tablet in {successorAlias}; do vtctlclient -server vtctld:15999 TabletExternallyReparented $tablet; done"]

[yzsfly]

One question: two keyspaces against one shard, will the orchestrator still work well as expected?

[yzsfly]

if multiple database in one mysql, how to set cluster_to_watch flag, will the vtorc detecting result still ok?

[deepthi]

if multiple database in one mysql, how to set cluster_to_watch flag, will the vtorc detecting result still ok?

good question. vtorc must be configured to watch all keyspaces in the same mysql instance.