Slack vitess 2021.01.11 release

[ajm188]

For vtctld, just to get the diff reviewed and approved. We shouldn't merge this to tinyspeck/master.

Contains the following PRs:

• Initial implementation of vtctld service vitessio/vitess#7128
• Add FindAllShardsInKeyspace to vtctldserver vitessio/vitess#7201
• [vtctldclient] vtctldclient generator vitessio/vitess#7238
• [vtctld] Fix accidentally-broken legacy vtctl output format vitessio/vitess#7285

I ran make build at each cherry-pick; the code is buildable and I can run the local example at hit a grpc endpoint:

❯ vtctldclient --server "localhost:15999" FindAllShardsInKeyspace commerce | jq '.'
{
  "shards": {
    "0": {
      "keyspace": "commerce",
      "name": "0",
      "shard": {
        "master_alias": {
          "cell": "zone1",
          "uid": 100
        },
        "master_term_start_time": {
          "seconds": 1610323010,
          "nanoseconds": 829901000
        },
        "is_master_serving": true
      }
    }
  }
}

And the retaining of the legacy format (note vtctlclient not vtctldclient here):

❯ vtctlclient -server "localhost:15999" FindAllShardsInKeyspace commerce
{
  "0": {
    "master_alias": {
      "cell": "zone1",
      "uid": 100
    },
    "master_term_start_time": {
      "seconds": 1610412569,
      "nanoseconds": 995650000
    },
    "is_master_serving": true
  }
}

[ajm188]

@setassociative @doeg I've updated this branch to include the following:

• #7302
• #7311
• #7321 — with the InitShardPrimary command removed
• #7334
• #7346 — Note this should be a transparent refactor of the legacy GetSchema; also ripped out the small change to a vtadmin test file
• #7352 — s/ListBackups/GetBackups/
• #7360
• #7395 — This is the big topo-management one you both already reviewed separately. I had to remove some GenericVExec that tmclient.TabletManagerClient doesn't have yet in our branch
• #7404

Here are the relevant test results:

❯ go test -v ./go/vt/vtctl/... ./go/vt/wrangler/... | grep -E '^ok'
ok      vitess.io/vitess/go/vt/vtctl    (cached)
ok      vitess.io/vitess/go/vt/vtctl/endtoend   (cached)
ok      vitess.io/vitess/go/vt/vtctl/fakevtctlclient    (cached)
ok      vitess.io/vitess/go/vt/vtctl/grpcvtctlclient    (cached)
ok      vitess.io/vitess/go/vt/vtctl/grpcvtctldclient   (cached)
ok      vitess.io/vitess/go/vt/vtctl/grpcvtctldserver   (cached)
ok      vitess.io/vitess/go/vt/wrangler (cached)
ok      vitess.io/vitess/go/vt/wrangler/testlib (cached)

If you want the diff between the last deploy and now: slack-vitess-2021.01.11.rc1...slack-vitess-2021.01.11-release

[doeg]

:rubber_stamp:

[ajm188]

And against the local example

GetTablets + GetSchema

❯ vtctldclient --server "localhost:15999" GetTablets
zone1-0000000100 commerce 0 master SFO-M-AMASON02:15100 SFO-M-AMASON02:17100 [] 2021-02-11T16:59:35Z
zone1-0000000101 commerce 0 replica SFO-M-AMASON02:15101 SFO-M-AMASON02:17101 [] <null>
zone1-0000000102 commerce 0 rdonly SFO-M-AMASON02:15102 SFO-M-AMASON02:17102 [] <null>
❯ vtctldclient --server "localhost:15999" GetSchema zone1-0000000100
{
  "database_schema": "CREATE DATABASE /*!32312 IF NOT EXISTS*/ {{.DatabaseName}} /*!40100 DEFAULT CHARACTER SET utf8 */",
  "table_definitions": [
    {
      "name": "corder",
      "schema": "CREATE TABLE `corder` (\n  `order_id` bigint(20) NOT NULL AUTO_INCREMENT,\n  `customer_id` bigint(20) DEFAULT NULL,\n  `sku` varbinary(128) DEFAULT NULL,\n  `price` bigint(20) DEFAULT NULL,\n  PRIMARY KEY (`order_id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8",
      "columns": [
        "order_id",
        "customer_id",
        "sku",
        "price"
      ],
      "primary_key_columns": [
        "order_id"
      ],
      "type": "BASE TABLE",
      "data_length": "16384",
      "row_count": "0",
      "fields": [
        {
          "name": "order_id",
          "type": "INT64",
          "table": "corder",
          "org_table": "corder",
          "database": "vt_commerce",
          "org_name": "order_id",
          "column_length": 20,
          "charset": 63,
          "decimals": 0,
          "flags": 49667
        },
        {
          "name": "customer_id",
          "type": "INT64",
          "table": "corder",
          "org_table": "corder",
          "database": "vt_commerce",
          "org_name": "customer_id",
          "column_length": 20,
          "charset": 63,
          "decimals": 0,
          "flags": 32768
        },
        {
          "name": "sku",
          "type": "VARBINARY",
          "table": "corder",
          "org_table": "corder",
          "database": "vt_commerce",
          "org_name": "sku",
          "column_length": 128,
          "charset": 63,
          "decimals": 0,
          "flags": 128
        },
        {
          "name": "price",
          "type": "INT64",
          "table": "corder",
          "org_table": "corder",
          "database": "vt_commerce",
          "org_name": "price",
          "column_length": 20,
          "charset": 63,
          "decimals": 0,
          "flags": 32768
        }
      ]
    },
    {
      "name": "customer",
      "schema": "CREATE TABLE `customer` (\n  `customer_id` bigint(20) NOT NULL AUTO_INCREMENT,\n  `email` varbinary(128) DEFAULT NULL,\n  PRIMARY KEY (`customer_id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8",
      "columns": [
        "customer_id",
        "email"
      ],
      "primary_key_columns": [
        "customer_id"
      ],
      "type": "BASE TABLE",
      "data_length": "16384",
      "row_count": "0",
      "fields": [
        {
          "name": "customer_id",
          "type": "INT64",
          "table": "customer",
          "org_table": "customer",
          "database": "vt_commerce",
          "org_name": "customer_id",
          "column_length": 20,
          "charset": 63,
          "decimals": 0,
          "flags": 49667
        },
        {
          "name": "email",
          "type": "VARBINARY",
          "table": "customer",
          "org_table": "customer",
          "database": "vt_commerce",
          "org_name": "email",
          "column_length": 128,
          "charset": 63,
          "decimals": 0,
          "flags": 128
        }
      ]
    },
    {
      "name": "product",
      "schema": "CREATE TABLE `product` (\n  `sku` varbinary(128) NOT NULL,\n  `description` varbinary(128) DEFAULT NULL,\n  `price` bigint(20) DEFAULT NULL,\n  PRIMARY KEY (`sku`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8",
      "columns": [
        "sku",
        "description",
        "price"
      ],
      "primary_key_columns": [
        "sku"
      ],
      "type": "BASE TABLE",
      "data_length": "16384",
      "row_count": "0",
      "fields": [
        {
          "name": "sku",
          "type": "VARBINARY",
          "table": "product",
          "org_table": "product",
          "database": "vt_commerce",
          "org_name": "sku",
          "column_length": 128,
          "charset": 63,
          "decimals": 0,
          "flags": 20611
        },
        {
          "name": "description",
          "type": "VARBINARY",
          "table": "product",
          "org_table": "product",
          "database": "vt_commerce",
          "org_name": "description",
          "column_length": 128,
          "charset": 63,
          "decimals": 0,
          "flags": 128
        },
        {
          "name": "price",
          "type": "INT64",
          "table": "product",
          "org_table": "product",
          "database": "vt_commerce",
          "org_name": "price",
          "column_length": 20,
          "charset": 63,
          "decimals": 0,
          "flags": 32768
        }
      ]
    }
  ],
  "version": "4a696261cc73c605a887ad6a547dce18"
}

[ajm188]

Tagged current branch HEAD as slack-vitess-2021.01.11.rc2

[ajm188]

I've pushed up 2 more cherry-picks to allow us to start working with the new workflows API!

Contains:

• #7518
• #7575
• A rewrite of the new query-planner to use the old types in sqlparser, not in any cherry-pick.
  • Aside: @setassociative maybe this is related to the parser issues you were having?

Testing:

❯ make build && go test -count=1 ./go/vt/vtctl/grpcvtctldserver/... ./go/vt/vtctl/grpcvtctldclient/... ./go/vt/vtctl/workflow/...
Fri Mar 5 10:14:17 EST 2021: Building source tree
ok      vitess.io/vitess/go/vt/vtctl/grpcvtctldserver   0.480s
?       vitess.io/vitess/go/vt/vtctl/grpcvtctldserver/testutil  [no test files]
ok      vitess.io/vitess/go/vt/vtctl/grpcvtctldclient   0.674s
?       vitess.io/vitess/go/vt/vtctl/grpcvtctldclient/codegen   [no test files]
?       vitess.io/vitess/go/vt/vtctl/workflow   [no test files]
ok      vitess.io/vitess/go/vt/vtctl/workflow/vexec     0.318s
?       vitess.io/vitess/go/vt/vtctl/workflow/vexec/testutil    [no test files]

@setassociative @doeg mind taking a look? Then I'll tag it and start to deploy.

[doeg]

@ajm188 looks reasonable to me! Thank you for doing this. Are the test failures significant or is that "expected"?

[ajm188]

......... "expected" unfortunately

[ajm188]

Tagged current branch as slack-vitess-2021.01.11.rc3