Adding doc about vtgate routing.

[alainjobart]

No description provided.

[demmer]

This is a really great and thorough write-up of how things work. Thanks!

I have one question and a comment about one of the the proposed future changes.

[demmer]

I'm a bit confused why this particular issue is only a problem with the new cross-cell region routing.

Specifically doesn't the same issue occur when routing to a master in another cell since the ServedFrom or the shard partitioning for a master in cell2 might not be propagated to the SrvKeyspace in cell1?

[alainjobart]

The master case is actually a bit different, since there is only one. We always migrate all cells at once when migrating the master, as it doesn't make sense to have one master one way for one cell, and another way for another cell. The old / new masters are only configured one way, and blacklist all queries for what is not there, there is only one choice.

For replica and rdonly though, I kinda agree it doesn't matter much to use the wrong shard map when going cross-cell. Unless:

• there is no tablet of one kind in the other cell. Like you have both old and new tablets in the current cell, and use the new tablets, but the other cell only has old tablets.
• the other case we ran into was no tablets for a keyspace in the local cell. There was no SrvKeyspace at all. So the routing failed much earlier.

[demmer]

I see -- so the problem comes up in the case where we migrate replicas differently in different cells.

Also the case of no SrvKeyspace in the current cell if there are no tablets for a keyspace is definitely an issue and is worth calling out explicitly, thanks for reminding me. From my standpoint though, that seems like something we should be able to handle externally via vtctl since it is only an issue when setting up the cell for the first time or adding a new keyspace, right?

[demmer]

I really like the cleanliness of this design, and in some environments this will likely be true, but for deployments like Slack is planning, I don't believe this is necessarily the case.

Specifically, AWS round trip times as measured from one of our vtgates is as follows:

from vtgate in us-east-1a:
tablet in us-east-1a: rtt min/avg/max/mdev = 0.100/0.117/0.133/0.012 ms
tablet in us-east-1b: rtt min/avg/max/mdev = 0.662/0.668/0.677/0.032 ms
tablet in us-east-1d: rtt min/avg/max/mdev = 0.401/0.408/0.419/0.007 ms
tablet in us-east-1e: rtt min/avg/max/mdev = 0.687/0.698/0.714/0.021 ms

So there is a clear penalty to cross AZs (both added latency and cost) but it is on the order of 4-6x, not orders of magnitude.

Still our plan is to deploy a vitess cell per AZ for both the latency benefits and fault tolerance, so I'd want to make sure this design still allows for the ability for one vtgate to cross cells.

More concretely, this design consideration is a consistency vs performance tradeoff -- Slack (and others) might be willing to tolerate a small amount of time where queries could be misrouted during a shard split while the various SrvKeyspace records are being updated, if it means we don't have to pay an extra hop for every cross-cell query.

[alainjobart]

I agree, and I think the proposed design would make this clear: you can still go cross-cell, but you need your vtgates to health-check the tablets cross-cell. That can be a large number of tablets. Otherwise just point at a vtgate pool somewhere else, and it will add the extra hop.

[demmer]

Great! Thanks for clarifying.

[derekperkins]

discovery

[derekperkins]

@enisoc This will be good to monitor as you're starting with the operator - #3500

[sougou]

LGTM