orc: vitess mode by sougou · Pull Request #6613 · vitessio/vitess

sougou · 2020-08-23T17:17:32Z

This is a POC/MVP of some sort where Orchestrator runs as a vitess component. Features of this PR are described in #6612.

This is not production ready yet, but people can start playing with it. The code mostly works, but there are lots of TODOs, and tests have to be written.

This version does not handle semi-sync settings correctly.

High level

Accept topo command line flags so Orc can directly read and write.
Orc polls the topo to discover tablet records, saves them in a new vitess_tablet table, and uses them as the authoritative source of mysql instances. For example, if a tablet record is deleted, the mysql instance is forgotten.
Analysis code drives from the vitess_tablet table and left joins on database_instance.
Additional tablet specific rules have been added in the analysis phase to improve decision-making.
Recovery code has new actions for initializating a brand new cluster (ISM) and wiring up new vttablets.
The code is written using a "desired state" approach: If a previous action failed and was incomplete, the newer analysis will detect the differences and converge the system to a consistent state.

Details

Replica info is currently provided in the config file. This will eventually be fetched from vttablets.
Improved logging to print file and line. We'll need to eventually merge this with the cloud-friendly vitess logging.
Removed the additional filtering from analysis. We want to monitor at all instances.
cluster_alias_override is not used because vitess is the authority here.
For now, we use SuggestedClusterAlias as the authoritative source. We'll need to eventually change this to ClusterName. But there are places where the cluster name is inferred from the instance key.
ChangeMasterTo always sets the credentials, with deprecates ChangeMasterCredentials.
There is a logic/tablet_discover.go and an inst/tablet_dao.go. The line between the two is not well drawn. We'll need to fix this forward.
In topology_recovery, I had to comment out a call to inst.BeginDowntime of the dead master. In the case of vitess, we should wire it back to the cluster when it comes back up. But this downtime was preventing it.
LockShard is sprayed in a few places, but it's not watertight yet.
During recovery, once the new candidate is identified, we first call ChangeType(MASTER) on the tablet. If anything fails after this, this master status is used to authoritatively finish anything that was incomplete.
Comaster situation becomes obsolete with the new approach: A new analysis called MasterHasMaster has been created, which is evaluated based on whether the tablet record is the master.
An electNewMaster function has been added to elect a brand new master if no previous master existed. This replaces the vitess InitShardMaster.

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

shlomi-noach

Please see comments/questions inline.

Overall everything here makes sense and seems like a pretty smooth integration; that is, it feels like the code changes belong here and play well with existing logic. Good work!

go/vt/orchestrator/db/generate_base.go

go/vt/orchestrator/external/golib/log/log.go

go/vt/orchestrator/inst/analysis.go

shlomi-noach · 2020-08-24T06:36:20Z

go/vt/orchestrator/inst/analysis.go

 	AnalyzedInstanceKey                       InstanceKey
 	AnalyzedInstanceMasterKey                 InstanceKey
+	TabletType                                topodatapb.TabletType
+	MasterTimeStamp                           time.Time


What does this timestamp stand for?

This was our first step towards a consensus protocol. Every newly elected master saves its timestamp. If the previous master did not get properly demoted (maybe it wasn't reachable), then we use the timestamp to resolve who the real master is.

I'm not happy that we currently rely on a timestamp because rogue clocks can mess things up. We should eventually move towards a system that doesn't depend on the clock.

go/vt/orchestrator/logic/topology_recovery.go

shlomi-noach · 2020-08-24T07:18:14Z

go/vt/orchestrator/logic/topology_recovery.go

+				if !inst.IsBannedFromBeingCandidateReplica(replica) {
+					candidate = replica
+				}
+			}


a bit too much if-else-if-else. Can you please explain the logic in words?

I've refactored this. LMK if the new code reads better.

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

shlomi-noach · 2020-08-26T05:05:53Z

go/vt/orchestrator/logic/tablet_discovery.go

-	}
+	refreshTabletsUsing(func(instanceKey *inst.InstanceKey) {
+		_ = inst.InjectSeed(instanceKey)
+	})


This should work well upon startup

shlomi-noach · 2020-08-26T05:06:22Z

go/vt/orchestrator/logic/tablet_discovery.go

 }

 // RefreshTablets reloads the tablets from topo.
 func RefreshTablets() {


What's the purpose/usage of this function?

This is called from the main discovery loop. It polls for new/deleted tablets, and updates Orc accordingly.

got it. Makes sense; one thing to note is that this causes a synchronous topology read on all tablets. Possibly this isn't what you wanted? Consider calling DiscoverInstance() instead, which will queue the instance for discovery, and avoid double-probing it within a pre configured timeslice.

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

shlomi-noach · 2020-08-26T05:37:19Z

go/vt/orchestrator/logic/tablet_discovery.go

 func RefreshTablets() {
 	refreshTabletsUsing(func(instanceKey *inst.InstanceKey) {
-		_, _ = inst.ReadTopologyInstance(instanceKey)
+		DiscoverInstance(*instanceKey)


🤞 let's see how this goes!

harshit-gangal

Code walkthrough and getting all the TODOs makes code read better.

shlomi-noach

Also LGTM

sougou added 12 commits August 21, 2020 20:02

orc: tablet discovery

a56d7b4

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

orc: reparent calls into vitess

2f70bae

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

orc: add custom replica user credentials

b39bf89

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

orc: NotConnectedToMaster

9b75b74

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

orc: new failure modes

06cce27

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

orc: ClusterHasNoMaster

23d9929

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

orc: fix promotion rule

7a98cfa

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

orc: use SuggestedClusterAlias for analysis

acebb77

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

orc: improve logging: add file & line

2dcde5c

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

orc: add cell resilience

0a7d83a

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

orc: tweak some rules for somoother behavior

88a8549

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

orc: linter fixes

5d699e5

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

sougou requested review from deepthi, harshit-gangal and shlomi-noach August 23, 2020 19:09

songgavin approved these changes Aug 24, 2020

View reviewed changes

shlomi-noach reviewed Aug 24, 2020

View reviewed changes

sougou added 2 commits August 24, 2020 15:14

orc: address review comments

9c3e067

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

orc: use InjectSeed to preload tablets

5821782

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

shlomi-noach reviewed Aug 26, 2020

View reviewed changes

orc: ReadTopologyInstance->DiscoverInstance

e2ccf88

Signed-off-by: Sugu Sougoumarane <ssougou@gmail.com>

shlomi-noach reviewed Aug 26, 2020

View reviewed changes

harshit-gangal approved these changes Sep 1, 2020

View reviewed changes

shlomi-noach approved these changes Sep 1, 2020

View reviewed changes

sougou merged commit 187be0a into vitessio:master Sep 3, 2020

shlomi-noach deleted the ss-oc2-vt-mode branch September 3, 2020 03:58

askdba added this to the v8.0 milestone Oct 6, 2020

setassociative mentioned this pull request Mar 5, 2021

Vitess v8.0 Release branch tinyspeck/vitess#194

Merged

Conversation

sougou commented Aug 23, 2020

High level

Details

Uh oh!

shlomi-noach left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harshit-gangal left a comment

Choose a reason for hiding this comment

Uh oh!

shlomi-noach left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants