raft: introduce/fix TestPreVoteWithCheckQuorum #8334

lishuai87 · 2017-07-29T09:28:02Z

raft: set leader id to none when becomePreCandidate.

When a follower election timeout, it should mark leader to none (so that old leader is not in lease). Current implementation of Pre-Vote ignore this, which may cause wrong judgement .

xiang90 · 2017-07-29T22:28:50Z

@javaforfun can you add a test case?

lishuai87 · 2017-07-30T09:11:41Z

@xiang90 done. add two test

lishuai87 · 2017-07-31T16:52:18Z

seems confusing when raft recv a message with higher term, i will create a new pr for this

xiang90 · 2017-08-01T02:26:12Z

raft/raft_test.go

+
+	sm := nt.peers[1].(*raft)
+	if sm.state != StateLeader {
+		t.Errorf("peer 1 state: %s, want %s", sm.state, StateLeader)


use t.fatalf for checking the pre-condition. use t.errorf for the actual state checking.

xiang90 · 2017-08-01T02:26:26Z

raft/raft_test.go

+		t.Errorf("peer 3 state: %s, want %s", sm.state, StatePreCandidate)
+	}
+
+	sm.logger.Infof("going to bring back peer 3 and kill peer 1")


remove the logging.

xiang90 · 2017-08-01T02:27:02Z

raft/raft_test.go

+	sm2 := nt.peers[2].(*raft)
+	sm3 := nt.peers[3].(*raft)
+	if sm2.state != StateLeader && sm3.state != StateLeader {
+		t.Errorf("no leader")


errorf needs better context. see other tests's errorf output as examples

xiang90 · 2017-08-01T02:27:26Z

@bdarnell

can you take a quick look too?

bdarnell · 2017-08-07T01:50:35Z

raft/raft.go

@@ -611,6 +611,7 @@ func (r *raft) becomePreCandidate() {
 	// but doesn't change anything else. In particular it does not increase
 	// r.Term or change r.Vote.
 	r.step = stepCandidate
+	r.lead = None


It is strange to change r.lead here when we do not change r.Term. r.lead is still the leader of that term. I think we should leave r.lead unchanged here, and if we change anything, we should change the inLease check at line 711. We can't become a pre-candidate if we've heard from the leader within the lease timeout, so if state == StatePreCandidate inLease must be false.

But some users may use hasLeader() to check whether cluster is in normal state. It's too tricky to let users know StatePreCandidate or inLease.

Also in raft paper, it just say:

If a follower receives no communication over a period of time called the election timeout,
then it assumes there is no viable leader and begins an election to choose a new leader.

@javaforfun

it assumes there is no viable leader and begins an election to choose a new leader.

begins election means increasing term, right? The pre candidate phase seems like a grey area. But I prefer to keep the leader field to be not changed as @bdarnell suggested.

@xiang90 I think thesis section 6.2 point out how to handle this situation. see #8334 (comment)

lishuai87 · 2017-08-24T03:56:17Z

I've updated this pr, please take a look. @xiang90 @bdarnell

lishuai87 · 2017-08-24T12:12:33Z

I've found another interesting scenario for Pre-Vote together with checkQuorum.

cluster have 5 nodes [A, B, C, D, E], with leader A, all terms are 1.
network partition happen, cluster have two partition [A, B], [C, D, E].
so [A, B] has leader A with term 1, [C, D, E] has leader C with term 2.
A will checkQuorum after election timeout, and become follower, set leader to None with term 1.
B start pre election, but B has leader = A with term 1.

it's strange that one node think another node is leader, but the other node is not leader, and they are in same partition!
so if leader can become follower, and change r.lead to none with term not changed, follower also can do this.

bdarnell

Does it make sense to use both CheckQuorum and PreVote together? In CockroachDB we were using them for the same purpose so we turned off CheckQuorum when we enabled PreVote.

I think that only part of CheckQuorum's functionality is needed when PreVote is enabled: you want the leader to step down, but you don't need to change the way votes are granted (I think). So what if we changed

https://github.com/coreos/etcd/blob/4ec31f4f7f95b8ffbacc7a3d0f7173a44b1f11e8/raft/raft.go#L712

to if !force && inLease && !r.preVote?

bdarnell · 2017-08-24T16:53:58Z

raft/raft.go

+	if r.checkQuorum {
+		// If network partition happen, and leader is in minority partition,
+		// it will step down, become follower without notifying others.
+		r.lead = None


Why does this change matter?

It doesn't seem that strange to me that other nodes would still consider node A to be the leader when it has stepped down due to CheckQuorum.

As described in thesis section 6.2, Routing requests to the leader,

Raft must also prevent stale leadership information from delaying client requests indefinitely. Leadership information can become stale all across the system, in leaders, followers, and clients: • Leaders: A server might be in the leader state, but if it isn’t the current leader, it could be needlessly delaying client requests. For example, suppose a leader is partitioned from the rest of the cluster, but it can still communicate with a particular client. Without additional mechanism, it could delay a request from that client forever, being unable to replicate a log entry to any other servers. Meanwhile, there might be another leader of a newer term that is able to communicate with a majority of the cluster and would be able to commit the client’s request. Thus, a leader in Raft steps down if an election timeout elapses without a successful round of heartbeats to a majority of its cluster; this allows clients to retry their requests with another server. • Followers: Followers keep track of the leader’s identity so that they can redirect or proxy clients. They must discard this information when starting a new election or when the term changes. Otherwise, they might needlessly delay clients (for example, it would be possible for two servers to redirect to each other, placing clients in an infinite loop).

followers need discard leader information when starting a new election or when the term changes.

if not, in the previous scenario, partition [A, B] has A with lead = None, B with lead = A,
if client request servers with alphabetical order, it could delay forever.

lishuai87 · 2017-08-25T03:04:02Z

CheckQuorum can prevent disruptive servers, which described in thesis section 4.2.3.

es-chow · 2017-08-30T02:52:54Z

Refer to raft thesis 9.6 Preventing disruptions when a server rejoins the cluster.
it seems Pre-Vote is good for network partition or jitter. For example.

Assume cluster have 3 nodes [A, B, C], the network between C and A,B is not stable, the link will jitter every hour.
If without Pre-vote, node C will election periodically to make the cluster leader step down. But if with Pre-Vote, node A/B will work as node C cannot increase term to step down cluster leader.

checkQuorum seems good for disruptive server which had been removed from the cluster, but it don't resolve the issue of network jitter.

So it's good to enable both checkQuorum and Pre-vote.

We discovered some instances where prevote and check quorum interact poorly. This commit introduces a test showcasing this. Related to etcd-io/etcd#8334.

* tests: Introduce prevote with check quorum test. We discovered some instances where prevote and check quorum interact poorly. This commit introduces a test showcasing this. Related to etcd-io/etcd#8334. * raft: Introduce prevote/check_quorum compatability fix. * raft: Always reset leader in precandidate

robin865 · 2018-11-09T19:41:32Z

So is this ever going to get merged? We've hit this issue in production, where with checkQuorum and preVote activated we are in an endless loop of ignoring MsgPreVotes. 3 nodes. Leader goes down. Other two endlessly try to PreVote and keep ignoring each other. The randomization in "pastElectionTimeout" which I think is supposed to prevent this is not working.

lishuai87 · 2018-11-21T09:48:00Z

So is this ever going to get merged? We've hit this issue in production, where with checkQuorum and preVote activated we are in an endless loop of ignoring MsgPreVotes. 3 nodes. Leader goes down. Other two endlessly try to PreVote and keep ignoring each other. The randomization in "pastElectionTimeout" which I think is supposed to prevent this is not working.

or maybe this: #8346

robin865 · 2018-11-21T15:36:20Z

No, that is not what I am seeing. In that scenario, B and C grant each other's votes. In my scenario B ignores Cs vote and C ignores B's vote. Also, we have that change in the version we are using. I am fairly confident we are seeing the issue described here related to the lease not expiring. So I am wondering if we can get these changes merged as they are fairly small, and have been sitting here for over a year. On Nov 21, 2018 1:48 AM, ls <[email protected]> wrote: So is this ever going to get merged? We've hit this issue in production, where with checkQuorum and preVote activated we are in an endless loop of ignoring MsgPreVotes. 3 nodes. Leader goes down. Other two endlessly try to PreVote and keep ignoring each other. The randomization in "pastElectionTimeout" which I think is supposed to prevent this is not working. I don't think your problem is related to this RP. But it seems you've met this: #8346<#8346> — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#8334 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAvL8nte6qFehQqFeMNJtcDpFDPT-kF7ks5uxSGIgaJpZM4OnTvd>.

lishuai87 · 2018-11-22T02:04:36Z

@xiang90 PTAL

bdarnell · 2018-11-25T18:00:47Z

raft/raft.go

@@ -611,6 +611,11 @@ func (r *raft) becomePreCandidate() {
 	// but doesn't change anything else. In particular it does not increase
 	// r.Term or change r.Vote.
 	r.step = stepCandidate
+	if r.checkQuorum {


I don't think this should be based on r.checkQuorum. That implies that we are doing this because we know that the leader is failing its quorum check, but we can't know that. The leader could be in the majority side of the partition and still be acting as leader.

#9586 discussed setting r.lead = None in becomePreCandidate whether checkQuorum is true or not. I think that could work here by keeping things from getting permanently stuck, but it has the odd effect that the second (pre-)campaigner wins. For that reason I still think it would be better to modify the definition of inLease at line 711 by adding && r.state == StateFollower.

On the other hand, that's a more complex and invasive change, and I think this version is simple and safe. It makes sense that pre-candidates act like they don't know the leader (same as candidates). I was previously concerned that it would be unusual for a node to transition between r.lead set and unset within a single term, but leaders already do that for checkQuorum so it should be fine.

bdarnell

Approved, but I think it would be better to set r.lead = None in becomeCandidate unconditionally, instead of only when r.checkQuorum is enabled.

There's also a jenkins failure that I can't see (any more?); maybe it will go away if the branch is updated.

codecov-io · 2018-12-28T05:22:23Z

Codecov Report

Merging #8334 into master will decrease coverage by 0.07%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #8334      +/-   ##
==========================================
- Coverage   71.84%   71.77%   -0.08%     
==========================================
  Files         391      391              
  Lines       36417    36418       +1     
==========================================
- Hits        26164    26139      -25     
- Misses       8438     8464      +26     
  Partials     1815     1815

Impacted Files	Coverage Δ
raft/raft.go	`92.16% <100%> (+0.01%)`	⬆️
proxy/grpcproxy/register.go	`69.44% <0%> (-13.89%)`	⬇️
etcdctl/ctlv3/command/lease_command.go	`65.34% <0%> (-5.95%)`	⬇️
etcdserver/v2_server.go	`80.76% <0%> (-3.85%)`	⬇️
etcdserver/api/v3rpc/lease.go	`69.31% <0%> (-2.28%)`	⬇️
raft/progress.go	`94.17% <0%> (-1.95%)`	⬇️
lease/leasehttp/http.go	`63.97% <0%> (-1.48%)`	⬇️
etcdserver/api/v3rpc/watch.go	`82.35% <0%> (-1.31%)`	⬇️
etcdserver/api/rafthttp/peer.go	`78.77% <0%> (-1.12%)`	⬇️
etcdserver/cluster_util.go	`58.74% <0%> (-0.9%)`	⬇️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6937b77...23731bf. Read the comment docs.

lishuai87 · 2018-12-28T12:04:47Z

@bdarnell I've updated the test case.

xiang90 · 2018-12-28T17:51:01Z

lgtm

jkessler93 · 2019-01-14T19:42:34Z

@lishuai87 Would it be possible for you to release a new version with this fix? I am currently experiencing this bug.

lishuai87 · 2019-01-15T03:09:47Z

@lishuai87 Would it be possible for you to release a new version with this fix? I am currently experiencing this bug.

@xiang90 @gyuho PTAL

brk0v · 2019-01-18T12:00:15Z

+1. Can reproduce this with my jepsen test as well.

raft: cherry pick of #8334 to release-3.3

* tests: Introduce prevote with check quorum test. We discovered some instances where prevote and check quorum interact poorly. This commit introduces a test showcasing this. Related to etcd-io/etcd#8334. * raft: Introduce prevote/check_quorum compatability fix. * raft: Always reset leader in precandidate

lishuai87 force-pushed the shawnsli/reset-leader-when-become-pre-candidate branch from fa9c6f7 to bf883bc Compare July 30, 2017 09:09

lishuai87 changed the title ~~set lead = None when becomePreCandidate~~ raft: introduce/fix TestPreVoteWithCheckQuorum Jul 30, 2017

lishuai87 force-pushed the shawnsli/reset-leader-when-become-pre-candidate branch from bf883bc to 22fb6a3 Compare July 31, 2017 17:19

xiang90 added the area/raft label Aug 1, 2017

xiang90 reviewed Aug 1, 2017

View reviewed changes

lishuai87 mentioned this pull request Aug 2, 2017

[DNM] raft: enable Pre-Vote tikv/tikv#2110

Closed

bdarnell reviewed Aug 7, 2017

View reviewed changes

irfansharif mentioned this pull request Aug 7, 2017

storage: re-enable Raft PreVote RPC cockroachdb/cockroach#16950

Closed

bdarnell reviewed Aug 24, 2017

View reviewed changes

irfansharif mentioned this pull request Sep 5, 2017

raft: lower test coverage for {CheckQuorum,PreVote} permutations #8498

Closed

bdarnell mentioned this pull request Apr 20, 2018

No leader using prevote feature #9586

Closed

Hoverbear mentioned this pull request Jul 3, 2018

Make peace between Prevote and Check Quorum tikv/raft-rs#84

Merged

bdarnell reviewed Nov 25, 2018

View reviewed changes

nolouch mentioned this pull request Dec 19, 2018

pd cluster become unhealthy when node fail happened pingcap/tidb-operator#236

Closed

nolouch mentioned this pull request Dec 26, 2018

Prevote never elect a new leader #10355

Closed

bdarnell approved these changes Dec 28, 2018

View reviewed changes

lishuai87 force-pushed the shawnsli/reset-leader-when-become-pre-candidate branch 2 times, most recently from f35ab44 to ae62732 Compare December 28, 2018 04:14

lishuai87 force-pushed the shawnsli/reset-leader-when-become-pre-candidate branch from ae62732 to f720bc4 Compare December 28, 2018 11:41

raft: set lead to none when becomePreCandidate

23731bf

lishuai87 force-pushed the shawnsli/reset-leader-when-become-pre-candidate branch from f720bc4 to 23731bf Compare December 28, 2018 11:59

xiang90 merged commit cc8d446 into etcd-io:master Dec 28, 2018

lishuai87 deleted the shawnsli/reset-leader-when-become-pre-candidate branch December 29, 2018 11:36

sbenderli pushed a commit to sbenderli/etcd that referenced this pull request Mar 21, 2019

raft: cherry pick of etcd-io#8334 to release-3.3

ec22eb9

sbenderli mentioned this pull request Mar 21, 2019

raft: cherry pick of #8334 to release-3.3 #10570

Merged

weekface mentioned this pull request May 24, 2019

stability test: PD cluster failed to start when we stop the kube node of a PD Pod pingcap/tidb-operator#512

Closed

gyuho added a commit that referenced this pull request Jul 23, 2019

Merge pull request #10570 from sbenderli/cherry-pick-of-#8334

789ff21

raft: cherry pick of #8334 to release-3.3

hexfusion pushed a commit to openshift/etcd that referenced this pull request Sep 26, 2019

raft: cherry pick of etcd-io#8334 to release-3.3

83469e1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

raft: introduce/fix TestPreVoteWithCheckQuorum #8334

raft: introduce/fix TestPreVoteWithCheckQuorum #8334

lishuai87 commented Jul 29, 2017 •

edited

Loading

xiang90 commented Jul 29, 2017

lishuai87 commented Jul 30, 2017

lishuai87 commented Jul 31, 2017 •

edited

Loading

xiang90 Aug 1, 2017

xiang90 Aug 1, 2017

xiang90 Aug 1, 2017

xiang90 commented Aug 1, 2017

bdarnell Aug 7, 2017

lishuai87 Aug 8, 2017 •

edited

Loading

xiang90 Aug 8, 2017

lishuai87 Aug 25, 2017

lishuai87 commented Aug 24, 2017

lishuai87 commented Aug 24, 2017

bdarnell left a comment

bdarnell Aug 24, 2017

lishuai87 Aug 25, 2017 •

edited

Loading

lishuai87 commented Aug 25, 2017

es-chow commented Aug 30, 2017 •

edited

Loading

robin865 commented Nov 9, 2018

lishuai87 commented Nov 21, 2018 •

edited

Loading

robin865 commented Nov 21, 2018 via email

lishuai87 commented Nov 22, 2018

bdarnell Nov 25, 2018

bdarnell left a comment

codecov-io commented Dec 28, 2018 •

edited

Loading

lishuai87 commented Dec 28, 2018

xiang90 commented Dec 28, 2018

jkessler93 commented Jan 14, 2019

lishuai87 commented Jan 15, 2019

brk0v commented Jan 18, 2019 •

edited

Loading

raft: introduce/fix TestPreVoteWithCheckQuorum #8334

raft: introduce/fix TestPreVoteWithCheckQuorum #8334

Conversation

lishuai87 commented Jul 29, 2017 • edited Loading

xiang90 commented Jul 29, 2017

lishuai87 commented Jul 30, 2017

lishuai87 commented Jul 31, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiang90 commented Aug 1, 2017

Choose a reason for hiding this comment

lishuai87 Aug 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lishuai87 commented Aug 24, 2017

lishuai87 commented Aug 24, 2017

bdarnell left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lishuai87 Aug 25, 2017 • edited Loading

Choose a reason for hiding this comment

lishuai87 commented Aug 25, 2017

es-chow commented Aug 30, 2017 • edited Loading

robin865 commented Nov 9, 2018

lishuai87 commented Nov 21, 2018 • edited Loading

robin865 commented Nov 21, 2018 via email

lishuai87 commented Nov 22, 2018

Choose a reason for hiding this comment

bdarnell left a comment

Choose a reason for hiding this comment

codecov-io commented Dec 28, 2018 • edited Loading

Codecov Report

lishuai87 commented Dec 28, 2018

xiang90 commented Dec 28, 2018

jkessler93 commented Jan 14, 2019

lishuai87 commented Jan 15, 2019

brk0v commented Jan 18, 2019 • edited Loading

lishuai87 commented Jul 29, 2017 •

edited

Loading

lishuai87 commented Jul 31, 2017 •

edited

Loading

lishuai87 Aug 8, 2017 •

edited

Loading

lishuai87 Aug 25, 2017 •

edited

Loading

es-chow commented Aug 30, 2017 •

edited

Loading

lishuai87 commented Nov 21, 2018 •

edited

Loading

codecov-io commented Dec 28, 2018 •

edited

Loading

brk0v commented Jan 18, 2019 •

edited

Loading