Skip to content

NRG (2.11): Vote request cancels catchup, new leader could have rejected subsequent catchup messages#6072

Merged
derekcollison merged 1 commit intomainfrom
maurice/desync-with-vote-req-during-catchup
Nov 4, 2024
Merged

NRG (2.11): Vote request cancels catchup, new leader could have rejected subsequent catchup messages#6072
derekcollison merged 1 commit intomainfrom
maurice/desync-with-vote-req-during-catchup

Conversation

@MauriceVanVeen
Copy link
Copy Markdown
Member

@MauriceVanVeen MauriceVanVeen commented Nov 4, 2024

Antithesis triggered a case where:

  • a follower (S0) is behind and triggers catchup from the leader (S2)
  • another follower (S1) triggers leader election
  • the follower (S0) allows follower (S1) to become leader
  • the leader (S2) sends a new message, which new leader (S1) rejects because it's on a higher term
  • the follower (S0) now receives above message that new leader (S1) rejected, providing previous leader (S2) with quorum
  • the new leader (S1) rejected that message, so previous leader (S2) has now desynced since it applied the message

This is solved by canceling catchup once a leader election happens for a higher term. The new leader will then resume our catchup once it's elected.

Signed-off-by: Maurice van Veen github@mauricevanveen.com

@MauriceVanVeen MauriceVanVeen changed the title NRG: Vote request cancels catchup, new leader could have rejected subsequent catchup messages NRG (2.11): Vote request cancels catchup, new leader could have rejected subsequent catchup messages Nov 4, 2024
neilalexander
neilalexander reviewed Nov 4, 2024
View reviewed changes
@MauriceVanVeen
NRG: Vote request cancels catchup, new leader could have rejected sub…
0f7cf3d 
…sequent catchup messages

Signed-off-by: Maurice van Veen <github@mauricevanveen.com>
@MauriceVanVeen MauriceVanVeen force-pushed the maurice/desync-with-vote-req-during-catchup branch from 552658b to 0f7cf3d Compare November 4, 2024 13:35
neilalexander
neilalexander approved these changes Nov 4, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@neilalexander neilalexander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@neilalexander neilalexander marked this pull request as ready for review November 4, 2024 15:14
@neilalexander neilalexander requested a review from a team as a code owner November 4, 2024 15:14
derekcollison
derekcollison approved these changes Nov 4, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@derekcollison derekcollison left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@derekcollison derekcollison merged commit 633bb94 into main Nov 4, 2024
@derekcollison derekcollison deleted the maurice/desync-with-vote-req-during-catchup branch November 4, 2024 16:21
@neilalexander neilalexander mentioned this pull request Nov 25, 2024
Merged
neilalexander added a commit that referenced this pull request Nov 25, 2024
@neilalexander
Cherry-picks for 2.10.23-RC.5 (#6171)
24acb45 
Includes the following:

- #5661
- #5666
- #5671
- #5344
- #5684
- #5689
- #5691
- #5714
- #5717
- #5707
- #5792
- #5912
- #5957
- #5700
- #5975
- #5991
- #5987
- #6027
- #6038
- #6053
- #5848
- #6055
- #6056
- #6060
- #6061
- #6072
- #5832
- #6073
- #6107

Signed-off-by: Neil Twigg <neil@nats.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@derekcollison derekcollison derekcollison approved these changes

@neilalexander neilalexander neilalexander approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@MauriceVanVeen @derekcollison @neilalexander