Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unexpected election when one of followers is network partition with leader #9020

Closed
lhy1024 opened this issue Jan 26, 2025 · 0 comments · Fixed by #9021
Closed

unexpected election when one of followers is network partition with leader #9020

lhy1024 opened this issue Jan 26, 2025 · 0 comments · Fixed by #9021
Labels
type/enhancement The issue or PR belongs to an enhancement.

Comments

@lhy1024
Copy link
Contributor

lhy1024 commented Jan 26, 2025

Enhancement Task

An unexpected election occurs when one of the followers experiences a network partition from the leader.

If the etcd client is using a follower endpoint and encounters a failure when attempting to save a timestamp, the allocator_manager will reset the leader.

[2025/01/21 20:39:49.149 +08:00] [WARN] [lease.go:187] ["lease keep alive failed"] [purpose="leader election"] [start=2025/01/21 20:39:46.149 +08:00] [error="context deadline exceeded"]
[2025/01/21 20:39:53.032 +08:00] [WARN] [etcd_kv.go:180] ["txn runs too slow"] [response=null] [cost=5.721552451s] [error="rpc error: code = Unavailable desc = error reading from server: read tcp [10.200.26.212:34736](http://10.200.26.212:34736/)->[10.200.27.59:2379](http://10.200.27.59:2379/): read: connection timed out"]
[2025/01/21 20:39:53.032 +08:00] [WARN] [tso.go:333] ["save timestamp failed"] [] [timestamp-path=timestamp] [error="rpc error: code = Unavailable desc = error reading from server: read tcp [10.200.26.212:34736](http://10.200.26.212:34736/)->[10.200.27.59:2379](http://10.200.27.59:2379/): read: connection timed out"]
[2025/01/21 20:39:53.032 +08:00] [WARN] [allocator_manager.go:289] ["failed to update allocator's timestamp"] [] [name=tc-pd-1] [error="rpc error: code = Unavailable desc = error reading from server: read tcp [10.200.26.212:34736](http://10.200.26.212:34736/)->[10.200.27.59:2379](http://10.200.27.59:2379/): read: connection timed out"]

In fact, manual retry is needed, next request succeeds with the new endpoint, according to etcd-io/etcd#8711

@lhy1024 lhy1024 added the type/enhancement The issue or PR belongs to an enhancement. label Jan 26, 2025
@ti-chi-bot ti-chi-bot bot closed this as completed in #9021 Feb 5, 2025
ti-chi-bot bot added a commit that referenced this issue Feb 5, 2025
close #9020

Signed-off-by: lhy1024 <[email protected]>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant