Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry cluster join on "too many learners" error #7351

Merged
merged 1 commit into from
Apr 28, 2023

Conversation

brandond
Copy link
Member

@brandond brandond commented Apr 25, 2023

Proposed Changes

Retry cluster join on "too many learners" error

Types of Changes

bugfix

Verification

See linked issue

Testing

Linked Issues

User-Facing Change

K3s now retries the cluster join operation when receiving a "too many learners" error from etcd. This most frequently occurred when attempting to add multiple servers at the same time.

Further Comments

With this change, when adding multiple servers at the same time, you should now see this repeat:

INFO[0011] Adding member k3s-server-3-88058c08=https://172.17.0.6:2380 to etcd cluster [k3s-server-1-c18966c7=https://172.17.0.4:2380 k3s-server-2-8b00056d=https://172.17.0.5:2380]
{"level":"warn","ts":"2023-04-25T23:12:35.335Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b03340/172.17.0.4:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: too many learner members in cluster"}
INFO[0011] Waiting for other members to finish joining etcd cluster

instead of K3s exiting with a fatal error:

INFO[0001] Adding member k3s-server-3-300fe049=https://172.17.0.6:2380 to etcd cluster [k3s-server-2-47159b14=https://172.17.0.5:2380 k3s-server-1-75ea6829=https://172.17.0.4:2380]
{"level":"warn","ts":"2023-04-25T21:05:26.878Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00056f880/172.17.0.4:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: too many learner members in cluster"}
FATA[0001] ETCD join failed: etcdserver: too many learner members in cluster

@brandond brandond requested a review from a team as a code owner April 25, 2023 22:37
@brandond brandond force-pushed the handle-too-many-learners branch from 1a727d5 to fa9baf2 Compare April 25, 2023 22:48
@brandond brandond force-pushed the handle-too-many-learners branch from fa9baf2 to 1d0f7cc Compare April 25, 2023 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants