feat(leader-election): leader election with multiple replicas support

[ivankatliarchuk]

First step to resolve #2430

Description

Proposal: #5051

Reason to close: discussion with maintainers/owners

A naive implementation for a leader election, based on official example. Could split this pull request to ease review

Note; testing in progress. Feel free to comment, suggest changes

The proposal is to release leader election functionality and iterate/add improvements based on feedback. Flag had a suffix -experimental to make sure it 100% clear that logic or implementation may change at some point. Would be nice to collect feedback from community and iterate.

For leader election to be fully functional-opearational the number of bugs/currently open issues may need to be resolved as well. But this this not sound like a blocker.

kubernetes-sigs leader-election configured for

• https://github.com/search?q=org%3Akubernetes-sigs%20k8s.io%2Fclient-go%2Ftools%2Fleaderelection&type=code

kuberntes org leader-elections configured for

• https://github.com/search?q=org%3Akubernetes%20k8s.io%2Fclient-go%2Ftools%2Fleaderelection&type=code

In case we think that external-dns do not require HA setup with leader election or library is not fit for purpose. Same leader-election library used in (implementation sometimes differ). Most of this services have always enabled leader election funtionality

• discheduler
• scheduler
• cluster-api
• aws-load-balancer-contoller
• certificate-manager
• ingress-nginx
• vertical-pod-autoscaler
• autoscaler
• pod-autoscaler-recommender
• cloud-provider-openstak
• kubernetes controller
• ingress-gce
• sig-storage-provisioner
• cloud-provider-azure
• karpenter
• all csi-drivers
• cluster-api-provider-aws

Checklist

• how to test on real cluster
• Unit tests updated
• Check if metrics still available. curl localhost:7979/metrics OK
• Check healthcheck still working. curl localhost:7979/healthz OK
• End user documentation updated

some checks
running leader election (leader identified)

standby replica

leader identify changed

lease updated or leader change

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign mloiseleur for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ivankatliarchuk]

/retest-required

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ivankatliarchuk]

/close not-planned