k8s: allow setting multiple k8s API server addresses by fristonio · Pull Request #20090 · cilium/cilium

fristonio · 2022-06-06T22:58:47Z

See commit message for detailed description

Introduce a new command line parameter(--k8s-api-server-urls) and helm option(k8s.apiServerURLs) to specify multiple k8s API server addresses for the client to use.

Fixes: #19038

Allow setting of multiple k8s API server URLs for client

brb · 2022-06-07T10:02:36Z

@fristonio Welcome back! 😅 Would it be possible to set the new param from Helm? Currently, we use this hack to pass the API server endpoint addr - https://github.com/cilium/cilium/blob/master/install/kubernetes/cilium/templates/cilium-agent/daemonset.yaml#L193.

fristonio · 2022-06-07T19:52:39Z

Hey @brb 👋
Its good to be back. 😄
Yeah, I have some changes locally for the helm option that I need to test. Will update the PR soon.

fristonio · 2022-06-08T01:49:02Z

/test

sayboras

👋 super happy to see you around 🎖️

One small comment as per below, the rest looks reasonable to me.

operator/flags.go

qmonnet

Looks good, with a few nits - see below (plus a typo on --k8s-api-erver-addresses in your first commit log).

One question: I'm wondering if it would make sense to have “URLs” instead of “Addresses” in the flag (and Helm value) name? Reading “addresses” I'd first think of IP addresses, maybe URLs would make it more explicit?

Awesome to have you contributing again! ❤️

pkg/k8s/client.go

pkg/k8s/config.go

pkg/option/config.go

fristonio · 2022-06-08T18:15:26Z

Hey @qmonnet! 👋
Yeah, changing the CLI flag to use urls instead of addresses makes sense to me.
I have updated the PR and addressed the changes you requested.

daemon/cmd/daemon_main.go

operator/main.go

pkg/k8s/config.go

qmonnet

Good points raised by Tom, but I'm good for my own review. Thanks again! :)

fristonio · 2022-06-28T08:14:30Z

/test

Job 'Cilium-PR-K8s-GKE' hit: #17628 (93.58% similarity)

joestringer · 2022-07-05T18:45:02Z

Looks like this just needs rebase, address any CI failures, plus operator / cli codeowner review. If there's something particular you'd like me to look at, then let me know - otherwise I'll let the other core reviewers follow up.

* This commit introduces a new command line flag to specify multiple kubernetes addresses that can be used with clients. By default the client will pick the first APIServer address for setting up the client. During a failed heartbeat check for k8s API server we will rotate the API server to use with the client. * cilium-agent --k8s-api-server-urls=IP1,IP2,IP3 ... to use this feature. * The flag takes precedence over already existing `--k8s-api-server` flag. * The flag modfies the host address for client-go http.RoundTripper to use configured APIServerURL in k8s.config. The heartbeat thread rotates the APIServer URL when there is a failure causing subsequent request to be sent to the rotated host. Signed-off-by: Deepesh Pathak <deepeshpathak09@gmail.com>

* Use helm option `k8s.apiServerURLs` to set multiple k8s server urls for the client to use. * Example helm command ``` helm install cilium cilium/cilium \ --namespace kube-system \ --set k8s.apiServerURLs="https://172.18.0.4:6443 https://172.18.0.3:6443 https://172.18.0.5:6443" ``` Signed-off-by: Deepesh Pathak <deepeshpathak09@gmail.com>

Signed-off-by: Deepesh Pathak <deepeshpathak09@gmail.com>

* This commit adds APIServer URL rotation logic during initial client setup. * Update tests helper to use the new helm option `k8s.apiServerURLs` to specify Kubernetes API server address in kube-proxy free setup. Signed-off-by: Deepesh Pathak <deepeshpathak09@gmail.com>

fristonio · 2022-07-19T02:42:13Z

/test

aanm · 2022-07-29T14:37:46Z

pkg/k8s/config.go

+	config.APIServerURL = nil
+	config.APIServerURLs = []*url.URL{}


These fields are being accessed without being protected by their mutex

aanm · 2022-07-29T14:38:49Z

pkg/k8s/config.go

+type httpRoundTripper struct {
+	delegate http.RoundTripper
+}
+
+func (rt *httpRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
+	req.URL.Host = GetAPIServerURL().Host
+	return rt.delegate.RoundTrip(req)
+}
+
+func defaultHTTPRoundTripper(rt http.RoundTripper) http.RoundTripper {
+	return &httpRoundTripper{
+		delegate: rt,
+	}
+}


Can you leave a comment stating what this is doing?

aanm · 2022-07-29T14:40:41Z

pkg/option/config.go

+	if viper.IsSet(K8sAPIServer) {
+		if len(c.K8sAPIServerURLs) > 0 {
+			log.Warningf("The option %s has been deprecated in favour of %s. Ignoring %s: %s",
+				K8sAPIServer, K8sAPIServerURLs, K8sAPIServer, viper.GetString(K8sAPIServer))
+		} else {
+			c.K8sAPIServer = viper.GetString(K8sAPIServer)
+		}
+	}
+}


in the daemon/flags.go we should be marking the flag as deprecated.

christarazi · 2022-08-31T15:37:20Z

Moving to draft, please mark it ready for review when you are ready to pick it up.

squeed · 2022-08-31T16:39:37Z

pkg/k8s/config.go

 type configuration struct {
 	// APIServerURL is the URL address of the API server
-	APIServerURL string
+	APIServerURL *url.URL


Can this member be dropped?

github-actions · 2022-10-01T02:16:57Z

This pull request has been automatically marked as stale because it
has not had recent activity. It will be closed if no further activity
occurs. Thank you for your contributions.

github-actions · 2022-10-15T02:17:42Z

This pull request has not seen any activity since it was marked stale.
Closing.

maintainer-s-little-helper bot added the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Jun 6, 2022

fristonio added the area/daemon Impacts operation of the Cilium daemon. label Jun 6, 2022

fristonio added the release-note/minor This PR changes functionality that users may find relevant to operating Cilium. label Jun 7, 2022

maintainer-s-little-helper bot removed the dont-merge/needs-release-note-label The author needs to describe the release impact of these changes. label Jun 7, 2022

fristonio force-pushed the pr/fristonio/feat-19038 branch 2 times, most recently from d978a26 to dfc36a5 Compare June 7, 2022 23:15

fristonio marked this pull request as ready for review June 8, 2022 01:45

fristonio requested a review from a team June 8, 2022 01:45

fristonio requested a review from a team as a code owner June 8, 2022 01:45

fristonio requested a review from a team June 8, 2022 01:45

fristonio requested review from a team as code owners June 8, 2022 01:45

fristonio requested review from jibi, nathanjsweet, nebril, sayboras and twpayne June 8, 2022 01:45

sayboras reviewed Jun 8, 2022

View reviewed changes

operator/flags.go Show resolved Hide resolved

fristonio requested a review from a team as a code owner June 8, 2022 08:31

fristonio requested a review from qmonnet June 8, 2022 08:31

qmonnet requested changes Jun 8, 2022

View reviewed changes

pkg/k8s/client.go Outdated Show resolved Hide resolved

pkg/k8s/config.go Outdated Show resolved Hide resolved

pkg/k8s/config.go Outdated Show resolved Hide resolved

pkg/k8s/config.go Outdated Show resolved Hide resolved

pkg/option/config.go Outdated Show resolved Hide resolved

fristonio force-pushed the pr/fristonio/feat-19038 branch from 13aa021 to dd44542 Compare June 8, 2022 17:58

tommyp1ckles reviewed Jun 8, 2022

View reviewed changes

daemon/cmd/daemon_main.go Outdated Show resolved Hide resolved

operator/main.go Outdated Show resolved Hide resolved

pkg/k8s/config.go Outdated Show resolved Hide resolved

qmonnet approved these changes Jun 8, 2022

View reviewed changes

fristonio force-pushed the pr/fristonio/feat-19038 branch from dd44542 to 96f167d Compare June 8, 2022 22:04

fristonio force-pushed the pr/fristonio/feat-19038 branch from 53a2159 to e61c4bb Compare June 27, 2022 20:22

maintainer-s-little-helper bot mentioned this pull request Jun 28, 2022

CI: K8sDatapathConfig Host firewall With native routing #17628

Closed

joestringer added the dont-merge/needs-rebase This PR needs to be rebased because it has merge conflicts. label Jul 5, 2022

joestringer removed their request for review July 5, 2022 18:43

fristonio added 5 commits July 18, 2022 15:36

k8s: add k8s-api-server-urls flag to cilium-operator

0fd77a7

Signed-off-by: Deepesh Pathak <deepeshpathak09@gmail.com>

cmd: mark k8s-api-server flag as deprecated and update upgrade-guide

9fdf038

Signed-off-by: Deepesh Pathak <deepeshpathak09@gmail.com>

fristonio force-pushed the pr/fristonio/feat-19038 branch from e61c4bb to ea41042 Compare July 19, 2022 01:16

fristonio removed the dont-merge/needs-rebase This PR needs to be rebased because it has merge conflicts. label Jul 19, 2022

aanm requested changes Jul 29, 2022

View reviewed changes

twpayne removed their request for review August 14, 2022 22:44

christarazi marked this pull request as draft August 31, 2022 15:37

squeed reviewed Aug 31, 2022

View reviewed changes

github-actions bot added the stale The stale bot thinks this issue is old. Add "pinned" label to prevent this from becoming stale. label Oct 1, 2022

github-actions bot closed this Oct 15, 2022

brb mentioned this pull request Oct 18, 2023

k8sServiceHost multiple IPs/Hosts #19038

Closed

bittermandel mentioned this pull request Oct 23, 2023

[WIP] feat: support k8s multiple apiservers #28741

Closed

aditighag mentioned this pull request Feb 17, 2025

kpr: Support kube-apiserver HA #37601

Merged

This was referenced Feb 27, 2026

🌱 CNCF mission generation 2026-02-27 kubestellar/console-kb#6

Closed

🌱 CNCF mission generation 2026-02-27 kubestellar/console-kb#11

Merged

🌱 CNCF mission generation 2026-03-02 kubestellar/console-kb#251

Merged

clubanderson mentioned this pull request Mar 10, 2026

🌱 cilium: k8sServiceHost multiple IPs/Hosts kubestellar/console-kb#364

Merged

		config.APIServerURL = nil
		config.APIServerURLs = []*url.URL{}

Conversation

fristonio commented Jun 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

brb commented Jun 7, 2022

Uh oh!

fristonio commented Jun 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fristonio commented Jun 8, 2022

Uh oh!

sayboras left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

qmonnet left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fristonio commented Jun 8, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

qmonnet left a comment

Choose a reason for hiding this comment

Uh oh!

fristonio commented Jun 28, 2022 • edited by maintainer-s-little-helper bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joestringer commented Jul 5, 2022

Uh oh!

fristonio commented Jul 19, 2022

Uh oh!

aanm Jul 29, 2022

Choose a reason for hiding this comment

Uh oh!

aanm Jul 29, 2022

Choose a reason for hiding this comment

Uh oh!

aanm Jul 29, 2022

Choose a reason for hiding this comment

Uh oh!

christarazi commented Aug 31, 2022

Uh oh!

squeed Aug 31, 2022

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 1, 2022

Uh oh!

github-actions bot commented Oct 15, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

fristonio commented Jun 6, 2022 •

edited

Loading

fristonio commented Jun 7, 2022 •

edited

Loading

fristonio commented Jun 28, 2022 •

edited by maintainer-s-little-helper bot

Loading