Add HTTP proxy setting to kubeconfig

[vjsamuel]

Whilst managing several Kubernetes clusters, there could be potential scenarios where certain clusters require the usage of an HTTP proxy to be accessible. It would be useful to have http-proxy as a per cluster setting on the kube config so that it is easy to talk to clusters that require and don't require the usage of a proxy.

[aakp10]

I've been looking into this issue ,I'd like to work on this

[aakp10]

Does this issue require to add something like this for the http-proxy , where details like the port and the ip on which the proxy server is running can be stored in a struct type and stored in the kubeconfig for the cluster.

[dlipovetsky]

@mikedanese Are you still working on this? This will be useful for the case where a Cluster API control plane needs to reach clusters via proxies..

[dlipovetsky]

Some details on our use case:

We will run Cluster API (CAPI) controllers in a management cluster (in our infrastructure), and create workload clusters in other infrastructures. The CAPI controllers (as of v1alpha2) require access the workload clusters' APIs. However, some workload clusters are deployed in a network that does not allow inbound connections. Therefore we are looking for a way to send requests from the CAPI controllers to the workload clusters' APIs that does not require inbound connections.

In general, there are many ways to solve this problem. We would like to find one that (a) requires no changes to CAPI controller code, and (b) no changes to the workload cluster.

The CAPI controllers use a kubeconfig (stored as a Secrets) to talk to each workload cluster's API. There is one kubeconfig for each workload cluster. If we modify the kubeconfig, we can modify how the CAPI controllers reach the workload cluster, without changing the controller code.

We can update the server property to point to a TCP tunnel. This requires we add the TCP tunnel hostname or IP to the cluster's kube-apiserver server cert's Subject Alt Names (SANs). That's a change we want to avoid making.

We can update the kubeconfig to point to an HTTP/S proxy. This requires no changes to the workload cluster's kube-apiserver server cert.

[dlipovetsky]

sig-apimachinery meeting discussion comments:

• HTTP proxy may need its own set of certificates
• We need to consider interaction between HTTP_PROXY env variable and each kubeconfig's proxy setting
• Is it possible to reuse https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190226-network-proxy.md
• Trade-off: Adding this to kubeconfig makes general use of kubeconfig more complicated.
• Is it possible to run a separate process (configured to use a specific HTTP proxy) to talk to each workload cluster
• Is it possible to provide a separate proxy configuration? (What work is required for CAPI?)

[mikedanese]

I wish I had been to the sig meeting.

• HTTP proxy may need its own set of certificates

Nope. https_proxy are authenticate via whatever is in the cert bundle. http_proxy and https_proxy need to trust the kube-apiserver but the client won't. socks5 doesn't use ssl between client and proxy, and the connection is e2e encrypted to the apiserver.

• Is it possible to reuse https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190226-network-proxy.md

Why would we ask everyone who wants to use a proxy to implement a custom protocol when e.g. ssh supports socks5 tunneling? This seems like a different FR.

• Is it possible to run a separate process (configured to use a specific HTTP proxy) to talk to each workload cluster

Why? We already support this with an env variable. The FR is to add an override to kubeconfig.

[dlipovetsky]

@mikedanese Thanks for following up here. I apologize you missed the meeting, I assumed you'd be there, but in hindsight I should have pinged you about it.

[jacobstr]

Wanted to add my user story here because I think @mikedanese MR would could nicely simplify our own workflows in GKE, but moving parts would still remain in order to make the experience seamless.

We're using private clusters in GKE. Right now, we're having to wrap kubectl config set-context with our own wrapper to also set the https_proxy environment variable out of band from the kubeconfig.

We're typically using gcloud container clusters get-credentials to configure local kubecontexts for our GKE clusters. In addition to using this command to manage my kubecontext, I need to know to construct the https_proxy url accordingly for each cluster.

Every time I switch into a new shell, I need to export this environment variable. Because I work with multiple clusters, no single value works. It also inadvertently affects any other network interactions I might have.

This makes using GKE private clusters slightly more painful than it ought to be. FWIW GKE's own documentation recommends running privoxy to act as an https_proxy to Kubernetes masters.

If kubernetes/kubernetes#81443 gets merged, it may be helpful to make the gcloud/gke folks aware of it, in order to update gcloud container clusters get-credentials accordingly. That might not be straightforward because they're asking users to deploy privoxy themselves, which means GKE has no control / predictability about what address it's actually going to be routable at. E.g. we're managing own DNS records when we spin up our GKE clusters.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[alexppg]

Nice, thanks! Do you know if this kind of things get backported or they only enter in the latest binaries?

[liggitt]

This will be 1.19+ only

[alexellis]

@liggitt re: your comment in May, is this feature now available in kubectl / KUBECONFIG from 1.19 onwards?

#351 (comment)

I have a use-case where I can use HTTP_PROXY with kubectl, but need to use a tool that reads and parses a kubeconfig instead, so I don't have a way to pass this in the usual env-var way http_proxy=http://localhost:3128 kubectl get nodes

[vickyvikas7988]

How do I set no_proxy in kube config?

[Trolldemorted]

@vickyvikas7988 did you find out how to do it?

[xurizaemon]

@Trolldemorted @vickyvikas7988 does setting the proxy per-cluster do what you need? kubernetes/website#32894

[Trolldemorted]

I want to use no_proxy, i.e. not use the proxy defined in environment variables, and use no proxy at all. What should I set the proxy to if I want no proxy to be used?

[xurizaemon]

Note that this issue is closed already.

According to https://github.com/kubernetes/kubernetes/blob/f02682c628c530219966a00ae002d799f0d813dc/staging/src/k8s.io/client-go/rest/config.go#L137-L139 the environment variable will take effect if the configuration is set to nil.

If Proxy is nil, http.ProxyFromEnvironment is used. If Proxy returns a nil *URL, no proxy is used.

That last sentence suggests you might be able to configure it from there by setting proxy-url to a "nil *URL"? Maybe try eg empty string.

If you can't do it from proxy-url, another option is to unset the environment variable before calling kubectl. (A wrapper function can ensure this always happens.)