Can skaffold accept https proxy for running kubectl command?

[dshetty312]

Expected behavior

Sample app deployed to GKE cluster

Actual behavior

1.FATA[0011] reading manifests: kubectl create: Running [kubectl --context create --dry-run -oyaml -f k8s-pod.yaml]: stdout , stderr: Unable to connect to the server: net/http: TLS handshake timeout
, err: exit status 1: exit status 1

Information

• Skaffold version: v1.0.0
• Operating system: Mac
• Contents of skaffold.yaml:

apiVersion: skaffold/v1
kind: Config
build:
  artifacts:
  - image: <image-registry>/skaffold-example
deploy:
  kubectl:
    manifests:
      - k8s-*

Steps to reproduce the behavior

1.skaffold run

My analysis:

Due to security constraints i cannot directly run kubectl commands from my laptop.
But if i run https_proxy=$K8API_PROXY_IP:PORT kubectl get all , i get back the output.

So i was wondering if there was a way for me to let skaffold know about this https_proxy before running kubectl command.

Regards,
DS

[briandealwis]

Setting HTTPS_PROXY in the environment before invoke Skaffold works for me. I tried both proxying with a non-existent proxy (below) to show that it fails, and with a local proxy via ssh -D xxx which worked.

$ HTTPS_PROXY=nonexistent.example.org skaffold dev
Listing files to watch...
 - gcr.io/k8s-skaffold/skaffold-jib
Generating tags...
 - gcr.io/k8s-skaffold/skaffold-jib -> gcr.io/k8s-skaffold/skaffold-jib:v1.0.0-70-ga045e9d7f
Checking cache...
 - gcr.io/k8s-skaffold/skaffold-jib: Found Locally
Tags used in deployment:
 - gcr.io/k8s-skaffold/skaffold-jib -> gcr.io/k8s-skaffold/skaffold-jib:e54f9f370f16c00cb7c0e8ba45354fab55a37ff913997fcb207009d3e9b208ca
   local images can't be referenced by digest. They are tagged and referenced by a unique ID instead
Starting deploy...
Cleaning up...
WARN[0005] deployer cleanup: reading manifests: kubectl create: Running [kubectl --context minikube create --dry-run -oyaml -f /Users/bsd/Projects/GPE/repo-skaffold/examples/jib/k8s/web.yaml]: stdout , stderr: Unable to connect to the server: proxyconnect tcp: dial tcp: lookup nonexistent.example.org on 1.1.1.1:53: no such host
, err: exit status 1: exit status 1 
FATA[0005] exiting dev mode because first deploy failed: reading manifests: kubectl create: Running [kubectl --context minikube create --dry-run -oyaml -f /Users/bsd/Projects/GPE/repo-skaffold/examples/jib/k8s/web.yaml]: stdout , stderr: Unable to connect to the server: proxyconnect tcp: dial tcp: lookup nonexistent.example.org on 1.1.1.1:53: no such host
, err: exit status 1: exit status 1 

[balopat]

@dshetty312 would @briandealwis's solution be sufficient?

[dshetty312]

@briandealwis : Thank you. It works!

@balopat : Yes the solution works.

[ideasculptor]

This solution doesn't really work - other commands pick up that environment variable and use it (specifically, pretty much everything - curl, gcloud, things written in Go), and there is a high likelihood that some of those things will also be run by skaffold. Since my proxy is specific to running kubectl for one particular cluster, and is only useful for accessing the private ip address of that GKE endpoint, not anywhere else on the internet, I can have access to kubectl OR the gcloud cli, but never both. This is fairly problematic.

Nor is there a way to configure the proxy setting separately for kubectl within a kubectl config. Nor is there a way to tell gcloud to ignore the environment variable. There is just this assumption that if someone wants to configure a proxy, they want to configure it for every tool in the universe. Additionally, the NO_PROXY environment variable doesn't provide any way to specify everywhere with just a few exceptions. I want NO_PROXY to apply to everywhere EXCEPT 3 subnets plus variants of localhost, but there is no way to do that. There is probably some way to figure out every hostname the gcloud tool will try to access, but that only fixes the conflict with gcloud, not curl or any other tools which use the same environment variables.

Someone needs to come up with a proxy configuration mechanism that is tool specific, or at least provide tool-specific overrides of the environment vars.

[ideasculptor]

something that just allows control over custom environment variables when running kubectl would be sufficient. I don't mind modifying a skaffold file to include the extra environment var, but they have to apply just to kubectl, not any other commands.

[balopat]

Gotcha, I agree that this is hard to get around currently.
It would be great if kubectl would have --http(s)-proxy, then we could just use that in deploy.kubectl.global.
Because they don't I am open to have httpsProxy and httpProxy fields for kubectl. That should be relatively easy to implement, and would specify the HTTP_PROXY and HTTPS_PROXY env vars for all kubectl commands.

[balopat]

This will require introducing two new fields as a config change in latest.go and as such, a new unreleased version. Should be relatively easy to implement it. PRs are welcome!

[ideasculptor]

I appreciate that, but my go skills are basically non-existent, and I am already kept incredibly busy submitting multiple PRs every week for terraform-google-modules.

This is one that someone else is going to have to do. I already spend well more than half my time fixing/modifying terraform module behaviour rather than working on my own infrastructure. I can't add skaffold changes to my workload. Sometimes, I need a tool to just be a tool, not a project. Sorry.

[ideasculptor]

As it happens, running this seems to work in my particular case:

HTTPS_PROXY="localhost:8888" skaffold run --default-repo=gcr.io/...

If I set it as an environment var and then run skaffold as a separate command, that breaks things when it checks to see if the image already exists in the repo. The following does NOT work correctly:

$> export HTTPS_PROXY="locahost:8888"
$> skaffold run --default-repo=gcr.io/...

When I set it as a var in the same command that runs skaffold, it seems to magically do the right thing. I assume vars are inherited differently by forked processes depending on how they are declared or something. I've validated it repeatedly. Every once in a while, it errors out even when the var is declared inline, but then re-running it works. It doesn't make much sense to me, but it got me moving again, so I'm not arguing with it.

[tstromberg]

@ideasculptor - I agree, that does not seem to make sense to me. If you export a variable, it is passed down through forks.

FWIW, the correct format for this environment variable is:

• HTTPS_PROXY=https://10.0.0.1:8080/
• HTTPS_PROXY=https://USERNAME:[email protected]:8080/

[tvvignesh]

@tstromberg Thanks for the tip regarding the format. It was failing randomly without https:// prefix. Had a question. kubectl v1.19+ introduced proxy-url flag (kubernetes/client-go#351) to allow specifying proxy via config and it works great with kubectl. However, skaffold and some other tools fails with timeout like below:

exiting dev mode because first deploy failed: unable to connect to Kubernetes: Get "https://10.0.0.2/version?timeout=32s": dial tcp 10.0.0.2:443: i/o timeout

Any plans to support this? It would be great to have this.

[tvvignesh]

Not sure, but I guess skaffold is using an old kubectl binary. Probably upgrading to latest version v19.2+ might fix the issue ^^

[briandealwis]

Or we might be linking with an older client-go?

[briandealwis]

Just to confirm client-go's proxy-url seems to work.

[sandygvs]

I'm defining HTTPS_PROXY=https://10.0.0.1:8080/ in skaffold.env file along with skaffold.yaml configs will that help to set the proxy before execution. https://skaffold.dev/docs/environment/env-file/

However still getting the time out error unable to connect to Kubernetes: Get "https://10.0.0.2/version?timeout=32s": dial tcp 10.0.0.2:443: i/o timeout