k8s client forgets to use CA when API URL specified #1810

lukemarsden · 2016-08-17T10:52:26Z

One user found that due to their in-cluster networking, they had to specify --probe.kubernetes.api=https://loadbalanced-k8s-api.mycorp.com:8443. When this was enabled, the k8s client in the scope probe forgot about the CA cert that was available in the pod secrets, and errored out with x509: certificate signed by unknown authority.

Let's find a way to allow the user to specify that they should use the in-pod secret with an override on the API URL.

cc @2opremio @errordeveloper

The text was updated successfully, but these errors were encountered:

errordeveloper · 2016-08-17T11:21:49Z

We might want to provide a flag to trust the CA as well.

2opremio · 2016-08-17T11:26:47Z

We might want to provide a flag to trust the CA as well.

I am not sure we want to do that, it encourages insecure use.

2opremio · 2016-08-17T11:35:25Z

I think the simplest and most complete way to provide an off-cluster authentication scheme is to specify the kubeconfig file like kube2sky does: https://github.com/gravitational/kube2sky/blob/dmitri/etcd-tls/cmd/kube2sky/kube2sky.go#L66

The kubeconfig file could be mounted in the probes as a secret.

Otherwise we need to provide a gazillion arguments for the client certificate, server certificate, key etc ...

errordeveloper · 2016-08-17T11:36:01Z

I am not sure we want to do that, it encourages insecure use.

I agree. The reason I brought this up, is that from the UX perspective it may be sensible to mirror kubectl flags, as it would match user expectations.

Here are the flags it has in relation to how you connect to the API server:

      --certificate-authority="": Path to a cert. file for the certificate authority.
      --client-certificate="": Path to a client certificate file for TLS.
      --client-key="": Path to a client key file for TLS.
      --cluster="": The name of the kubeconfig cluster to use
      --context="": The name of the kubeconfig context to use
      --insecure-skip-tls-verify[=false]: If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure.
      --kubeconfig="": Path to the kubeconfig file to use for CLI requests.
      --password="": Password for basic authentication to the API server.
  -s, --server="": The address and port of the Kubernetes API server
      --token="": Bearer token for authentication to the API server.
      --user="": The name of the kubeconfig user to use
      --username="": Username for basic authentication to the API server.

On that note, our existing --probe.kubernetes.api flag doesn't match it as is, but anyhow... If we are adding new flags, it'd be good to consider mirroring flag names.

2opremio · 2016-08-17T11:37:42Z

Here are the flags it has in relation to how you connect to the API server:

Thanks, that's very complete but overkill IMHO. Won't a kubeconfig file do ?

errordeveloper · 2016-08-17T11:38:49Z

Won't a kubeconfig file do?

Yes, that a good idea. The kubeconfig file will contain all the flags there are, including insecure-skip-tls-verify and others. However, I am not sure if there is a kubecofig file in a pod.

errordeveloper · 2016-08-17T11:39:46Z

The kubeconfig file could be mounted in the probes as a secret.

Ah, I can be indeed.

mindfulmonk · 2016-08-17T11:42:15Z

👍 For mirroring kubectl flags

I like it how Prometheus handles this too:

   tls_config:
      insecure_skip_verify: true
    bearer_token: token_here

    kubernetes_sd_configs:
    - api_servers:
      - 'https://api_url:6443'
      in_cluster: false
      role: pod

      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_file: path_to_file/

Kubeconfig is doable, but more complicated to get started.

errordeveloper · 2016-08-17T11:47:04Z

@2opremio for the record, if you chose the kubeconfig route, here is how you initialise the client with explicit config path:

loadingRules := &clientcmd.ClientConfigLoadingRules{ExplicitPath: kube.ConfigPath}
configOverrides := &clientcmd.ConfigOverrides{}
kubeconfig := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(loadingRules, configOverrides)

config, err := kubeconfig.ClientConfig()
if err != nil {
    return err
}

Which wasn't obvious from the docs and took me a while to figure out. See the rest of the code for more context.

2opremio · 2016-08-17T11:53:37Z

Kubeconfig is doable, but more complicated to get started.

I am a bit reluctant to provide separate config flags since I only envision custom client configurations in rare cases and kubeconfig fulfills all the possibilites. In addition it is safer to provide the configurations as a k8s secret instead of flags.

Would it be too much of a hassle?

mindfulmonk · 2016-08-17T12:09:09Z

I usually use configmaps when the config is quite large and it is faster to update the configmap than relaunch the pods. In Scope case I don't think it would auto-reload on configmap/secret change(it requires relaunching all daemonsets).

For smaller configs and particularly daemonsets I prefer specifying flags.

I have to agree though this is an edge case.

2opremio · 2016-08-17T15:09:18Z

OK, I will bite the bullet and implement all the flags :)

mindfulmonk · 2016-08-18T09:02:35Z

Based on the testing this morning it looks like it doesn't respect the options anymore.

I run scope with:

- '--probe.kubernetes=true'
- '--probe.kubernetes.server=https://api_server_domain_name:6443'
- '--probe.kubernetes.insecure-skip-tls-verify=true

and get this in probe logs:

<probe> ERRO: 2016/08/18 08:58:31.860888 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Service: Get https://192.168.0.1:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.1:443: i/o timeout

2opremio · 2016-08-18T09:13:51Z

@mindfulmonk server expects an address + port (without the https part). Can you please try with api_server_domain_name:6443 ?

mindfulmonk · 2016-08-18T09:18:29Z

Without the https:// it still gives:

<probe> ERRO: 2016/08/18 09:17:25.793717 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Service: Get https://192.168.0.1:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host

2opremio · 2016-08-18T09:24:37Z

Uhm, let me take a deeper look.

2opremio · 2016-08-18T11:03:41Z

@mindfulmonk it seems that insecure-skip-tls-verify doesn't behave as we expect when being passed to the kubernetes library (preventing the server from being overriden). In the meantime, since there is some urgency for this (and while I dive deeper in the kubernetes libraries), would you mind trying out a kubeconfig file?

mindfulmonk · 2016-08-18T11:05:32Z

Sure, can you let me know the flag and what it expects as param?

2opremio · 2016-08-18T11:08:22Z

@mindfulmonk --probe.kubernetes.kubeconfig

2opremio · 2016-08-18T11:09:28Z

In fact, it's kubecconfig (rushed code, sorry) I will fix it in the PR but use that for now.

mindfulmonk · 2016-08-18T11:46:10Z

It is still trying to go through 192.168.0.1 (kubernetes service IP) and not the one specified.

<probe> INFO: 2016/08/18 11:41:51.789469 command line args: --logtostderr=true --mode=probe --no-app=true --probe.docker=true --probe.docker.bridge=docker0 --probe.kubernetes=true --probe.kubernetes.kubecconfig=/home/weave/kube/config 192.168.0.168:80
<probe> INFO: 2016/08/18 11:41:51.789596 probe starting, version 1bdff95, ID 5c5944169e2c220a
<probe> INFO: 2016/08/18 11:41:51.789622 command line: [scope-probe --mode probe --no-app --probe.docker.bridge=docker0 --probe.docker=true --probe.kubernetes=true --probe.kubernetes.kubecconfig=/home/weave/kube/config 192.168.0.168:80]
<probe> INFO: 2016/08/18 11:41:51.789677 publishing to: 192.168.0.168:80
<probe> ERRO: 2016/08/18 11:41:51.960543 Error checking version: Get https://checkpoint-api.weave.works/v1/check/scope-probe?arch=amd64&flag_kubernetes_enabled=true&os=linux&signature=PrhVwqhAG3yvlDJl0E4TWAH%2B7jr9eWuzE96uXdz9eew%3D&version=1bdff95: read tcp 10.93.4.161:46160->64.233.166.214:443: read: connection reset by peer
<probe> ERRO: 2016/08/18 11:41:52.073754 Error checking version: Get https://checkpoint-api.weave.works/v1/check/scope-probe?arch=amd64&flag_kubernetes_enabled=true&os=linux&signature=PrhVwqhAG3yvlDJl0E4TWAH%2B7jr9eWuzE96uXdz9eew%3D&version=1bdff95: read tcp 10.93.4.161:46162->64.233.166.214:443: read: connection reset by peer
<probe> ERRO: 2016/08/18 11:41:52.794869 Error fetching app details: Get http://192.168.0.168:80/api: dial tcp 192.168.0.168:80: getsockopt: connection refused
<probe> ERRO: 2016/08/18 11:41:54.840141 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Pod: Get https://192.168.0.1:443/api/v1/pods?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> INFO: 2016/08/18 11:41:54.840164 Deployments and ReplicaSets are not supported by this Kubernetes version
<probe> ERRO: 2016/08/18 11:41:54.840271 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Service: Get https://192.168.0.1:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:41:54.840343 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Node: Get https://192.168.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:41:54.840432 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.ReplicationController: Get https://192.168.0.1:443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> WARN: 2016/08/18 11:41:54.840668 Error collecting weave status, backing off 10s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: getsockopt: connection refused
<probe> ERRO: 2016/08/18 11:41:54.842408 plugins: problem loading: no such file or directory
<probe> WARN: 2016/08/18 11:41:54.846061 Cannot resolve scope.weave.local.: dial tcp 172.16.1.1:53: getsockopt: connection refused
<probe> WARN: 2016/08/18 11:41:55.355700 Error collecting weave ps, backing off 20s: exit status 1
<probe> ERRO: 2016/08/18 11:41:57.795414 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
<probe> ERRO: 2016/08/18 11:42:01.795786 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
<probe> WARN: 2016/08/18 11:42:04.841815 Error collecting weave status, backing off 20s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: getsockopt: connection refused
<probe> ERRO: 2016/08/18 11:42:05.796141 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
<probe> ERRO: 2016/08/18 11:42:07.847056 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Node: Get https://192.168.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:07.847139 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Service: Get https://192.168.0.1:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:07.847194 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Pod: Get https://192.168.0.1:443/api/v1/pods?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:07.847221 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.ReplicationController: Get https://192.168.0.1:443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:11.798488 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
<probe> WARN: 2016/08/18 11:42:15.913566 Error collecting weave ps, backing off 40s: exit status 1
<probe> WARN: 2016/08/18 11:42:16.829439 background /proc reader: full pass took 15.006123214s: 50% more than expected (10s)
<probe> ERRO: 2016/08/18 11:42:20.856066 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Node: Get https://192.168.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:20.856101 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Service: Get https://192.168.0.1:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:20.856162 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Pod: Get https://192.168.0.1:443/api/v1/pods?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:20.856203 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.ReplicationController: Get https://192.168.0.1:443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:21.799139 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
<probe> WARN: 2016/08/18 11:42:24.842609 Error collecting weave status, backing off 40s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: getsockopt: connection refused
<probe> ERRO: 2016/08/18 11:42:31.798693 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
<probe> ERRO: 2016/08/18 11:42:33.868533 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.ReplicationController: Get https://192.168.0.1:443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:33.868677 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Node: Get https://192.168.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:33.868757 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Service: Get https://192.168.0.1:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:33.868820 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Pod: Get https://192.168.0.1:443/api/v1/pods?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:41.798850 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
<probe> ERRO: 2016/08/18 11:42:46.874910 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Node: Get https://192.168.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:46.874941 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.ReplicationController: Get https://192.168.0.1:443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:46.875003 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Service: Get https://192.168.0.1:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:46.875020 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Pod: Get https://192.168.0.1:443/api/v1/pods?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:51.798960 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
<probe> WARN: 2016/08/18 11:42:56.452054 Error collecting weave ps, backing off 1m0s: exit status 1
<probe> ERRO: 2016/08/18 11:42:59.880931 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Pod: Get https://192.168.0.1:443/api/v1/pods?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:59.881061 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Service: Get https://192.168.0.1:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:59.881146 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.ReplicationController: Get https://192.168.0.1:443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:42:59.881160 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Node: Get https://192.168.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:01.799194 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
<probe> WARN: 2016/08/18 11:43:04.848856 Error collecting weave status, backing off 1m0s: Get http://127.0.0.1:6784/report: dial tcp 127.0.0.1:6784: getsockopt: connection refused
<probe> ERRO: 2016/08/18 11:43:11.799681 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
<probe> ERRO: 2016/08/18 11:43:12.893236 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.ReplicationController: Get https://192.168.0.1:443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:12.893234 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Node: Get https://192.168.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:12.893331 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Service: Get https://192.168.0.1:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:12.893373 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Pod: Get https://192.168.0.1:443/api/v1/pods?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:21.799594 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
<probe> ERRO: 2016/08/18 11:43:25.899041 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Pod: Get https://192.168.0.1:443/api/v1/pods?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:25.899050 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Service: Get https://192.168.0.1:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:25.899157 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.ReplicationController: Get https://192.168.0.1:443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:25.899207 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Node: Get https://192.168.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:31.799757 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
<probe> ERRO: 2016/08/18 11:43:38.907590 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Pod: Get https://192.168.0.1:443/api/v1/pods?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:38.907590 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Service: Get https://192.168.0.1:443/api/v1/services?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:38.907690 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.ReplicationController: Get https://192.168.0.1:443/api/v1/replicationcontrollers?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:38.907692 Kubernetes reflector: github.com/weaveworks/scope/probe/kubernetes/client.go:164: Failed to list *api.Node: Get https://192.168.0.1:443/api/v1/nodes?resourceVersion=0: dial tcp 192.168.0.1:443: getsockopt: no route to host
<probe> ERRO: 2016/08/18 11:43:41.800004 Error fetching app details: Get http://192.168.0.168:80/api: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

2opremio · 2016-08-18T12:00:54Z

Alright, let's stop rushing. I will look into this properly before bothering you any further. Sorry for the inconvenience.

2opremio · 2016-08-18T15:12:17Z

@mindfulmonk It should work now both providing kubeconfig and insecure-skip-tls-verify.

Please test with weaveworks/scope:1810-extend-k8s-client-flags-4b0f152.

For the record: It was a nasty problem in config merging, partly due to Go's recursive vendoring limitations. I had to fetch a kubernetes dependency (https://github.com/imdario/mergo) for the new flags to work. But instead of vendoring the exact same version kubernetes uses, I vendored master, which is buggy (darccio/mergo#33 ) causing the flags to be incorrectly merged (specifically the target host was wrong).

Sorry for the confusion.

2opremio · 2016-08-18T15:19:43Z

BTW, I was wrong about this:

@mindfulmonk server expects an address + port (without the https part). Can you please try with api_server_domain_name:6443 ?

It works perfectly fine with https://api_server_domain_name:6443 (otherwise it won't know the transport needs to be TLS).

2opremio · 2016-08-18T22:33:41Z

Closed via #1813

lukemarsden added k8s Pertains to integration with Kubernetes feature Indicates that issue is related to new end user functionality labels Aug 17, 2016

rade added bug Broken end user or developer functionality; not working as the developers intended it and removed feature Indicates that issue is related to new end user functionality labels Aug 17, 2016

rade added this to the August2016 milestone Aug 17, 2016

2opremio modified the milestones: 0.17.1, August2016 Aug 17, 2016

2opremio mentioned this issue Aug 17, 2016

Extend kubernetes client flags to match kubectl #1813

Merged

2opremio closed this as completed Aug 18, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

k8s client forgets to use CA when API URL specified #1810

k8s client forgets to use CA when API URL specified #1810

lukemarsden commented Aug 17, 2016 •

edited by errordeveloper

Loading

errordeveloper commented Aug 17, 2016

2opremio commented Aug 17, 2016

2opremio commented Aug 17, 2016 •

edited

Loading

errordeveloper commented Aug 17, 2016

2opremio commented Aug 17, 2016

errordeveloper commented Aug 17, 2016

errordeveloper commented Aug 17, 2016

mindfulmonk commented Aug 17, 2016

errordeveloper commented Aug 17, 2016 •

edited

Loading

2opremio commented Aug 17, 2016

mindfulmonk commented Aug 17, 2016

2opremio commented Aug 17, 2016

mindfulmonk commented Aug 18, 2016

2opremio commented Aug 18, 2016 •

edited

Loading

mindfulmonk commented Aug 18, 2016

2opremio commented Aug 18, 2016

2opremio commented Aug 18, 2016 •

edited

Loading

mindfulmonk commented Aug 18, 2016

2opremio commented Aug 18, 2016

2opremio commented Aug 18, 2016 •

edited

Loading

mindfulmonk commented Aug 18, 2016

2opremio commented Aug 18, 2016 •

edited

Loading

2opremio commented Aug 18, 2016 •

edited

Loading

2opremio commented Aug 18, 2016

2opremio commented Aug 18, 2016 •

edited

Loading

k8s client forgets to use CA when API URL specified #1810

k8s client forgets to use CA when API URL specified #1810

Comments

lukemarsden commented Aug 17, 2016 • edited by errordeveloper Loading

errordeveloper commented Aug 17, 2016

2opremio commented Aug 17, 2016

2opremio commented Aug 17, 2016 • edited Loading

errordeveloper commented Aug 17, 2016

2opremio commented Aug 17, 2016

errordeveloper commented Aug 17, 2016

errordeveloper commented Aug 17, 2016

mindfulmonk commented Aug 17, 2016

errordeveloper commented Aug 17, 2016 • edited Loading

2opremio commented Aug 17, 2016

mindfulmonk commented Aug 17, 2016

2opremio commented Aug 17, 2016

mindfulmonk commented Aug 18, 2016

2opremio commented Aug 18, 2016 • edited Loading

mindfulmonk commented Aug 18, 2016

2opremio commented Aug 18, 2016

2opremio commented Aug 18, 2016 • edited Loading

mindfulmonk commented Aug 18, 2016

2opremio commented Aug 18, 2016

2opremio commented Aug 18, 2016 • edited Loading

mindfulmonk commented Aug 18, 2016

2opremio commented Aug 18, 2016 • edited Loading

2opremio commented Aug 18, 2016 • edited Loading

2opremio commented Aug 18, 2016

2opremio commented Aug 18, 2016 • edited Loading

lukemarsden commented Aug 17, 2016 •

edited by errordeveloper

Loading

2opremio commented Aug 17, 2016 •

edited

Loading

errordeveloper commented Aug 17, 2016 •

edited

Loading

2opremio commented Aug 18, 2016 •

edited

Loading

2opremio commented Aug 18, 2016 •

edited

Loading

2opremio commented Aug 18, 2016 •

edited

Loading

2opremio commented Aug 18, 2016 •

edited

Loading

2opremio commented Aug 18, 2016 •

edited

Loading

2opremio commented Aug 18, 2016 •

edited

Loading