start race: x509: certificate is valid for <ips>, not <ip>

[medyagh]

making issue out of @tstromberg conversations n slack:

as seen in integration tests ( the parallel version)

https://storage.googleapis.com/minikube-builds/logs/4966/Linux-KVM.txt seems to indicate that tests may be running kubelet commands against the wrong minikube context (race condition?).

   Temporary Error: error running command [--context=TestContainerd replace -f testdata/nginx-untrusted.yaml --force]: exit status 1. Stdout: 
             Unable to connect to the server: x509: certificate is valid for 192.168.39.145, 10.96.0.1, 10.0.0.1, not 192.168.39.105

more errors

        kubectl_runner.go:83: Temporary Error: error running command [--context=TestContainerd replace -f testdata/nginx-untrusted.yaml --force]: exit status 1. Stdout: 
             Unable to connect to the server: x509: certificate is valid for 192.168.39.145, 10.96.0.1, 10.0.0.1, not 192.168.39.105
             
        containerd_test.go:95: creating untrusted nginx resource: Temporary Error: error running command [--context=TestContainerd replace -f testdata/nginx-untrusted.yaml --force]: exit status 1. Stdout: 
             Unable to connect to the server: x509: certificate is valid for 192.168.39.145, 10.96.0.1, 10.0.0.1, not 192.168.39.105
             
            Temporary Error: error running command [--context=TestContainerd replace -f testdata/nginx-untrusted.yaml --force]: exit status 1. Stdout: 
             Unable to connect to the server: x509: certificate is valid for 192.168.39.145, 10.96.0.1, 10.0.0.1, not 192.168.39.105
             

I feel like it’s cleanest for certs to be per-profile, to avoid raciness. Apparently in that log, one IP is for TestDocker and the other TestContainerd

The issue appears to be two profiles using the same certs.

[tstromberg]

Even with locking in generateCerts, this is still an issue. Perhaps because the IP information from a previous unlocked bit of code. Here's a recent failure:

https://storage.googleapis.com/minikube-builds/logs/5341/VirtualBox_Linux.txt

        I0913 08:03:48.016591   10758 certs.go:61] Setting up certificates for IP: 192.168.99.103
        I0913 08:03:48.016624   10758 certs.go:135] acquiring lock: {Name:generateCerts Clock:{} Delay:10s Timeout:0s Cancel:<nil>}
        I0913 08:03:48.017030   10758 crypto.go:69] Generating cert /home/jenkins/minikube-integration/linux-amd64-virtualbox-5341-7993-6097f758dfd8b9da573df2e2cd8fb2593b1aab13/.minikube/client.crt with IP's: []
...
        W0913 08:07:02.072005   10758 exit.go:99] Error starting cluster: timed out waiting to elevate kube-system RBAC privileges: creating clusterrolebinding: Post https://192.168.99.103:8443/apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings?timeout=1m0s: x509: certificate is valid for 192.168.99.102, 10.96.0.1, 10.0.0.1, not 192.168.99.103

[tstromberg]

This appears to be gone?

[priyawadhwa]

Seen in Cloud Shell:

Description:
When running "Cloud Code: Run on Kubernetes", with minikube as context. I get an error message that kubectl create can't connect to the server




Running "Run/Debug on Kubernetes" configuration (.vscode/launch.json) in Run mode
Running: skaffold dev -v info --port-forward --rpc-http-port 40051 --filename skaffold.yaml
starting gRPC server on port 50051
Using kubectl context: minikube
Listing files to watch...
 - go-hello-world
List generated in 6.066462ms
Generating tags...
 - go-hello-world -> go-hello-world:latest
Checking cache...
Tags generated in 105.186µs
 - go-hello-world: Found Locally
Cache check complete in 10.737784ms
Tags used in deployment:
 - go-hello-world -> go-hello-world:a85f9e4910ec6b4a1dfe32cb0b72c6824ca37b8a2544517506b1424cd45c38d7
exiting dev mode because first deploy failed: unable to connect to Kubernetes: Get "https://[REDACTED_IP_ADDRESS]/version?timeout=32s": x509: certificate is valid for 172.18.0.2, 10.96.0.1, 127.0.0.1, 10.0.0.1, not 172.18.0.3
Exited with code 1.
To disable cleaning up resources, set cleanUp to false in your launch configuration '/home/grappeggia/cloudcode-projects/go-hello-world-2/.vscode/launch.json'
Cleaning up...
reading manifests: kubectl create: running [kubectl --context minikube create --dry-run=client -oyaml -f /home/grappeggia/cloudcode-projects/go-hello-world-2/kubernetes-manifests/hello.deployment.yaml -f /home/grappeggia/cloudcode-projects/go-hello-world-2/kubernetes-manifests/hello.service.yaml]
 - stdout: ""
 - stderr: "Unable to connect to the server: x509: certificate is valid for 172.18.0.2, 10.96.0.1, 127.0.0.1, 10.0.0.1, not 172.18.0.3\n"
 - cause: exit status 1

Running minikube delete && minikube start resolved this issue

[medyagh]

this could be fixed by this #7756

[medyagh]

I haven't seen this issue in a while, I am gonna close this till I am convineced it it is not fixed

[cabaumgartner]

I still see this problem with minikube v1.16.0 on Windows 10 with Hyper V. I can reproduce it by stopping and starting minikube. Every time I stop and restart minikube, it has a different IP address. Based on the error message, I would expect to see it every time, but that doesn't happen. Some times minikube restarts and some times it doesn't. Is there any way to work around this?