multinode: minikube cluster NotReady after restart

[eachirei]

minikube start --driver=virtualbox -n 3 --extra-config=apiserver.service-node-port-range=1-65535
Steps to reproduce the issue:

1. minikube start --driver=virtualbox -n 3 --extra-config=apiserver.service-node-port-range=1-65535
2. minikube stop
3. minikube start --driver=virtualbox -n 3 --extra-config=apiserver.service-node-port-range=1-65535

Full output of failed command:
The command does not fail. The minikube primary kubelet config gets replaced by the one for my third node.
I have deleted the cluster multiple times and this behavior is constant.

Full output of minikube start command used, if not already included:
😄 minikube v1.11.0 on Darwin 10.15.5
▪ KUBECONFIG=******
✨ Using the virtualbox driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🔄 Restarting existing virtualbox VM for "minikube" ...
🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
▪ apiserver.service-node-port-range=1-65535
🔎 Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass, storage-provisioner
❗ The cluster minikube already exists which means the --nodes parameter will be ignored. Use "minikube node add" to add nodes to an existing cluster.
👍 Starting node minikube-m02 in cluster minikube
🔄 Restarting existing virtualbox VM for "minikube-m02" ...
🌐 Found network options:
▪ NO_PROXY=192.168.99.116
🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
▪ env NO_PROXY=192.168.99.116
👍 Starting node minikube-m03 in cluster minikube
🔄 Restarting existing virtualbox VM for "minikube-m03" ...
🌐 Found network options:
▪ NO_PROXY=192.168.99.116,192.168.99.117
🐳 Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
▪ env NO_PROXY=192.168.99.116
▪ env NO_PROXY=192.168.99.116,192.168.99.117
🏄 Done! kubectl is now configured to use "minikube"

Optional: Full output of minikube logs command:

[medyagh]

@sharifelgamal do you mind verifying if this issue is still going on ?

[sharifelgamal]

So where exactly are you seeing the issue? Is it with kubectl get nodes?

[eachirei]

Yes, kubectl get nodes indicates the master node is not ready. Upon accessing the node using minikube ssh -n minikube and running systemctl status kubelet the result shows the node is running with hostname override minikube-m03 and host ip the one assigned to the aforementioned minikube-m03. After modifying the configuration, I restart the docker service and the kubelet service, and delete any pods regarding dns or networking (those managed by deployments). I found out I had to delete those pods, especially the ones assigned to master, as the dns wouldn't work properly anymore (and the network one just to be safe). Things seem to act as normal, but I still had some unexplained issues with pods residing on the master node. I have to follow this procedure after each restart...

[sharifelgamal]

I have reproduced the issue with both hyperkit and virtualbox. We'll look more closely at it soon.

[sharifelgamal]

It seems to be specific to VM drivers for whatever reason.