profiles don't work with kvm2 driver?

[eric-am]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Please provide the following details:

Environment:

Minikube version (use minikube version): v0.24.1

• OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus)
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): kvm2
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): minikube-v0.23.6.iso
• Install tools:
• Others:

What happened:

It appears minikube profiles don't work with the kvm2 driver...?

./minikube start --profile any-changed-value does not work, failing with the error:
E1205 16:03:36.824791 12899 start.go:150] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Machine didn't return an IP after 120 seconds.
(virsh list --all does say a machine is up, and a reasonable-looking qemu-system-x86_64 process is in ps.)

./minikube start --profile minikube -- e.g. explicitly stating the default value -- works fine.

What you expected to happen:

Profiles should work.

How to reproduce it (as minimally and precisely as possible):

See above. I've done everything I can fathom to clean up qemu state (reboots, virsh destroy, virsh undefine, simple process kills, blowing away minikube's state, and pretty much all combinations of orders the above) -- it doesn't seem like anything stateful on my machine matters. I'm also having reproducible success with --profile minikube and consistent failure for any other --profile wat{1..20}.

It was a shot in the dark, but switching to --bootstrapper kubeadm also made no difference.

Output of minikube logs (if applicable):

E1205 16:03:36.824791 12899 start.go:150] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Machine didn't return an IP after 120 seconds.

Anything else do we need to know:

Happy to run more introspection commands, kinda new to kvm. (Have been running vbox minikube for quite a while...)

[dfedyk]

I found exactly the same problem kvm2 driver no profile defaults to a minikube VM. Any profile other than minikube results in a VM with that name - as seen in Virtual Machine manager but it hangs.
E0123 13:25:53.315659 12501 start.go:150] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Machine didn't return an IP after 120 seconds.
minikube version: v0.24.1 Ubuntu 16.04

[lakano]

Hi @eric-am / @don57hpe,
Did you found a workaround for this bug? It's reported since 3 months and still present in the last version. How do you actually manage your profiles please?

[eric-am]

I stopped trying to use minikube with kvm.

[maximebedard]

I think I have found the root cause of this issue, but I have no idea to fix it since we would need to manipulate the ISO used to boot the minikube image.

The issue has to do with the fact that both kvm2 domains uses the same hostname. If we look at the status file generated by KVM, we can see that dhcp leases generated for both VM worked, but shares the same hostname.

some-user@minikube-ci-test-vm1:~/go/src/k8s.io/minikube$ cat /var/lib/libvirt/dnsmasq/virbr1.status 
[
    {
        "ip-address": "192.168.39.17",
        "mac-address": "c0:d5:46:7b:7c:18",
        "hostname": "minikube",
        "client-id": "ff:e2:34:3f:3e:00:02:00:00:ab:11:14:27:15:bd:04:de:93:02",
        "expiry-time": 1523313221
    },
    {
        "ip-address": "192.168.39.3",
        "mac-address": "20:85:1c:8c:76:3c",
        "hostname": "minikube",
        "client-id": "ff:e2:34:3f:3e:00:02:00:00:ab:11:35:5a:0b:38:db:4c:81:7c",
        "expiry-time": 1523313286
    }
]

When booting a 2nd image, it polls indefinitely since the ip is always set to "" due to this bit of code:

minikube/pkg/drivers/kvm/network.go

Lines 135 to 141 in 8f91c37

	ipAddress := ""
	for _, status := range statusEntries {
	if status.Hostname == d.MachineName {
	ipAddress = status.IPAddress
	}
	}

.

If anyone has an idea on how we could change the hostname, I'm all in to implement a fix for it.

[r2d4]

@maximebedard I've actually done this before, so its doable. I can't find the branch right now, but the driver accepts the machine name, which is the profile. So it should be easy to plumb through here.

[maximebedard]

That's what I was looking for, but at first look I didn't find anything in libvirt that allowed me to change it. We currently use the profile name for the machine's domain, but the hostname is set in the image.

One way to do it would be to ssh into it when we have the IP, however since it's generated via DHCP, it gets a bit tricky. We could bruteforce and try every ip for that given hostname, but that does not feel right at all 😅 If you ever find that branch back, I'm pretty curious on how you've done it!

[maximebedard]

I've pushed a fix here, however now I'm getting an error in kubeadm init for the 2nd profile:

[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node minikube2 as master by adding a label and a taint
: running command: sudo /usr/bin/kubeadm init --config /var/lib/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests --ignore-preflight-errors=DirAvailable--data --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml --ignore-preflight-errors=FileAvailable--etc-kubernetes-manifests-etcd.yaml --ignore-preflight-errors=Swap
.: Process exited with status 1

[gbraad]

@maximebedard @lukeweber

The driver relies on a hostname to lookup the correct information, but seems something is either not correctly passed in, or not working as expected from the DHCP Request:

minikube/pkg/provision/buildroot.go

Lines 145 to 148 in f5d9455

	log.Debugf("setting hostname %q", p.Driver.GetMachineName())
	if err := p.SetHostname(p.Driver.GetMachineName()); err != nil {
	return err
	}

[shashidharatd]

I am still not able to bringup minikube using profile with v0.28.1 release. below is the log.

minikube start --vm-driver kvm2 --kubernetes-version v1.11.1 -p c1
Starting local Kubernetes v1.11.1 cluster...
Starting VM...
Downloading Minikube ISO
 160.27 MB / 160.27 MB [============================================] 100.00% 0s
E0718 10:09:49.025444    7671 start.go:174] Error starting host: Error creating host: Error creating machine: Error in driver during machine creation: Machine didn't return an IP after 120 seconds.

 Retrying.
E0718 10:09:49.025645    7671 start.go:180] Error starting host:  Error creating host: Error creating machine: Error in driver during machine creation: Machine didn't return an IP after 120 seconds
================================================================================
An error has occurred. Would you like to opt in to sending anonymized crash
information to minikube to help prevent future errors?
To opt out of these messages, run the command:
	minikube config set WantReportErrorPrompt false
================================================================================
Please enter your response [Y/n]: