Better documentation needed for virtual desktop

[afbjorklund]

More and more users are trying to run minikube on a virtual desktop (VM).
Typically this fails, because nested virtualization is not enabled for them...

vagrant@ubuntu-xenial:~$ egrep --color 'vmx|svm' /proc/cpuinfo
vagrant@ubuntu-xenial:~$ wget -q https://github.com/kubernetes/minikube/releases/download/v1.2.0/minikube_1.2.0.deb
vagrant@ubuntu-xenial:~$ sudo dpkg -i minikube_1.2.0.deb 
(Reading database ... 54233 files and directories currently installed.)
Preparing to unpack minikube_1.2.0.deb ...
Unpacking minikube (1.2.0) ...
Setting up minikube (1.2.0) ...
vagrant@ubuntu-xenial:~$ minikube start
😄  minikube v1.2.0 on linux (amd64)
💿  Downloading Minikube ISO ...
 129.33 MB / 129.33 MB [============================================] 100.00% 0s
🔥  Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
E0711 16:56:33.992057    1778 start.go:559] StartHost: create: precreate: VBoxManage not found. Make sure VirtualBox is installed and VBoxManage is in the path

💣  Unable to start VM
❌  Error:         [VBOX_NOT_FOUND] create: precreate: VBoxManage not found. Make sure VirtualBox is installed and VBoxManage is in the path
💡  Advice:        Install VirtualBox, ensure that VBoxManage is executable and in path, or select an alternative value for --vm-driver
📘  Documentation: https://www.virtualbox.org/wiki/Downloads
⁉️   Related issues:
    ▪ https://github.com/kubernetes/minikube/issues/3784
    ▪ https://github.com/kubernetes/minikube/issues/3776

😿  If the above advice does not help, please let us know: 
👉  https://github.com/kubernetes/minikube/issues/new
vagrant@ubuntu-xenial:~$ sudo apt install -y virtualbox
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  binutils cpp cpp-5 dkms fakeroot fontconfig fontconfig-config fonts-dejavu-core gcc gcc-5 libasan2 libasound2 libasound2-data libasyncns0
  libatomic1 libavahi-client3 libavahi-common-data libavahi-common3 libboost-filesystem1.58.0 libboost-system1.58.0 libc-dev-bin libc6-dev
  libcaca0 libcapnp-0.5.3 libcc1-0 libcilkrts5 libcups2 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libegl1-mesa libevdev2
  libfakeroot libflac8 libfontconfig1 libgbm1 libgcc-5-dev libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libgomp1 libgraphite2-3 libgsoap8
  libgudev-1.0-0 libharfbuzz0b libice6 libinput-bin libinput10 libisl15 libitm1 libjpeg-turbo8 libjpeg8 libllvm6.0 liblsan0 libmirclient9
  libmircommon7 libmircore1 libmirprotobuf3 libmpc3 libmpx0 libmtdev1 libogg0 libpciaccess0 libpcre16-3 libprotobuf-lite9v5 libproxy1v5
  libpulse0 libqt5core5a libqt5dbus5 libqt5gui5 libqt5network5 libqt5opengl5 libqt5printsupport5 libqt5svg5 libqt5widgets5 libqt5x11extras5
  libquadmath0 libsdl1.2debian libsensors4 libsm6 libsndfile1 libtsan0 libtxc-dxtn-s2tc0 libubsan0 libvncserver1 libvorbis0a libvorbisenc2
  libvpx3 libwacom-bin libwacom-common libwacom2 libwayland-client0 libwayland-server0 libx11-xcb1 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0
  libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-present0 libxcb-randr0 libxcb-render-util0 libxcb-render0 libxcb-shape0 libxcb-shm0
  libxcb-sync1 libxcb-util1 libxcb-xfixes0 libxcb-xkb1 libxcursor1 libxdamage1 libxfixes3 libxi6 libxinerama1 libxkbcommon-x11-0
  libxkbcommon0 libxmu6 libxrender1 libxshmfence1 libxt6 libxxf86vm1 linux-libc-dev make manpages-dev qttranslations5-l10n virtualbox-dkms
  virtualbox-qt x11-common
Suggested packages:
  binutils-doc cpp-doc gcc-5-locales gcc-multilib autoconf automake libtool flex bison gdb gcc-doc gcc-5-multilib gcc-5-doc libgcc1-dbg
  libgomp1-dbg libitm1-dbg libatomic1-dbg libasan2-dbg liblsan0-dbg libtsan0-dbg libubsan0-dbg libcilkrts5-dbg libmpx0-dbg libquadmath0-dbg
  libasound2-plugins alsa-utils glibc-doc cups-common pulseaudio libthai0 libqt5libqgtk2 qt5-image-formats-plugins qtwayland5 lm-sensors
  make-doc vde2 virtualbox-guest-additions-iso
The following NEW packages will be installed:
  binutils cpp cpp-5 dkms fakeroot fontconfig fontconfig-config fonts-dejavu-core gcc gcc-5 libasan2 libasound2 libasound2-data libasyncns0
  libatomic1 libavahi-client3 libavahi-common-data libavahi-common3 libboost-filesystem1.58.0 libboost-system1.58.0 libc-dev-bin libc6-dev
  libcaca0 libcapnp-0.5.3 libcc1-0 libcilkrts5 libcups2 libdrm-amdgpu1 libdrm-intel1 libdrm-nouveau2 libdrm-radeon1 libegl1-mesa libevdev2
  libfakeroot libflac8 libfontconfig1 libgbm1 libgcc-5-dev libgl1-mesa-dri libgl1-mesa-glx libglapi-mesa libgomp1 libgraphite2-3 libgsoap8
  libgudev-1.0-0 libharfbuzz0b libice6 libinput-bin libinput10 libisl15 libitm1 libjpeg-turbo8 libjpeg8 libllvm6.0 liblsan0 libmirclient9
  libmircommon7 libmircore1 libmirprotobuf3 libmpc3 libmpx0 libmtdev1 libogg0 libpciaccess0 libpcre16-3 libprotobuf-lite9v5 libproxy1v5
  libpulse0 libqt5core5a libqt5dbus5 libqt5gui5 libqt5network5 libqt5opengl5 libqt5printsupport5 libqt5svg5 libqt5widgets5 libqt5x11extras5
  libquadmath0 libsdl1.2debian libsensors4 libsm6 libsndfile1 libtsan0 libtxc-dxtn-s2tc0 libubsan0 libvncserver1 libvorbis0a libvorbisenc2
  libvpx3 libwacom-bin libwacom-common libwacom2 libwayland-client0 libwayland-server0 libx11-xcb1 libxcb-dri2-0 libxcb-dri3-0 libxcb-glx0
  libxcb-icccm4 libxcb-image0 libxcb-keysyms1 libxcb-present0 libxcb-randr0 libxcb-render-util0 libxcb-render0 libxcb-shape0 libxcb-shm0
  libxcb-sync1 libxcb-util1 libxcb-xfixes0 libxcb-xkb1 libxcursor1 libxdamage1 libxfixes3 libxi6 libxinerama1 libxkbcommon-x11-0
  libxkbcommon0 libxmu6 libxrender1 libxshmfence1 libxt6 libxxf86vm1 linux-libc-dev make manpages-dev qttranslations5-l10n virtualbox
  virtualbox-dkms virtualbox-qt x11-common
0 upgraded, 131 newly installed, 0 to remove and 22 not upgraded.
Need to get 88.0 MB of archives.
After this operation, 474 MB of additional disk space will be used.
[...]
Loading new virtualbox-5.1.38 DKMS files...
First Installation: checking all kernels...
Building only for 4.4.0-151-generic
Building initial module for 4.4.0-151-generic
[...]
vagrant@ubuntu-xenial:~$ minikube start
😄  minikube v1.2.0 on linux (amd64)
🔥  Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
E0711 17:00:32.729865   21889 start.go:559] StartHost: create: precreate: This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory

💣  Unable to start VM
❌  Error:         [VBOX_VTX_DISABLED] create: precreate: This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory
💡  Advice:        In some environments, this message is incorrect. Try 'minikube start --no-vtx-check'
⁉️   Related issues:
    ▪ https://github.com/kubernetes/minikube/issues/3900

😿  If the above advice does not help, please let us know: 
👉  https://github.com/kubernetes/minikube/issues/new
vagrant@ubuntu-xenial:~$ minikube start --no-vtx-check
😄  minikube v1.2.0 on linux (amd64)
🔥  Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
E0711 17:02:31.248432   21909 start.go:559] StartHost: create: creating: Unable to start the VM: /usr/bin/VBoxManage startvm minikube --type headless failed:
VBoxManage: error: VT-x is not available (VERR_VMX_NO_VMX)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole

Details: 00:00:00.252851 Power up failed (vrc=VERR_VMX_NO_VMX, rc=NS_ERROR_FAILURE (0X80004005))

💣  Unable to start VM
❌  Error:         [VBOX_VERR_VMX_NO_VMX] create: creating: Unable to start the VM: /usr/bin/VBoxManage startvm minikube --type headless failed:
VBoxManage: error: VT-x is not available (VERR_VMX_NO_VMX)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole

Details: 00:00:00.252851 Power up failed (vrc=VERR_VMX_NO_VMX, rc=NS_ERROR_FAILURE (0X80004005))
💡  Advice:        Please check your BIOS, and ensure that you are running without HyperV or other nested virtualization that may interfere
⁉️   Related issues:
    ▪ https://github.com/kubernetes/minikube/issues/1994

😿  If the above advice does not help, please let us know: 
👉  https://github.com/kubernetes/minikube/issues/new

We should inform about this situation, and maybe even try to detect it too ?
Then we can come up with a better solution suggestion than "reboot your PC"

[afbjorklund]

Normally the best solution is to create a new VM, that is: a sibling, rather than a child
For this new VM (dedicated to minikube), it is then possible to use the none driver.

https://github.com/kubernetes/minikube/blob/master/docs/vmdriver-none.md

The remaining part to do is configuring the desktop VM to talk to the minikube VM.
Like figuring out the IP (as in: not 127.0.0.1) and setting up the kubeconfig/context

Update: this can done, by using the "generic" driver rather than the "none" driver

[afbjorklund]

As long as we require systemd on the host, we can use systemd-detect-virt
https://www.freedesktop.org/software/systemd/man/systemd-detect-virt.html

It returns "none" on a real machine, and something like "oracle" on VirtualBox.
This is of course only relevant on Linux, and only if the program can be found.

[afbjorklund]

Maybe we should add support for the "generic" driver as well ?
But that kinda implies we have to provision docker and friends

• the "none" machine driver runs everything on localhost
• the "generic" machine driver uses SSH on a provided IP

[medyagh]

I know that @josedonizetti is working on docs for remote qemu with KVM !, maybe we could it be called "Remote" driver ?

[medyagh]

it would be great to find out if they are running inside nested VM and add it to theoutput, so at least when they come for help, we know they are coming from nested VM, maybe we can detect it by something like this ? https://unix.stackexchange.com/questions/3685/find-out-if-the-os-is-running-in-a-virtual-environment

[afbjorklund]

I'm not sure remote qemu+tcp sessions are a good idea (without encryption), and Linux-only as well ?
So maybe #4069 wasn't a good idea, and #4635 seems to be struggling due to some XML syntax.

Will open a separate issue for supporting the "generic" driver, since that only requires a VM with ssh.
Then libmachine will provision docker on it (for multiple distros) and the bootstrapper will do the rest.

Compared with the setup above, it will be much better since it will set up the kube context and so on...
Also minikube ssh works out of the box, since you will have to provide the user/key/port needed.

It is still a much heavier, in terms of resources, and harder to set up solution than minikube start.
But at least the user don't have to bother with installing docker and kubeadm, and all features work.

[afbjorklund]

Added #4733 for "generic"

[afbjorklund]

For the scenario in the beginning, here is the output:

vagrant@ubuntu-xenial:~$  LANG=C lscpu | grep Virtualization
Virtualization type:   full
vagrant@ubuntu-xenial:~$ systemd-detect-virt 
oracle

Compared to the "normal" output from the laptop:

anders@anders-ThinkPad:~/ubuntu$ LANG=C lscpu | grep Virtualization
Virtualization:        VT-x
anders@anders-ThinkPad:~/ubuntu$ systemd-detect-virt 
none

Note the difference between "Virtualization" (host) and "Virtualization type" (guest)

Also I needed to set the locale, since the lscpu command is helpfully localized...

[afbjorklund]

We should also clarify this text:

Can the none driver be used outside of a VM?

Yes, but please avoid doing so if at all possible.

minikube was designed to run Kubernetes within a dedicated VM, and assumes that it has complete control over the machine it is executing on. With the none driver, minikube and Kubernetes run in an environment with very limited isolation, which could result in:

• Decreased security
• Decreased reliability
• Data loss

• https://github.com/kubernetes/minikube/blob/master/docs/vmdriver-none.md

It is actually not the VM that is the problem here, it is the lack of a dedicated VM.

If you decide to install the control-plane node directly on your development machine,
you still have all the risks above (security/reliability/data loss) in your virtual desktop.

And if you instead decide to do all your development as root on a headless master,
then you will probably have an inferior experience instead (no desktop, no dashboard).

So: currently minikube does not support running on the same machine, that is #4389

[afbjorklund]

Note how I failed at the first step of the installl instruction, but still continued until the bitter end ?

To check if virtualization is supported on Linux, run the following command and verify that the output is non-empty:
egrep --color 'vmx|svm' /proc/cpuinfo

This is typical of documentation, I guess you can't write it clear enough. Or double-check, in code.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[medyagh]

@afbjorklund
I am curious is this issue addressed in the new minikube website ?

[afbjorklund]

@medyagh : I don't think there is any major difference, between the old and the new (well, colors)

But it gets confusing already from the start (page 1). Like, which OS do you choose ?
Do you choose the OS of your client (Windows, say) or the OS of your VM (Linux) ?

How do you "verify your system has virtualization support" ? Nested virtualization ?
Pretty much the same situation as described above. (using KVM2 doesn't really help)

So most people go for the none driver... And then you need to do all the configuration yourself.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[afbjorklund]

Users are still having problems with this

/remove-lifecycle rotten

[sharifelgamal]

/lifecycle frozen