From a79cf6ca2d6cf6eaf8d7d3b85859b96ff9bb9712 Mon Sep 17 00:00:00 2001 From: Rohit Agarwal Date: Tue, 26 Jun 2018 18:50:28 -0700 Subject: [PATCH] Add documentation. --- docs/README.md | 2 + docs/gpu.md | 115 +++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 117 insertions(+) create mode 100644 docs/gpu.md diff --git a/docs/README.md b/docs/README.md index 6b36ad3ba3d9..bdfd18a68eb2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -12,6 +12,8 @@ * **Caching Images** ([cache.md](cache.md)): Caching non-minikube images in minikube +* **GPUs** ([gpu.md](gpu.md)): Using NVIDIA GPUs on minikube + ### Installation and debugging * **Driver installation** ([drivers.md](drivers.md)): In depth instructions for installing the various hypervisor drivers diff --git a/docs/gpu.md b/docs/gpu.md new file mode 100644 index 000000000000..fecd2dbafc26 --- /dev/null +++ b/docs/gpu.md @@ -0,0 +1,115 @@ +# (Experimental) NVIDIA GPU support in minikube + +minikube has experimental support for using NVIDIA GPUs on Linux. + +## Using NVIDIA GPUs on minikube on Linux with `--vm-driver=kvm2` + +When using NVIDIA GPUs with the kvm2 vm-driver. We passthrough spare GPUs on the +host to the minikube VM. Doing so has a few prerequisites: + +- You must install the [kvm2 driver](drivers.md#kvm2-driver). If you already had + this installed make sure that you fetch the latest + `docker-machine-driver-kvm2` binary that has GPU support. + +- Your CPU must support IOMMU. Different vendors have different names for this + technology. Intel calls it Intel VT-d. AMD calls it AMD-Vi. Your motherboard + must also support IOMMU. + +- You must enable IOMMU in the kernel: add `intel_iommu=on` or `amd_iommu=on` + (depending to your CPU vendor) to the kernel command line. Also add `iommu=pt` + to the kernel command line. + +- You must have spare GPUs that are not used on the host and can be passthrough + to the VM. These GPUs must not be controlled by the nvidia/nouveau driver. You + can ensure this by either not loading the nvidia/nouveau driver on the host at + all or assigning the spare GPU devices to stub kernel modules like `vfio-pci` + or `pci-stub` at boot time. You can do that by adding the + [vendorId:deviceId](https://pci-ids.ucw.cz/read/PC/10de) of your spare GPU to + the kernel command line. For ex. for Quadro M4000 add `pci-stub.ids=10de:13f1` + to the kernel command line. Note that you will have to do this for all GPUs + you want to passthrough to the VM and all other devices that are in the IOMMU + group of these GPUs. + +- Once you reboot the system after doing the above, you should be ready to you + GPUs with kvm2. Run the following command to start minikube: + ``` + minikube start --vm-driver kvm2 --gpu + ``` + This command will check if all the above conditions are satisfied and + passthrough spare GPUs found on the host to the VM. + + If this succeeded, run the following commands: + ``` + minikube addons enable nvidia-gpu-device-plugin + minikube addons enable nvidia-driver-installer + ``` + This will install the NVIDIA driver on the VM. + +- If everything succeeded, you should be able to see `nvidia.com/gpu` in the + capacity: + ``` + kubectl get nodes -ojson | jq .items[].status.capacity + ``` + +### Where can I learn more about GPU passthrough? +See the excellent documentation at +https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF + +### Why are so many manual steps required to use GPUs with kvm2 on minikube? +These steps require elevated privileges which minikube doesn't run with and they +are disruptive to the host, so we decided to not do them automatically. + + +## Using NVIDIA GPU on minikube on Linux with `--vm-driver=none` + +NOTE: This approach used to expose GPUs here is different than the approach used +to expose GPUs with `--vm-driver=kvm2`. Please don't mix these instructions. + +- Install minikube. + +- Install the nvidia driver, nvidia-docker and configure docker with nvidia as + the default runtime. See instructions at + https://github.com/NVIDIA/nvidia-docker + +- Start minikube: + ``` + minikube start --vm-driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost + ``` + +- Install NVIDIA's device plugin: + ``` + kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.10/nvidia-device-plugin.yml + ``` + + +## Why does minikube not support NVIDIA GPUs on macOS? +VM drivers supported by minikube for macOS doesn't support GPU passthrough: +- [mist64/xhyve#108](https://github.com/mist64/xhyve/issues/108) +- [moby/hyperkit#159](https://github.com/moby/hyperkit/issues/159) +- [VirtualBox docs](http://www.virtualbox.org/manual/ch09.html#pcipassthrough) + +Also: +- For quite a while, all Mac hardware (both laptops and desktops) have come with + Intel or AMD GPUs (and not with NVIDIA GPUs). Recently, Apple added [support + for eGPUs](https://support.apple.com/en-us/HT208544), but even then all the + supported GPUs listed are AMD’s. + +- nvidia-docker [doesn't support + macOS](https://github.com/NVIDIA/nvidia-docker/issues/101) either. + + +## Why does minikube not support NVIDIA GPUs on Windows? +minikube suppports Windows host through Hyper-V or VirtualBox. + +- VirtualBox doesn't support PCI passthrough for [Windows + host](http://www.virtualbox.org/manual/ch09.html#pcipassthrough). + +- Hyper-V supports DDA (discrete device assignment) but [only for Windows Server + 2016](https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/plan/plan-for-deploying-devices-using-discrete-device-assignment) + +Since the only possibility of supporting GPUs on minikube on Windows is on a +server OS where users don't usually run minikube, we haven't invested time in +trying to support NVIDIA GPUs on minikube on Windows. + +Also, nvidia-docker [doesn't support +Windows](https://github.com/NVIDIA/nvidia-docker/issues/197) either.