nvidia-driver-installer: Unable to determine the device handle for GPU 0000:00:07.0: Unknown Error

[hendrikhalkow]

BUG REPORT

Environment: Fedora 29, minikube v0.30.0, kvm2 driver

Install minikube with gnu support according to https://github.com/kubernetes/minikube/blob/master/docs/gpu.md

minikube start --vm-driver kvm2 --gpu
minikube addons enable nvidia-gpu-device-plugin
minikube addons enable nvidia-driver-installer

# watch what happens in another terminal
watch -n1 kubectl get all --all-namespaces

# when the pod nvidia-driver-installer-xxx appears, look at the logs
kubectl logs nvidia-driver-installer-xxxxx --namespace kube-system -c nvidia-driver-installer -f

Instead of installing the graphics driver, the installation fails.

...
Configuring kernel sources... DONE
Running Nvidia installer...
/usr/local/nvidia /
Verifying archive integrity... OK
Uncompressing NVIDIA Accelerated Graphics Driver for Linux-x86_64 390.67.........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

WARNING: nvidia-installer was forced to guess the X library path '/usr/lib'
         and X module path '/usr/lib/xorg/modules'; these paths were not
         queryable from the system.  If X fails to find the NVIDIA X driver
         module, please install the `pkg-config` utility and the X.Org
         SDK/development package for your distribution and reinstall the
         driver.

/
Running Nvidia installer... DONE.
Updated cached version as:
CACHE_KERNEL_VERSION=4.15
CACHE_NVIDIA_DRIVER_VERSION=390.67
Verifying Nvidia installation...
Unable to determine the device handle for GPU 0000:00:07.0: Unknown Error
rpc error: code = Unknown desc = Error: No such container: 5b3b2079d9066f289f6a5ab2ed0b5744bbc12f2db87b7129d4883123e77673ac%

To confirm, lets compare the Minikube GPU with the host GPU. Both are with ID 10de:13b1:

$ minikube ssh 'lspci'
...
00:07.0 Class 0300: 10de:13b1

$ lspci -nnk -s 01:00.0
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107GLM [Quadro M1000M] [10de:13b1] (rev a2)
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau

[balopat]

/cc @mindprince

[alasdairtran]

PR #3459 might've fixed this issue.

[tstromberg]

@hendrikhalkow - Mind retesting with minikube v0.33?

[afbjorklund]

Retesting this with v1.0.0 would be nice.

minikube start --vm-driver kvm2 --gpu --hidden

[tstromberg]

I'm closing this issue as it hasn't seen activity in awhile, and it's unclear if this issue still exists. If this issue does continue to exist in the most recent release of minikube, please feel free to re-open it.

Thank you for opening the issue!