nvidia-driver-installer failed to build the driver

[ensonic]

We are following the instructions under https://minikube.sigs.k8s.io/docs/tutorials/nvidia_gpu/. We have minikube up and running and installed the addons. Both of them don't come up though.

repro

Follow the tutorial. Check that the addons run successfully:

kubectl get pods -n kube-system

The init-container of kubectl-driver-installer fails. logs here:
https://gist.github.com/ensonic/e23518399f040307597aac1f84aa7d47

Now this done not help a lot, so we copied /usr/local/nvidia/nvidia-installer.log:
https://gist.github.com/ensonic/ae9e466993a85b641ac3f6467d9642fd

3. Information to attach (optional if deemed irrelevant)

The graphics card is a NVIDIA Corporation GP102 [TITAN Xp] (rev a1). It is unbound from the via vfio-pci.
Minikube is the 1.5.2
Linux Kernel is 5.2.17, but the driver installer is using 4.19.76 (from minikube kvm2)

Can you pin to a driver version that builds?