[Feature Request] Enable host DNS resolution in virtualbox driver by default

[clocklear]

The default settings for the VirtualBox machine enables DNSProxy and disables HostDNSResolver. I'd like to propose reversing these settings. If the host is used for DNS resolution, we can enable the kubernetes cluster to resolve registry URLs like registry.kube-system.svc.cluster.local:80, assuming that the host has properly configured a cluster.local resolver that routes traffic to the minikube network.

Backstory

My local development workflow is something like this:

## Start our minikube and enable the local registry
minikube start --extra-config=apiserver.service-cluster-ip-range=10.96.0.0/24
minikube addons enable registry

## Set up local routing to the minikube
MINIKUBEIP=`minikube ip`
KUBEDNS=`kubectl get svc -o json kube-dns --namespace=kube-system | jq -r '.spec.clusterIP'`
route -n add 10.96.0.0/24 $MINIKUBEIP

cat << EOF | sudo tee /etc/resolver/cluster.local
nameserver $KUBEDNS
domain cluster.local
search_order 1
EOF

This allows for a rich local development experience, in which I can resolve cluster services by DNS without having to port-forward individual things (ie: http://kubernetes-dashboard.kube-system.svc.cluster.local). The problem comes when I push a new manifest to my kube installation that references containers I have pushed to my local registry. By default, since the VirtualBox host doesn't have HostDNSResolver enabled, it doesn't know how to resolve cluster.local, so the image pull ends up 404'ing. If we enable HostDNSResolver, the VB host will forward these resolution requests to my local machine, which will service them properly.

This should not cause any backwards-incompatible issues (so far as I can tell) and should be a drop-in replacement for what folks are already using. I am, of course, open to putting this behind some sort of runtime configuration flag.

I've made this change in my own fork of minikube and it works great for my use case. I'd be happen to submit a PR.

[ceason]

The core problem here seems to be that minikube addons enable registry creates a registry that minikube can't actually use (because the docker daemon inside the VM isn't connected to kube-dns and therefore can't resolve the registry's hostname).

[balopat]

This is very close to what I'm using to demo now in Kubecon shortly, using hyperkit - I guess host resolution there works by default.

HostDNS resolution in VBox: I don't see any fundamental issues with enabling it, if you'd raise a PR that would be awesome, and we can go from there.

network route creation: minikube tunnel does exactly that, so that part is done: #3015

accessing the registry from within minikube (@ceason): pull/push should be enabled with the --insecure-registry flag for the docker daemon. For containerd it's a bit trickier (#3444) and I haven't tried it yet with CRI-O. The resolution works as soon as there is a host based DNS resolution and minikube tunnel is up (or you can create the network routes manually too). It's a bit backwards but from within the VM, name resolution would go outside the VM to the host, that is pointing to 10.96.0.10, that is resolving to kube-dns, that is going to give back a 10.x.x.x service IP and it does work.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[woodcockjosh]

/remove-lifecycle stale

[erickalmeidaptc]

I tried to use minikube tunnel on Ubuntu 18.10 and DNS resolution of services does not work.

Work only after I make this changes:

1. Create a file: /etc/systemd/resolved.conf.d/99.minikube.conf

[Resolve]
DNS=10.96.0.10
Domains=svc.cluster.local
Cache=yes

2. Altered file: /etc/nsswitch.conf
   from:

hosts:          files mdns4_minimal [NOTFOUND=return] dns myhostname

to:

hosts:          files mdns4_minimal dns [NOTFOUND=return] myhostname

3. sudo systemctl restart systemd-resolved;

$ dig +short kubernetes-dashboard.kube-system.svc.cluster.local
10.100.252.203

But after some time the process CoreDNS inside VM hangs all CPU. I found this issue maybe was related. coredns/coredns#2083

Anyone know how I can solve this?

[tstromberg]

Since #3453 was merged - what is remaining here?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.