proposal: Enable host OS dns resolution of cluster ingress host names

[woodcockjosh]

Background

Kubernetes offers the rich ability to run multiple instances of the same service in the the cluster. For example I can have 2 redis database instances with different service names. Currently there are 2 ways of accessing these services from the host machine when running on minikube:

1. If the service has a NodePort can use a server's NodePort capability using minikube service redis-1 then I can access the service via the minikube ip and service's NodePort

2. If the service is a LoadBalancer I can use minikube tunnel then access the service via redis-1.default.svc.cluster.local

3. If however my services are ClusterIP I can only access these services via an ingress. Thankfully minikube has an ingress plugin which I can enable via minikube addons enable ingress. This works great with 1 additional step. I have to add the dns entry configured for the ingress to my hosts /etc/hosts file. This is potentially not such a big deal if it is a small number of entries. If however I have a larger number of hosts like 20 for example then it starts to become problematic. This becomes especially more problematic if I have to do either of the 2 things:

   1. Delete and re-create my minikube instance which will issue a new ip for the cluster. This can be potentially solved with the resolution of this issue: Support for predictable IP's across restarts #951

   2. If the hostnames for my cluster are updated or added. If I am a sole developer of a project updating or adding a host name for the cluster with a predictable ip address is somewhat trivial. However if the number of developers is more than a handful then keeping track of and reminding developers to update their hosts is fairly annoying and slows development efforts.

Why does hostname resolution especially matter on minikube?

Some capabilities in k8s can only be tested when using a domain name. An example would be if I am using a router like istio or traefik that matches inbound requests to a particular domain name.

When running a development cluster that has some type of server, I can use external dns to automatically configure my DNS to point to the right server. However, when I am using minikube I can only resolve host names locally since the cluster ip is only available to my Host OS.

Possible solutions to host ingress dns resolution

1. Create an external tool that will that runs a service on the host OS which will synchronize minikube ip addresses with /etc/hosts file and a configurable set of domain names. Updating the /etc/hosts file can be somewhat dangerous since it could potentially overwrite or confuse existing configurations on the development machine as well as run into lock-writing issues with other applications which might also try to write to /etc/hosts on the host OS. In general writing to /etc/hosts is frowned upon. If this is selected as a solution, there are 2 ways to determine which set of hosts to add to the /etc/hosts file.

   1. Have a centralized or decentralized set of configurations which could be installed in ~/.minikube/profiles/profilename
   2. Use a background service on the host OS that would watch the ingress resources for the minikube clusters and dynamically add and update entries in the /etc/hosts file.

   While feasible, IMO it would be better to include these capabilities within minikube itself either with a plugin or within the minikube core codebase. Currently developers are creating similar solutions due to the gap of minikube's management of host resolvers for example:

• https://www.mailgun.com/blog/creating-development-environments-with-kubernetes-devgun
  This solution doesn't have any links to the source of the authors code. Based on the authors final comments devgun is perhaps not yet ready to be released as open source. This could potentially be a good replacement tool for minikube tunnel. It is also unknown whether it will solve the problem of ingress dns resolution.
• https://github.com/superbrothers/minikube-ingress-dns/blob/master/README.md
  This repo seems to have generally the right idea with using dnsmasq but does not seem to be currently maintained. It also has some key capabilities missing like automatically identifying which hostnames to add to /etc/resolver. The restart logic is also written in ruby which I don't have anything personally against but go lang is perhaps a better solution for code that runs on the host OS.

2. Use minikube as a DNS server for and have a DNS server which will resolve the DNS queries for ingress hosts to the IP address of the minikube instance. This is the most preferable solution since the ingress resources within the cluster would be limited to the cluster itself. For DNS resolution the A record for each DNS query would be the hostname of the ingress and the ip address would be the cluster ip. There is a working solution that currently does this here:
   https://gitlab.com/cryptexlabs/public/development/minikube-ingress-dns

3. Use a service like nip.io or xip.io to resolve a generic hostname to an IP address. This is workable but it doesn't provide a great user experience since the IP address of minikube is constantly changing everytime it is recreated. Even if we can solve the problem of a constantly changing ip, developers also would have to memorize an obscure IP address converted to a domain. This also creates a dependency on having an internet connection even though the rest of my services can be run independent of an internet connection.

Host DNS resolution with /etc/resolver/

With solution #3 being the most preferable, this still leaves a problem with setting the minikube instance as DNS server for the host OS with an entry in /etc/resolver/. Preferably each file would be start with minikube and be followed by the profile name. For example /etc/resolver/minikube-profilename. This way profiles don't potentially overwrite one another. In the case of overlapping dns entries per profile the first dns server to resolve the domain name would be the winner. If that profile is deleted. then only resolver file for that profile would also be deleted and the dns query would fall to the next profile.

There are 2 aspects which need consideration for a resolver configuration file.

1. The hostnames not including subdomains to resolve
2. The IP addresses to resolve to

1. Hostname list for resolver file

There are 2 good solutions for determining the list of host names that could be resolved to a minikube ip

1. Only allow the .test TLD. While this is the simplest and makes the most sense to me, we could end up with feature request to enable other domain names.
2. Have a service on the host that monitors the ingress resources in each cluster and adds a domain name for each top level host name. At this point this solution wouldn't be much different than writing to the /etc/hosts file other than the fact that it is not potentially breaking configurations in the /etc/hosts since new files are being created in /etc/resolver/

2. The ip address for the profile

There should only ever be 1 ip address for a profile and that is the minikube ip. Even if a multi-cluster is eventually created as a feature. Since the IP address of the ingress would most likely be the same as the minikube ip, the minikube ip could also always be the ip that is serving DNS queries.

Preferred interim solution

As a short term or interim solution I would be ok with just creating and updating the files each time an instance is updated which only resolves the .test domains. This would at least provide a stop gap for the most widely used use case of local development rather than trying to attend to every possible domain name that could ever be in an ingress running on minikube. This file would look like this:

domain test
nameserver 192.168.99.169
nameserver 192.168.99.170
search_order 1
timeout 5

Where there is 1 entry for each minikube cluster ip. This solution also would include a minikube addon which runs a dns server for the ingress resources installed in each cluster.

Preferred long term solution

Based on the available options I believe that creating a single file /etc/resolver/minikube-profilename-hostname along with a monitoring service that monitors the ingress resources for each cluster and creates entries for each domain in /etc/resolver is the best solution. Adding this capability will make a big difference for development teams with local ingresses to manage as well as brand new developers on the team that have to remember to add an entry to their /etc/hosts file when they try to spin up a project for the first time.

Considerations

Sudo permission required to write to `/etc/resolver/

As far as I know in order to add files to the /etc/resolver/ directory administrative permissions are required. At the present I don't know if minikube has administrative privileges.

Possible Workarounds

1. Use single resolve.conf file with symlink

In the case where administrative privileges are required it c better to create a single file during installation for every profile in ~/.minikube/resolve.conf then add a symlink pointing from /etc/resolver/minikube -> ~/.minikube/resolve.conf so that the file can be updated without administrative permissions. Although it wouldn't be preferred, alternatively if the resolve.conf file is created in ~/.minikube/resolve.conf, adding the symlink can be a one time separate step done after the minikube installation. AFAIK auto installation of the symlink could only be done inside package managers like brew, chocolatey, apt, etc.

2. Change group of /etc/resolver

We can also change the group of the /etc/resolver/ directory so that a minikube user has permission to write to the directory.

MacOS mDNS update

Sudo permission is required to reload mDNS and is required each time that an update occurs in /etc/resolver
For example:

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist

In this case the sudo permission would be required

[woodcockjosh]

Example pain points:
#3821
#5496
#2922 (comment)
#4937
#3677 (comment)
#2762
#3938
#2834
#1538
#347
#496
#38

[woodcockjosh]

Addon for ingress dns server get's us part of ther way towards solving this issue: #5507

[woodcockjosh]

Proposal for background service to install resolver configs on host OS #5511

[medyagh]

This would be great topic for an office hour discussion ! lets discuss it in the office hours.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tstromberg]

Closing as stale.