Virtualbox DNS relay fails under load: getaddrinfo EAI_AGAIN, i/o timeout

[kiboliu]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Please provide the following details:

Environment:

Minikube version (use minikube version): v0.33.1

• OS (e.g. from /etc/os-release): 16.04
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): virtualbox
• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION): v0.33.1
• Install tools:
• Others:
  The above can be generated in one go with the following commands (can be copied and pasted directly into your terminal):

minikube version
echo "";
echo "OS:";
cat /etc/os-release
echo "";
echo "VM driver:"; 
grep DriverName ~/.minikube/machines/minikube/config.json
echo "";
echo "ISO version";
grep -i ISO ~/.minikube/machines/minikube/config.json

What happened:
In my application running on the cluster, I need to upload local file to azure storage blob. For small files it works fine. However for large files (about 70mb), it always throws error

Error: getaddrinfo EAI_AGAIN xxx.blob.core.windows.net

The EAI_AGAIN error code is a DNS lookup timeout error. I check some settings:

# minikube ssh & sudo vi /etc/resolve.conf
nameserver 10.0.2.3

# Two coredns pods are running, with lots of timeout error in logs
A: unreachable backend: read udp 172.17.0.2:59109->10.0.2.3:53: i/o timeout

# A service named kube-dns exists, cluster-ip is 10.96.0.10
Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=KubeDNS
Annotations:       prometheus.io/port=9153
                   prometheus.io/scrape=true
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.96.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         172.17.0.2:53,172.17.0.3:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         172.17.0.2:53,172.17.0.3:53
Session Affinity:  None
Events:            <none>

# A endpoint named kube-dns exists
Name:         kube-dns
Namespace:    kube-system
Labels:       k8s-app=kube-dns
              kubernetes.io/cluster-service=true
              kubernetes.io/name=KubeDNS
Annotations:  <none>
Subsets:
  Addresses:          172.17.0.2,172.17.0.3
  NotReadyAddresses:  <none>
  Ports:
    Name     Port  Protocol
    ----     ----  --------
    dns      53    UDP
    dns-tcp  53    TCP

Events:  <none>

I'm not sure how the pod running in the cluster to do DNS lookup, so I did not catch anything wrong from above information.

I also try minikube ssh into the vm and do nslookup [dnsname], sometimes fail sometimes succeed.

What you expected to happen:
Succeed DNS lookup.
How to reproduce it (as minimally and precisely as possible):
Build a cluster and upload a large file to azure blob storage.
Output of minikube logs (if applicable):
This error repeatedly occurs, not sure if it's related

Jan 30 00:20:27 minikube kubelet[2862]: E0130 00:20:27.380930    2862 kubelet_volumes.go:154] Orphaned pod "08c3b471-2416-11e9-8c7a-08002706d1dd" found, but volume paths are still present on disk : There were a total of 1 errors similar to this. Turn up verbosity to see them.

Anything else do we need to know:
Virtualbox network settings (not changed, it's default):

Adapter 1: Paravirtualized Network (NAT)
Adapter 2: Paravirtualized Network (Host-only Adapter, 'vboxnet0')

[kiboliu]

For additional information to my problem, I also get this error for other dns name such as 'api.github.com'

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[tstromberg]

I suspect there may have been firewall interference here. Is it possible that there was a VPN or firewall configured on the host?

Also: Do you mind trying with minikube v1.1.0? Thanks!

[cvila84]

Hello, I confirm the issue even with minikube v1.1.1

From my understanding, it comes from Virtualbox NAT DNS proxy, which sometimes silently crashes, resulting in i/o timeout on kube-dns (at it refers to NAT DNS proxy to resolve addresses external to K8S cluster)

I tested with latest 5.2.30 and 6.0.8 VB but gets same error after several minutes/hours.

I tried to replace NAT by NAT network on minikube VM and for the moment, everything is working well.

[tstromberg]

Upstream: https://www.virtualbox.org/ticket/14736

[tstromberg]

Has anyone seen this with more recent releases of Virtualbox?

I don't see anything from their changelog that suggests the issue has been fixed.

[tstromberg]

Closing this as I haven't heard much here, and there is isn't much we can do about the VirtualBox DNS relay. If you run into this, try upgrading to VirtualBox 6.0.14+, and if that fails, try another hypervisor.