looking up node via spec.nodeName dial tcp: lookup minikube on 10.0.0.10:53: server misbehaving

[arussellsaw]

Bug Report:

minikube version: (use minikube version): v0.12.0

Environment:

• OS (e.g. from /etc/os-release): OSX 10.11
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName): VirtualBox
• Docker version (e.g. docker -v): Docker version 1.12.2, build bb80604, experimental

What happened: nodes trying to resolve minikube nodename but i get:

2016-10-26T14:10:56.356538450Z E 1026 14:10:56.356377 queue producer.go:318     1 (minikube:4150) error connecting to nsqd - dial tcp: lookup minikube on 10.0.0.10:53: server misbehaving

What you expected to happen: should resolve to node ip

How to reproduce it (as minimally and precisely as possible):

running nsq as DaemonSet:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: nsq
spec:
  template:
    metadata:
      labels:
        name: nsq
        member: nsq
    spec:
      containers:
        - image: nsqio/nsq:latest
          name: nsq
          args:
            - "nsqd"
          ports:
            - containerPort: 4150
              protocol: TCP
              name: nsqd-tcp
            - containerPort: 4151
              protocol: TCP
              name: nsqd-http

running another DaemonSet, a go binary that connects to nsqd:

the container is private, but all it needs to do is try to connect and publish to nsq on the passed NODE_NAME:4150

piVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: sidecard
spec:
  template:
    metadata:
      labels:
        name: sidecard
        member: sidecard
    spec:
      imagePullSecrets:
        - name: dockerhubsecret
      containers:
        - image: avocet/sidecard:latest
          name: sidecard
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName

[arussellsaw]

that was a bit long winded, i've created a container and yaml that will run on a fresh minikube and reproduce the issue.

tested only on OSX, haven't had access to a linux machine yet

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: test
spec:
  template:
    metadata:
      labels:
        name: test
        member: test
    spec:
      containers:
        - image: arussellsaw/test:latest
          name: test
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName

arussellsaw/test is simply this go binary:

package main

import (
        "fmt"
        "net"
        "os"
)

func main() {
        fmt.Println(net.LookupIP(os.Getenv("NODE_NAME")))
}

[arussellsaw]

i had a colleague running minikube on ubuntu 16.04 to run the above config and they suffered the same issue

[sevein]

Same here! Not sure if this is useful but these are the contents of /etc/resolv.conf in my containers:

search default.svc.cluster.local svc.cluster.local cluster.local baseone
nameserver 10.0.0.10
options ndots:5

[arussellsaw]

i see the kube-dns in services but i don't see a kube-dns pod running. I also don't see any kube-dns in minikube addons list. should there be a kube-dns addon running?

[dlorenc]

@arussellsaw we were running a different implementation of DNS. We merged a PR yesterday to switch us over to the normal addon implementation. #738

[sevein]

@arussellsaw , with v0.12.1 I can see the addon via minikube addons list and the corresponding kube-dns pod with kubectl get pods --namespace=kube-system.

However, the name populated via spec.nodeName (minikube) is not resolvable by kube-dns but I can definitely resolve other host names like my services.

[sevein]

I've been told in #kubernetes-users that @therc is working on a solution to ease local-node service discovery. More here: kubernetes/kubernetes#28610. CC @arussellsaw.

[medyagh]

I got the same issue !

[dlorenc]

@medyagh Could you open a new issue with more details? The DNS implementation we use now is completely different than when this bug was opened.