Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CoreDNS details to DNS Debug docs #10201

Merged
merged 2 commits into from
Sep 10, 2018
Merged

Add CoreDNS details to DNS Debug docs #10201

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b3db25d
add coredns details
chrisohaver Sep 5, 2018
68ffe0f
address nits, add query logging section
chrisohaver Sep 7, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 42 additions & 5 deletions content/en/docs/tasks/administer-cluster/dns-debugging-resolution.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ This page provides hints on diagnosing DNS problems.
{{% capture prerequisites %}}
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
* Kubernetes version 1.6 and above.
* The cluster must be configured to use the `kube-dns` addon.
* The cluster must be configured to use the `coredns` (or `kube-dns`) addons.
{{% /capture %}}

{{% capture steps %}}
Expand Down Expand Up @@ -68,7 +68,7 @@ nameserver 10.0.0.10
options ndots:5
```

Errors such as the following indicate a problem with the kube-dns add-on or
Errors such as the following indicate a problem with the coredns/kube-dns add-on or
associated Services:

```
Expand All @@ -93,6 +93,17 @@ nslookup: can't resolve 'kubernetes.default'

Use the `kubectl get pods` command to verify that the DNS pod is running.

For CoreDNS:
```shell
kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
...
coredns-7b96bf9f76-5hsxb 1/1 Running 0 1h
coredns-7b96bf9f76-mvmmt 1/1 Running 0 1h
...
```

Or for kube-dns:
```shell
kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
Expand All @@ -107,8 +118,26 @@ have to deploy it manually.

### Check for Errors in the DNS pod

Use `kubectl logs` command to see logs for the DNS daemons.
Use `kubectl logs` command to see logs for the DNS containers.

For CoreDNS:
```shell
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1)
```

Here is an example of a healthy CoreDNS log:

```
.:53
2018/08/15 14:37:17 [INFO] CoreDNS-1.2.2
2018/08/15 14:37:17 [INFO] linux/amd64, go1.10.3, 2e322f6
CoreDNS-1.2.2
linux/amd64, go1.10.3, 2e322f6
2018/08/15 14:37:17 [INFO] plugin/reload: Running configuration MD5 = 24e6c59e83ce706f07bcc82c31b1ea1c
```


For kube-dns, there are 3 sets of logs:
```shell
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c kubedns

Expand All @@ -117,8 +146,8 @@ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system
kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name | head -1) -c sidecar
```

See if there is any suspicious log. Letter '`W`', '`E`', '`F`' at the beginning
represent Warning, Error and Failure. Please search for entries that have these
See if there are any suspicious error messages in the logs. In kube-dns, a '`W`', '`E`' or '`F`' at the beginning
of a line represents a Warning, Error or Failure. Please search for entries that have these
as the logging level and use
[kubernetes issues](https://github.com/kubernetes/kubernetes/issues)
to report unexpected errors.
Expand All @@ -135,6 +164,8 @@ kube-dns ClusterIP 10.0.0.10 <none> 53/UDP,53/TCP 1h
...
```


Note that the service name will be "kube-dns" for both CoreDNS and kube-dns deployments.
If you have created the service or in the case it should be created by default
but it does not appear, see
[debugging services](/docs/tasks/debug-application-cluster/debug-service/) for
Expand All @@ -160,6 +191,12 @@ in the Kubernetes GitHub repository.

## Known issues

Some Linux distros (e.g. Ubuntu), use a local DNS resolver by default (systemd-resolved).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

distros -> distributions
so that we try to be more academic. :)

Systemd-resolved moves and replaces /etc/resolv.conf with a stub file that can cause a fatal forwarding
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/etc/resolv.conf should be quoted in ` quotes

loop when resolving names in upstream servers. This can be fixed manually by using kubelet's `--resolv-conf` flag
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extra space after servers.

to point to the correct resolv.conf (With systemd-resolved, this is `/run/systemd/resolve/resolv.conf`).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolv.conf (With

->

resolv.conf (with

resolv.conf is quoted with `

kubeadm 1.11 automatically detects systemd-resolved, and adjusts kubelet accordingly.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and adjusts kubelet accordingly

->

and adjusts the kubelet flags accordingly


Kubernetes installs do not configure the nodes' resolv.conf files to use the
cluster DNS by default, because that process is inherently distro-specific.
This should probably be implemented eventually.
Expand Down