Internet connectivity from pods not available #11490
Comments
|
Update, a full uninstall and reinstall later I can ping! My iptables version is: |
|
DNS ok but still can't wget. This issue is still open. |
|
Have the similar problem, supposed to be the Kubernetes API server not working... But I have not yet found the solution. |
|
Same issue here on single node cluster running in VPS K3S Version: v1.31.4+k3s1 but same happens on v1.30.6+k3s1 Attempt to access example.com from pod:
Fun detail is that if you'll try to specify dns config manually in manifest, everything will be working just fine apiVersion: v1
kind: Pod
metadata:
name: ubuntu-shell
spec:
dnsPolicy: None
dnsConfig:
nameservers:
- 8.8.8.8
- 1.1.1.1
containers:
- name: ubuntu
image: ubuntu:22.04
command: ["/bin/bash", "-c", "while true; do sleep 30; done"] # Keeps the container running
stdin: true
tty: true
restartPolicy: AlwaysUpdate!I enabled logs in coredns and got this result when tried to So, I came up with accessing google.com via FQDN and everything worked Update 2I configured Corefile like this and now everything works, but I'm not sure that this is right approach apiVersion: v1
data:
Corefile: |
.:53 {
log
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
fallthrough
}
hosts /etc/coredns/NodeHosts {
ttl 60
reload 15s
fallthrough
}
prometheus :9153
forward . /etc/resolv.conf
rewrite name regex (.*)\.cluster\.local\.$ {1}.
cache 30
loop
reload
loadbalance
import /etc/coredns/custom/*.override
}
import /etc/coredns/custom/*.server |
Environmental Info:
K3s Version:
k3s version v1.31.4+k3s1 (a562d09)
go version go1.22.9
Node(s) CPU architecture, OS, and Version:
Linux tom-nuc 6.8.0-49-generic #49~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Nov 6 17:42:15 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
Cluster Configuration:
Single machine install
Describe the bug:
I cannot connect to the internet from within pods. As this is Ubuntu I've disabled UFW.
Steps To Reproduce:
Used the quickstart script
kk run -i --tty --rm debug --image=busybox --restart=Never -- sh
Expected behavior:
I should be able to connect out to the internet.
Actual behavior:
The pod behaves as if sandboxed.
Additional context / logs:
My local dns setup is:
/etc/resolv.conf => nameserver 8.8.8.8
tom@tom-nuc:~$ kk run -i --tty --rm debug --image=busybox --restart=Never -- sh
/ #
/ # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5
/ # ping google.com
ping: bad address 'google.com'
Also can't ping the dns server: 10.43.0.10
I had a look at the core dns logs in kube-system - lots going there:
==== START logs for container coredns of pod kube-system/coredns-ccb96694c-wx7ww ====
maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
[WARNING] No files matching import glob pattern: /etc/coredns/custom/.override
[WARNING] No files matching import glob pattern: /etc/coredns/custom/.server
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.EndpointSlice: Get "https://10.43.0.1:443/apis/discovery.k8s.io/v1/endpointslices?limit=500&resourceVersion=0": dial tcp 10.43.0.1:443: connect: connection refused
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.Service: Get "https://10.43.0.1:443/api/v1/services?limit=500&resourceVersion=0": dial tcp 10.43.0.1:443: connect: connection refused
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.Namespace: Get "https://10.43.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.43.0.1:443: connect: connection refused
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/ready: Still waiting on: "kubernetes"
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.Namespace: Get "https://10.43.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0": dial tcp 10.43.0.1:443: connect: connection refused
[ERROR] plugin/kubernetes: Unhandled Error
[INFO] plugin/kubernetes: waiting for Kubernetes API before starting server
[INFO] plugin/kubernetes: pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:243: failed to list *v1.EndpointSlice: Get "https://10.43.0.1:443/apis/discovery.k8s.io/v1/endpointslices?limit=500&resourceVersion=0": dial tcp 10.43.0.1:443: connect: connection refused
Also had a look at iptables - this is where is got interesting... I have, being burnt several times, setup iptables to pass everything on unrestricted but lo I find a huge amount going on here... - all KUBE related, eg:
KUBE-POD-FW-QDSOJU2OKPP7DKIO all -- anywhere 10.42.0.16 /* rule to jump traffic destined to POD name:gitea-579b69946b-vgx4p namespace: gitea to chain KUBE-POD-FW-QDSOJU2OKPP7DKIO /
KUBE-POD-FW-QDSOJU2OKPP7DKIO all -- 10.42.0.16 anywhere / rule to jump traffic from POD name:gitea-579b69946b-vgx4p namespace: gitea to chain KUBE-POD-FW-QDSOJU2OKPP7DKIO /
KUBE-POD-FW-6OXKOEB4ZFSLRLFJ all -- anywhere 10.42.0.13 / rule to jump traffic destined to POD name:svclb-gitea-ssh-9e6489ae-7xfnz namespace: kube-system to chain KUBE-POD-FW-6OXKOEB4ZFSLRLFJ */
What would be the easiest approach to resetting all this properly? (not via a full reinstall pls) I've tried reinstalling k3s. I'm quite happy to lose the existing pods.
The text was updated successfully, but these errors were encountered: