k3s air-gapped env resolve dns fails for statefulset

[7ff5fC1E397d]

An air-gapped env has been configured with one master and a worker.

k3s

k3s version v1.30.3+k3s1 (f6466040)
go version go1.22.5

os

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

master

#!/bin/bash

set -ex

cp -v ./k3s /usr/local/bin/

export INSTALL_K3S_SKIP_DOWNLOAD=true
export INSTALL_K3S_EXEC="--disable=traefik --cluster-cidr=10.42.0.0/16 --service-cidr=10.43.0.0/16 --node-external-ip=172.16.110.124 --node-ip=172.16.110.124 --node-label=100m1"

./install.sh

worker

#!/bin/bash

set -ex

cp -v ./k3s /usr/local/bin/

export INSTALL_K3S_SKIP_DOWNLOAD=true
export K3S_URL=https://172.16.110.124:6443
export INSTALL_K3S_EXEC="--node-external-ip=172.16.110.126 --node-ip=172.16.110.126"
export K3S_TOKEN='xyz'

./install.sh

sample pod

Two pods have been created and set on a single node

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  labels:
    app: nginx
spec:
  serviceName: "nginx"
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      # hostname: nginx
      subdomain: nginx     # to resolve dns
      nodeName: 100w3    # <-- the worker
      containers:
      - name: nginx
        image: nginx:stable-alpine3.20
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
      volumes:
      - name: www
        emptyDir: {}

issue

On this air-gapped env web-0 and web-1 cannot see (ping) each other. On web-0 pod, it can ping itself but not web-1 and vice versa , on web-1 it can ping itself but not web-0 even if we use subdomain

• web-0.nginx
• web-1.nginx

tests

Out of the air-gapped env, (public access to Internet) with the exact same k3s version and nginx config it works fine.

question

Is this issue because of air-gapped env ?
Or perhaps any extra or special configs are needed ?

screenshots

air-gapped

public

more info

Nothing has been blocked on machines, all ports and protos are allowed . Using nc tested that udp 53 are passed.

kubectl get svc

NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   40m
nginx        ClusterIP   None         <none>        80/TCP    30m

kubectl get ep

NAME         ENDPOINTS                   AGE
kubernetes   172.16.110.124:6443         41m
nginx        10.42.1.2:80,10.42.1.3:80   32m

kubectl get ep -n kube-system

NAME             ENDPOINTS                                  AGE
kube-dns         10.42.0.3:53,10.42.0.3:53,10.42.0.3:9153   41m
metrics-server   10.42.0.2:10250                            41m

kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE

coredns-576bfc4dc7-4zftj                  1/1     Running   0          53m
local-path-provisioner-6795b5f9d8-hprmm   1/1     Running   0          53m
metrics-server-557ff575fb-vjwrh           1/1     Running   0          53m

crictl image ls

IMAGE                                        TAG                 IMAGE ID            SIZE
docker.io/library/nginx                      stable-alpine3.20   60847f99ea697       20.9MB
docker.io/rancher/local-path-provisioner     v0.0.28             5d221316a3c61       18.4MB
docker.io/rancher/mirrored-coredns-coredns   1.10.1              ead0a4a53df89       16.2MB
docker.io/rancher/mirrored-metrics-server    v0.7.0              b9a5a1927366a       19.4MB
docker.io/rancher/mirrored-pause             3.6                 6270bb605e12e       301kB

both web-0 and web-1 can ping directly using their IPs, but using name, no, so the issue is DNS

Does anyone have any idea ?

Regards

[dereknola]

Perhaps https://docs.k3s.io/installation/airgap#default-network-route

[7ff5fC1E397d]

I do not think so, each node has its interface and its default route and they have been configured and can see each other

[brandond]

each node has its interface and its default route

Is the default route associated with the interface the nodes use to communicate with each other?

Note that you're testing dns resolution with ping which is the wrong tool. You should use dig or nslookup so that you can see if you are actually getting a response from the cluster dns server or not.

[dereknola]

You disabled traefik, what are you using as its replacement?

[7ff5fC1E397d]

nothing, it is a test and I did not need the ingress-controller

[7ff5fC1E397d]

@brandond

kubectl get nodes -o wide

NAME    STATUS   ROLES                  AGE     VERSION        INTERNAL-IP      EXTERNAL-IP      OS-IMAGE                         KERNEL-VERSION   CONTAINER-RUNTIME
100m1   Ready    control-plane,master   4d21h   v1.30.3+k3s1   172.16.110.124   172.16.110.124   Debian GNU/Linux 12 (bookworm)   6.1.0-18-amd64   containerd://1.7.17-k3s1
100w3   Ready    100w3,worker           4d21h   v1.30.3+k3s1   172.16.110.126   172.16.110.126   Debian GNU/Linux 12 (bookworm)   6.1.0-18-amd64   containerd://1.7.17-k3s1

Also I tried other setups, like no external-ip, but same result
The environment is VMs managed by VMWare and all machines can see each other, no firewalls

master

ip route show

default via 172.16.110.1 dev ens192 
10.42.0.0/24 dev cni0 proto kernel scope link src 10.42.0.1 
10.42.1.0/24 via 10.42.1.0 dev flannel.1 onlink 
172.16.110.0/24 dev ens192 proto kernel scope link src 172.16.110.124 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1 

ip -br a

lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens192           UP             172.16.110.124/24 fe80::250:56ff:feb1:9400/64 
docker0          DOWN           172.17.0.1/16 
flannel.1        UNKNOWN        10.42.0.0/32 fe80::f824:2cff:fe82:62f7/64 
cni0             UP             10.42.0.1/24 fe80::494:f0ff:fec5:ef3d/64 
veth2d3efc06@if2 UP             fe80::80cd:2fff:febb:5f14/64 
veth8c02e70a@if2 UP             fe80::ec14:c3ff:fe37:2493/64 
veth01936567@if2 UP             fe80::90a7:f9ff:fe76:e133/64 
veth9e89c461@if2 UP             fe80::4839:41ff:fec1:5470/64 
docker_gwbridge  UP             172.18.0.1/16 fe80::42:19ff:fe96:82dc/64 
vetha18e8eb@if14 UP             fe80::6497:b2ff:fea6:e0dd/64 
veth1762b21@if39 UP             fe80::7470:c5ff:fe57:5328/64

worker

ip route show

default via 172.16.110.1 dev ens192 onlink 
10.42.0.0/24 via 10.42.0.0 dev flannel.1 onlink 
10.42.1.0/24 dev cni0 proto kernel scope link src 10.42.1.1 
172.16.110.0/24 dev ens192 proto kernel scope link src 172.16.110.126 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev docker_gwbridge proto kernel scope link src 172.18.0.1

ip -br a

lo               UNKNOWN        127.0.0.1/8 ::1/128 
ens192           UP             172.16.110.126/24 fe80::250:56ff:feb1:710a/64 
docker0          DOWN           172.17.0.1/16 
flannel.1        UNKNOWN        10.42.1.0/32 fe80::a8ae:e7ff:fe9f:deb0/64 
cni0             UP             10.42.1.1/24 fe80::6078:2eff:fe1e:65e9/64 
veth021ed3cf@if2 UP             fe80::e827:e4ff:fe08:89d7/64 
docker_gwbridge  UP             172.18.0.1/16 fe80::42:97ff:fe2f:484c/64 
veth0e84992@if13 UP             fe80::64d6:5bff:fe8b:2e28/64 
vetha1aec289@if2 UP             fe80::41:17ff:fec9:f251/64 
vethf59383df@if2 UP             fe80::84e1:7aff:fe9a:fb4d/64 

You can see Docker routers here , since I installed and tested Docker Swarm to test if the issue is by the Machines or not. Docker Swarm DNS resolution works fine

[brandond]

The environment is VMs managed by VMWare

• Networking issue between nodes rancher/rke2#6307 (comment)

For K3s, you should disable on flannel.1 instead of flannel.4096