Listen tcp :53: bind: permission denied ERROR!!

[Tej-Singh-Rana]

What happened?

kubectl get po -A
NAMESPACE      NAME                                   READY   STATUS             RESTARTS      AGE
kube-flannel   kube-flannel-ds-962vp                  1/1     Running            0             15m
kube-flannel   kube-flannel-ds-qs6xr                  1/1     Running            0             15m
kube-system    coredns-7db6d8ff4d-6w776               0/1     CrashLoopBackOff   1 (10s ago)   12s
kube-system    coredns-7db6d8ff4d-99tng               0/1     CrashLoopBackOff   1 (9s ago)    12s
kube-system    etcd-controlplane                      1/1     Running            0             12m
kube-system    kube-apiserver-controlplane            1/1     Running            0             12m
kube-system    kube-controller-manager-controlplane   1/1     Running            0             12m
kube-system    kube-proxy-hbt5b                       1/1     Running            0             12m
kube-system    kube-proxy-x96js                       1/1     Running            0             11m
kube-system    kube-scheduler-controlplane            1/1     Running            0             12m

I was setting up the single-node k8s cluster (1 controlplane and 1 worker node). After going through all the installation process from the official k8s site -> at last, after deploying the network plugin on the k8s cluster. CoreDNS pods went into a Crashloopbackoff state. I did check for the container logs and found the following error message:

 listen tcp :53: bind: permission denied

Please look into this and provide some insights. I have faced same issue while upgrading the cluster from v1.29 to v1.30.

Thanks & Regards,
Tej Singh Rana

What did you expect to happen?

Both coreDNS pods should be in the running state, after deploying the network plugin.

How can we reproduce it (as minimally and precisely as possible)?

Simply follow the steps from the official k8s site.

Anything else we need to know?

I did some tests, and I used 1024 instead of 53 port, and it started to work. (AFAIK, using below port ~1000 was not working)

kubectl logs -n kube-system coredns-7db6d8ff4d-wchnq 
.:1024
[INFO] plugin/reload: Running configuration SHA512 = e20da72760199c1bc59098f3ae16621ae48df8f7756e50bd0dfa5553ccb7be57af61562fff46a43fdcce51ac086b26aa19929386004908ad3afe3aea9b06316a
CoreDNS-1.11.1
linux/amd64, go1.20.7, ae2bbc2

Environment:

• the version of CoreDNS: registry.k8s.io/coredns/coredns:v1.11.1
• Corefile: The below content is from the coredns configMap.

  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }

• logs:

 listen tcp :53: bind: permission denied

Kubernetes version

$ kubectl version
Client Version: v1.30.0 <br>
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 <br>
Server Version: v1.30.1

Cloud provider

N/A

OS version

# On Linux:
$ cat /etc/os-release

PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ uname -a
Linux controlplane 5.4.0-1106-gcp #115~18.04.1-Ubuntu SMP Mon May 22 20:46:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Install tools

kubectl version 
Client Version: v1.30.0 
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 
Server Version: v1.30.1

kubelet --version

Kubernetes v1.30.0

kubeadm version 

kubeadm version: &version.Info{Major:"1", Minor:"30", GitVersion:"v1.30.0",  GitCommit:"7c48c2bd72b9bf5c44d21d7338cc7bea77d0ad2a", GitTreeState:"clean", BuildDate:"2024-04-17T17:34:08Z", GoVersion:"go1.22.2", Compiler:"gc", Platform:"linux/amd64"}

Container runtime (CRI) and version (if applicable)

containerd --version
containerd containerd.io 1.6.26 3dd1e886e55dd695541fdcd67420c2888645a495

Related plugins (CNI, CSI, ...) and versions (if applicable)

   Service Account:  flannel
  Init Containers:
   install-cni-plugin:
    Image:      docker.io/flannel/flannel-cni-plugin:v1.4.1-flannel1
    Port:       
    Host Port:  
    Command:
      cp
    Args:
      -f
      /flannel
      /opt/cni/bin/flannel
    Environment:  
    Mounts:
      /opt/cni/bin from cni-plugin (rw)
   install-cni:
    Image:      docker.io/flannel/flannel:v0.25.2
    Port:       

[k8s-ci-robot]

There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:

• /sig <group-name>
• /wg <group-name>
• /committee <group-name>

Please see the group list for a listing of the SIGs, working groups, and committees available.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[neolit123]

a better place to ask would be on the support channels. please see:
https://git.k8s.io/kubernetes/SUPPORT.md

/kind support
/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

a better place to ask would be on the support channels. please see:
https://git.k8s.io/kubernetes/SUPPORT.md

/kind support
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ascarl2010]

[root@node1 ~]# kubectl logs -n kube-system coredns-5f98f8d567-5h7b4
kubectl logs -n kube-system coredns-5f98f8d567-j2hl6
Listen: listen tcp :53: bind: permission denied
Listen: listen tcp :53: bind: permission denied
[root@node1 ~]# sudo lsof -i :53
[root@node1 ~]# sudo netstat -tuln | grep :53
[root@node1 ~]# sudo netstat -tuln | grep :53

K8S 1.29.5&kube-flannel

[Tej-Singh-Rana]

It's an issue, that's why I raised here to address.

[Tej-Singh-Rana]

Hi @ascarl2010 ,

What's your OS version? Can you share it here?

Regards,

[raphaelpoumarede]

I have similar issue with an RKE 1.29 install (1.6.0-rc6 v1.29.5-rancher1-1) with the CoreDNS deployment in the kube-system namespace. The pods are in Error and show the same message : "listen tcp :53: bind: permission denied"
I can fix the problem by setting the "allowPrivilegeEscalation" to true in the core-dns deployment Security Context but in my older RKE deployments (1.28) it is working fine with the default allowPrivilegeEscalation set to false...

I'm wondering if something has changed in the new Kubernetes or container version (I also now have a new seccompProfile set in the coredns deployment securityContext definition...) ?

securityContext:                                                                                                                                                                                                              
│           allowPrivilegeEscalation: true                                                                                                                                                                                              
│           capabilities:                                                                                                                                                                                                               
│             add:                                                                                                                                                                                                                      
│             - NET_BIND_SERVICE                                                                                                                                                                                                        
│             drop:                                                                                                                                                                                                                     
│             - all                                                                                                                                                                                                                     
│           readOnlyRootFilesystem: true                                                                                                                                                                                                
│           seccompProfile:                                                                                                                                                                                                             
│             type: RuntimeDefault

[humblec]

It has seen in few other setups as well with Kubernetes 1.29.4 and 1.30.1, coredns v1.11.1 ( non-root image and run) and containerd 1.6.

Workaround:

1. Non-system port binding
2. containerd config change net.ipv4.ip_unprivileged_port_start/EnableUnpriviledPorts
   looks to be helping to workaround the issue.

However still not able to completely figure out the exact reason. The capability bit looks to be not set for privileged bind by a non-root user in our context:

securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
            - 

CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	000001ffffffffff
CapAmb:	0000000000000000
NoNewPrivs:	0

Could be a combination of Kernel+capabilities , but yet to figure out the exact reason.

[raphaelpoumarede]

See below some additional info after more research and troubleshooting...I have found a fix and the problem might be related to the fact that I'm running on Centos7 nodes (with the Docker container runtime)...

There was a change in the SecurityContext with coreDNS 1.11.1 (coming with Rancher 1.29)

• Workaround : set "allowPrivilegeEscalation" to true in the SecurityContext.
• Notes:
• "kubernetes v1.29.0+ uses coredns v1.11.1 and from v1.11.0, coredns runs as non-root user"
  From add default sysctls to allow privileged ports with no capabilities minikube#18421
  ◊ Version in our RKE 1.29: rancher/mirrored-coredns-coredns:1.11.1
  ◊ Version in our RKE 1.28: rancher/mirrored-coredns-coredns:1.10.1
• "I think it's specifically related to the Centos 7 that's going end-of-life by the end of this months (June 2024) and therefore probably will not get the necessary fixes backported (eg, setting net.ipv4.ip_unprivileged_port_start)"
  From cenots 7: minikube v1.33.1 faces coredns CrashLoopBackOff. minikube#18950 (comment)

[gardo984]

Hello dear coders,

I was facing the same incident and the fix for the issue was the following (I added a comment on the original issue):

• minikube preload tarballs need to preserve extended attributes such as capabilities minikube#17804 (comment)

This was done in :

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy

$ k version
Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2

Hopefully this works for ya all.

[spantaleev]

Indeed containerd_enable_unprivileged_ports: true worked around the issue for me. It's likely a better fix than pinning coredns to an older version.

I'm still not sure why with the default Kubespray configuration, some of my nodes were allowing privileged ports to be used by non-root users, while others were not.

[BBQigniter]

We were seeing the same issue on a cluster that was upgraded from 1.29.7 to 1.29.11 - but before coredns pods worked and started normally.

By applying the changes to the coredns deployment-manifest like discussed in #105309 the pods came up immediately. So for now I'm using this coredns deployment-manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: kube-dns
  name: coredns
  namespace: kube-system
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kube-dns
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        k8s-app: kube-dns
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: k8s-app
                  operator: In
                  values:
                  - kube-dns
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - args:
        - -conf
        - /etc/coredns/Corefile
        image: registry.k8s.io/coredns/coredns:v1.11.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /health
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5
        name: coredns
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9153
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /ready
            port: 8181
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            memory: 170Mi
          requests:
            cpu: 100m
            memory: 70Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /etc/coredns
          name: config-volume
          readOnly: true
      dnsPolicy: Default
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        sysctls:
        - name: net.ipv4.ip_unprivileged_port_start
          value: "53"
      serviceAccount: coredns
      serviceAccountName: coredns
      terminationGracePeriodSeconds: 30
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      - effect: NoSchedule
        key: node-role.kubernetes.io/control-plane
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: Corefile
            path: Corefile
          name: coredns
        name: config-volume

But anyway it's super weird because I have other clusters with same OS (ALMA 9.5, Kernel 5.14.0-503.15.1.el9_5.x86_64, same kernel parameters, containerd.io-1.7.24-3.1.el9.x86_64, etc.) where I do not have those issues 🤷 - still investigating

[BBQigniter]

again we had this issue on one of our clusters

so, as far as I understood in containerd's config.toml the following two parameters have to be set like this

    enable_unprivileged_icmp = true
    enable_unprivileged_ports = true

to be future proof. With containerd v2 those two parameters seem to be set to true for default. There are quite a few threads about this topic lurking around and the documentation about those two parameters could be better. I only hope that those changed parameters do not have any security implications sooner or later as I'm not sure if it's really wise to enable those.

I would have preferred the sysctls-approach but it looks like proposed change from #105309 never made it 🤷