minikube preload tarballs need to preserve extended attributes such as capabilities

[afbjorklund]

The coredns version included with Kubernetes 1.29.0, that is coredns v1.11.1, now requires capabilities:

-ADD coredns /coredns
+COPY coredns /coredns
+RUN setcap cap_net_bind_service=+ep /coredns

So when you run coredns from that image (in the preload), it does not have the capability to listen below 1024:

Listen: listen tcp :53: bind: permission denied

coredns/coredns@d21537f

---

We need to make sure to include the extended attributes in the tarball, and to extract them again (requires root)

--xattrs --xattrs-include=security.capability

Then the capabilities gets included in the tarball, under a special header called "SCHILY.xattr.security.capability"

It requires a reasonable new tar version, like 1.27+

[afbjorklund]

The workaround to run --kubernetes-version=v1.29.0 before this fix, is to also include --preload=false

[afbjorklund]

This issue only affects containerd and cri-o (and any other container runtime, except for docker), because of:

sysctl net.ipv4.ip_unprivileged_port_start=0 means that capability CAP_NET_BIND_SERVICEis not needed

[afbjorklund]

Both containerd and cri-o have configuration to enable binding to ports below 1024, but it is off by default.

Docker has it on, since 20.10: moby/moby@dae652e adds sysctl net.ipv4.ip_unprivileged_port_start=0

containerd

    enable_unprivileged_ports = false

crio

# List of default sysctls. If it is empty or commented out, only the sysctls
# defined in the container json file by the user/kube will be added.
# default_sysctls = [
# ]

[gardo984]

Both containerd and cri-o have configuration to enable binding to ports below 1024, but it is off by default.

Docker has it on, since 20.10: moby/moby@dae652e adds sysctl net.ipv4.ip_unprivileged_port_start=0

containerd

    enable_unprivileged_ports = false

crio

# List of default sysctls. If it is empty or commented out, only the sysctls
# defined in the container json file by the user/kube will be added.
# default_sysctls = [
# ]

Hey sir, thank you very much for sharing ur knowledge, this was very helpful and fixed the issue that I was having.

I made the change on the kernel settings and containerd settings and restart the services and it worked. 🥳

cat /etc/sysctl.conf |egrep -i net.ipv4.ip_unprivileged_port_start
-> net.ipv4.ip_unprivileged_port_start = 0
sysctl -f /etc/sysctl.conf

cat /etc/containerd/config.toml | egrep -i enable_unprivileged_ports
-> enable_unprivileged_ports = true
systemctl stop containerd && systemctl start containerd
systemctl stop kubelet && systemctl start kubelet