kubeadm complains about bridge-nf-call and ip_forward if not using docker runtime

[sysrich]

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

After booting a clean system, running kubeadm init with a CRI runtime other than docker configured produces the following pair of error messages:

	[ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
	[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

What you expected to happen:

Those two pre-flight checks to pass because they could be satisfied automatically.

Obviously, these are handled automatically by the starting of the docker daemon normally, but for openSUSE Kubic we are investigating using CRI-O by default, where we do not have the luxury of an overbearing daemon to meddle with such things.

Therefore this bug report is either an opportunity for kubeadm to handle things itself. I think there is a case to be that because kubeadm knows what it needs, kubeadm should modprobe br_netfilter and echo '1' > /proc/sys/net/ipv4/ip_forward automatically rather than complaining about the issues.

Alternatively, if this suggestion is not acceptable, I'd appreciate a suggestion as to how openSUSE Kubic should automatically address these issues in a way that will remain aligned with kubeadm's general expectations.

How to reproduce it (as minimally and precisely as possible):

• Install kubeadm and cri-o
• systemctl enable --now crio
• Configure kubeadm to run kublet with additional args --container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --runtime-request-timeout=15m
• Run kubeadm init --cri-socket /var/run/crio/crio.sock

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): v1.11.2
• Cloud provider or hardware configuration: qemu-kvm x86_64 16GB RAM 2 cores
• OS (e.g. from /etc/os-release): opensuse-tumbleweed-kubic
• Kernel (e.g. uname -a): 4.17.13
• Install tools: kubeadm

[outcoldman]

Does not feel like a product bug, but documentation bug (just a missing manual). Current we have only pages on how to install kubernetes with kubeadm with docker as container runtime, we should have another one for cri-o.

[Vogtinator]

IMO kubeadm should at least do the modprobe itself. Whether it's the case for ip_forward as well is debatable as that is system configuration and up to the admin.

[sysrich]

@vrothberg - what do you think? Should the above 2 conditions (modprobe and sysctl) be rectified automatically by kubeadm, or do you think this is something better handled in CRI-O?

Docker does both automagically itself

And if you think CRI-O proper shouldn't take care, where do you think the dirty hack should be carried in openSUSE? in the cri-o package or in the kubeadm package? ;)

[vrothberg]

I think that’s something kubeadm package should do. After that, we can check if that’s really an error or if it can be demoted to an info log. Many things in K8s are still built around how Docker does things but they’re not always necessary. I am on vacation at the moment but will check mails here and then. Thanks for the ping.

…

On Wed 12. Sep 2018 at 10:30, Richard Brown ***@***.***> wrote: @vrothberg <https://github.com/vrothberg> - what do you think? Should the above 2 conditions (modprobe and sysctl) be rectified automatically by kubeadm, or do you think this is something better handled in CRI-O? Docker does both automagically itself And if you think CRI-O proper shouldn't take care, where do you think the dirty hack should be carried in openSUSE? in the cri-o package or in the kubeadm package? ;) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1062 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALI4g1dRKf5vWXz_7H27VktuD8nP5tAsks5uaMYogaJpZM4V_W70> .

[sysrich]

This issue will be handled in openSUSE with the following changes: https://build.opensuse.org/package/rdiff/devel:kubic/kubernetes?linkrev=base&rev=9

I'm planning on submitting something similar in the upstream rpm packaging promptly

[itpings]

Restarting Docker did the trick...Thanks you

[adobo76]

I second adding this information to the setup page. I was seeing this error and it caused me at least 20 min of pain until I came across this thread. Thank you for sharing the solution.

[danielsand]

make sure that you follow the procedure then the error doesnt occour
https://kubernetes.io/docs/setup/production-environment/container-runtimes/#cri-o

[natemoseman]

simply modprobe'ng things doesn't ensure they are running. If you reboot then everything breaks. It should tell you to make sure these things are enabled persistently. Which is does for the sysctl rules, but not the modules.

When following the directions on stuff like this I don't like to read into things and try to take extra steps based on hunches. If it tells me to do something, I do it. If it doesn't I don't. I wait for things to break and go back and fix them. This way I have a better idea if it's bad documentation that is the problem instead of some random thing I did when things break.

[jefflill]

I probably need to rebuild my local Hyper-V node images.

[antaloala]

I just fresh installed a k8s 1.24 cluster with containerd as runtime engine, using latest kubeadm tooling and following step by step the installation procedure described in latest (1.24) k8s documentation; I experienced the problem reported in this issue when raising kubeadm init and kubeadm join commands.
I solved the problem invoking (in master and worker nodes) the commands described at the beginning of this issue

modprobe br_netfilter and echo '1' > /proc/sys/net/ipv4/ip_forward

So I understand this has not been solved yet ...

P.S: It was an installation on Ubuntu-based Virtualbox VMs on a Windows10 host OS where Hyper-V has been fully disabled, so experienced issue is not related to any Hyper-V collision (not in my case).

[neolit123]

See https://kubernetes.io/docs/setup/production-environment/container-runtimes/#install-and-configure-prerequisites these are still required when setting up a CR that is k8s compatible.

[antaloala]

Thanks @neolit123 (yes, it is all there .. and I did not realize on it :-(

[RicHincapie]

I just fresh installed a k8s 1.24 cluster with containerd as runtime engine, using latest kubeadm tooling and following step by step the installation procedure described in latest (1.24) k8s documentation; I experienced the problem reported in this issue when raising kubeadm init and kubeadm join commands. I solved the problem invoking (in master and worker nodes) the commands described at the beginning of this issue

modprobe br_netfilter and echo '1' > /proc/sys/net/ipv4/ip_forward

So I understand this has not been solved yet ...

P.S: It was an installation on Ubuntu-based Virtualbox VMs on a Windows10 host OS where Hyper-V has been fully disabled, so experienced issue is not related to any Hyper-V collision (not in my case).

The same pre-flight erros happened to me with kubeadm, cri-o, and kubelet on 1.22

For any one looking for a fast work around, this is the answer.

[syamprasadsiddabattula]

thank u sir.......i am happy with ur solutions sir....tQ

[Napsty]

To make this boot-resistant, add br_netfilter to the list of kernel modules to load at boot time:

echo "br_netfilter" | sudo tee -a /etc/modules

Or into a dedicated file:

echo "br_netfilter" | sudo tee /etc/modules-load.d/netfilter.conf

This assumes you are using a Linux distribution with Systemd init, systemd-modules-load.service is responsible for loading these mentioned modules.

[WillNilges]

Still getting the ip_forward error on Raspberry Pi OS (Debian 11) and Kube 1.26

        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

Anyone got a boot-resistant solution for that one?

[pacoxu]

An example from kubespray: https://github.com/kubernetes-sigs/kubespray/blob/36e5d742dc2b3f7984398c38009f236be7c3c065/Vagrantfile#L224.

Another example in kubeadm blog: https://kubernetes.io/blog/2020/02/07/deploying-external-openstack-cloud-provider-with-kubeadm/#install-docker-and-kubernetes

[samnu88]

try
modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/ipv4/ip_forward