ip6tables bug causing malfunctioning of host ports redirection on Kubernetes

[yann-soubeyrand]

Image I'm using:

BOTTLEROCKET_x86_64 1.13.1-32e9bb46 AMI on EKS managed node group.

What I expected to happen:

Pod host ports working.

What actually happened:

Only the first host port of the pod is working.

How to reproduce the problem:

On an EKS IPv6 cluster with a Bottlerocket node group, deploy a pod with several host ports. Only the first host port will work.

After tracking down the issue to missing iptables rules, I ended up instrumenting the portmap CNI plugin binary with logs in this function https://github.com/containernetworking/plugins/blob/8813bfea7b706ee02acfa9687369c7ce0983bf02/pkg/utils/iptables.go#L125:

func InsertUnique(ipt *iptables.IPTables, table, chain string, prepend bool, rule []string) error {
	log.Printf("inserting: %#v in %s\n", rule, chain)
	exists, err := ipt.Exists(table, chain, rule...)
	if err != nil {
		return err
	}
	if exists {
		log.Println("already exists")
		return nil
	}
	log.Println("does not exist")

	if prepend {
		log.Println("inserting")
		return ipt.Insert(table, chain, 1, rule...)
	} else {
		log.Println("appending")
		return ipt.Append(table, chain, rule...)
	}
}

I was surprised to see that the rules for the first host port were added correctly whereas the rules for the remaining host ports were said to already exist. Manually issuing the ip6tables commands seems to confirm the issue. There’s an open ip6tables bug report describing the problem: https://bugzilla.netfilter.org/show_bug.cgi?id=1667.

[markusboehme]

Hi, thanks for the report and all your debugging work so far! This has been helpful in pinpointing the issue which indeed is caused by a commit introduced in iptables v1.8.9 (eb2546a84677, where the call to make_delete_mask in check_entry should pass sizeof(*fw) instead of sizeof(fw)). I will prepare a patch and propose it upstream.