linkerd-init: start fail, key msg RULE_APPEND failed : rule in chain PREROUTING

[hikari1124]

What is the issue?

linkerd-init start fail

How can it be reproduced?

when you linkerd inject to deployment yaml, the issue is always happen

Logs, error output, etc

time="2024-10-11T07:57:39Z" level=info msg="/sbin/iptables-nft-save -t nat"
time="2024-10-11T07:57:39Z" level=info msg="# Generated by iptables-nft-save v1.8.10 (nf_tables) on Fri Oct 11 07:57:39 2024\n*nat\n:PREROUTING ACCEPT [0:0]\n:INPUT ACCEPT [0:0]\n:OUTPUT ACCEPT [0:0]\n:POSTROUTING ACCEPT [0:0]\n:PROXY_INIT_REDIRECT - [0:0]\n-A PROXY_INIT_REDIRECT -p tcp -m multiport --dports 4190,4191,4567,4568 -m comment --comment "proxy-init/ignore-port-4190,4191,4567,4568" -j RETURN\n-A PROXY_INIT_REDIRECT -p tcp -m comment --comment "proxy-init/redirect-all-incoming-to-proxy-port" -j REDIRECT --to-ports 4143\nCOMMIT\n# Completed on Fri Oct 11 07:57:39 2024\n"
time="2024-10-11T07:57:39Z" level=info msg="/sbin/iptables-nft -t nat -F PROXY_INIT_REDIRECT"
time="2024-10-11T07:57:39Z" level=info msg="/sbin/iptables-nft -t nat -A PROXY_INIT_REDIRECT -p tcp --match multiport --dports 4190,4191,4567,4568 -j RETURN -m comment --comment proxy-init/ignore-port-4190,4191,4567,4568"
time="2024-10-11T07:57:39Z" level=info msg="/sbin/iptables-nft -t nat -A PROXY_INIT_REDIRECT -p tcp -j REDIRECT --to-port 4143 -m comment --comment proxy-init/redirect-all-incoming-to-proxy-port"
time="2024-10-11T07:57:39Z" level=info msg="/sbin/iptables-nft -t nat -A PREROUTING -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/install-proxy-init-prerouting"
time="2024-10-11T07:57:39Z" level=info msg="iptables v1.8.10 (nf_tables): RULE_APPEND failed (Invalid argument): rule in chain PREROUTING\n"
Error: exit status 4
Usage:
proxy-init [flags]

Flags:
--firewall-bin-path string Path to iptables binary
--firewall-save-bin-path string Path to iptables-save binary
-h, --help help for proxy-init
--inbound-ports-to-ignore strings Inbound ports and/or port ranges (inclusive) to ignore and not redirect to proxy. This has higher precedence than any other parameters.
-p, --incoming-proxy-port int Port to redirect incoming traffic (default -1)
--iptables-mode string Variant of iptables command to use ("legacy" or "nft"); overrides --firewall-bin-path and --firewall-save-bin-path
--ipv6 Set rules both via iptables and ip6tables to support dual-stack networking (default true)
--log-format string Configure log format ('plain' or 'json') (default "plain")
--log-level string Configure log level (default "info")
--netns string Optional network namespace in which to run the iptables commands
--outbound-ports-to-ignore strings Outbound ports and/or port ranges (inclusive) to ignore and not redirect to proxy. This has higher precedence than any other parameters.
-o, --outgoing-proxy-port int Port to redirect outgoing traffic (default -1)
-r, --ports-to-redirect ints Port to redirect to proxy, if no port is specified then ALL ports are redirected
-g, --proxy-gid int Group ID that the proxy is running under. Any traffic coming from this group will be ignored to avoid infinite redirection loops. (default -1)
-u, --proxy-uid int User ID that the proxy is running under. Any traffic coming from this user will be ignored to avoid infinite redirection loops. (default -1)
--simulate Don't execute any command, just print what would be executed
--subnets-to-ignore strings Subnets to ignore and not redirect to proxy. This has higher precedence than any other parameters.
--timeout-close-wait-secs int Sets nf_conntrack_tcp_timeout_close_wait
-w, --use-wait-flag Appends the "-w" flag to the iptables commands

output of linkerd check -o short

➜ ~ linkerd check -o short
Status check results are √

Environment

kubernetes version: 1.27.11
linkerd version: edge-24.10.2
host os: anolisos 8 (equal centos 8)
cluster env: self host by sealos

Possible solution

rule append duplicately?

Additional context

No response

Would you like to work on fixing this bug?

None

[adleong]

Thanks for reporting this, @hikari1124! I haven't seen this error before, so it may be specific to your OS or kernel version.

Some searching online for this error messages turns up a few results, mostly from people who are equally as confused as I am about how iptables works 😂

https://superuser.com/questions/1558335/rule-append-failed-iptables suggests that we might be missing the -p tcp flag in some of our iptables commands, but it's unclear to me when that's necessary and in what environments. This would require some experimentation to figure out what combinations of flags work in which environments.

https://serverfault.com/questions/999737/iptables-v1-8-2-nf-tables-rule-append-failed-invalid-argument-rule-in-chai suggests that upgrading to kernel 4.19 might fix the issue.

Another workaround to potentially try would be to try using iptable in legacy mode by installing Linkerd with:
linkerd install --set "proxyInit.iptablesMode=iptables" | kubectl apply -f -. (See https://linkerd.io/2-edge/features/nft/ for more details). Unclear if this would fix the problem or not, but it might be another thing to try.