Ipv4 no masq

[rdesaintleger]

Proposed Changes

Add the flag --flannel-ipv4-no-masq to k3s to disable IPv4 masquerade. Default k3s behaviour do not change and IPv4 masquerade is still activated if the flag is not specified.

Disabling NAT allows to replacing with regular BGP routing/advertisement using metallb FRR (for example). In my case metallb advertise local connected routes for each nodes as well as LoadBalancer services.

Types of Changes

This change is a new feature for the k3s flannel configuration (both for command line and yaml config). The advantage of flannel CNI beside being light is to be able to run in unprivileged incus/lxd/lxc containers.

Verification

I've installed my node with the following command line (in a debian bookworm unprivileged incus container) :

cp /root/k3s /usr/local/bin && chmod 755 /usr/local/bin/k3s && \
curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_DOWNLOAD=true sh -s - server --cluster-init \
	--cluster-domain=k3s.roroland.net  \
	--flannel-ipv4-no-masq \
	--flannel-backend=host-gw \
	--cluster-cidr=172.31.128.0/17,fda5:afa6:836c:007f:0:fe::/96 \
	--service-cidr=172.31.64.0/18,fda5:afa6:836c:007f:0:fd::/112 \
	--kubelet-arg=feature-gates=KubeletInUserNamespace=true \
    --kube-controller-manager-arg=feature-gates=KubeletInUserNamespace=true \
    --kube-controller-manager-arg=node-cidr-mask-size-ipv6=112 \
    --kube-apiserver-arg=feature-gates=KubeletInUserNamespace=true \
	--disable servicelb

If you wish to check with BGP routing I also give the metallb configuration:

First install the metallb CRD
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.9/config/manifests/metallb-frr-k8s.yaml

Then apply the following configuration (adjust IPs according to your installation)

kubectl apply -f - << _EOF_
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: lb-pool
  namespace: metallb-system
spec:
  addresses:
  - 172.31.63.0/24
  - fda5:afa6:836c:007f:0:fd:1:0/112
---
apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
 name: saga
 namespace: metallb-system
spec:
 myASN: 65020
 peerASN: 65030
 peerAddress: 172.31.7.30
---
# also receive/advertise internal services
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: k8s-services
  namespace: metallb-system
spec:
  bgp:
    routers:
    - asn: 65020
      neighbors:
      - address: 172.31.7.30
        asn: 65030
        port: 179
        toAdvertise:
          allowed:
            mode: all
      prefixes:
        - 172.31.64.0/18
        - fda5:afa6:836c:007f:0:fd::/112
---
# add manual configuration to advertise cluster node sub-networks
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: k8s-redistribute
  namespace: metallb-system
spec:
  raw:
    priority: 5
    rawConfig: |-
      ip prefix-list local-subnets-ipv4 seq 1 permit 172.31.128.0/17 ge 17
      ipv6 prefix-list local-subnets-ipv6 permit fda5:afa6:836c:007f:0:fe::/96 ge 96
      route-map 172.31.7.30-out permit 504
        match ip address prefix-list local-subnets-ipv4
      exit
      route-map 172.31.7.30-out permit 506
        match ipv6 address prefix-list local-subnets-ipv6
      router bgp 65020
        address-family ipv4 unicast
          redistribute connected
        exit-address-family
        address-family ipv6 unicast
          redistribute connected
        exit-address-family
      exit
---
apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: k8s-advert
  namespace: metallb-system
_EOF_

Testing

Did not write a test.

Linked Issues

User-Facing Change

NONE

Further Comments

I've seeked a long time the lighest solution for machines which have from 2G to 4G RAM. I've tested calico and cilium which either privileged containers or virtual machine/bare metal. So far, the best solution for me is to stick with flannel. The only problem that I had was with the hardcoded masquerade. This is why I did this PR.

[brandond]

I don't think we really want to add any more top-level flags for flannel.

The ADR you're touching here never made it past discussion, but we've recently considered bringing it up again. If we did do this, I think the option you're proposing should go in --flannel-opt instead of being a new top-level flag.

cc @manuelbuil @rbrtbnfgl

[rdesaintleger]

I can remove the global flag and implement the --flannel-opt if you wish. I did not implement the ADR because its status is not Accepted.

If I understand well the ADR and discussions in #6557 and #6897

--flannel-backend stays in global flags,
--flannel-opt is used for all other flags

Can I go with the ADR implementation now or do I have to wait for its status to change ? Is implementing the ADR is the way to go is the option name ipv4-no-masq OK ?

[brandond]

We're discussing internally.

[rdesaintleger]

Hi,

I did not see progress on the flannel ADR. May I suggest to deprecate the --flannel-ipv6-masq and replace it with --flannel-masq, taking the following values:

• none
• ipv4 (default)
• ipv6
• both

Except for the deprecation period, this would not add a new flannel top-level flag but replace the existing one, adding the ability to disable ipv4 masquerading. I can implement (or help to) this solution as well as the the flannel opt if you wish. Please let me know.

[rdesaintleger]

Dropping PR. Found a way to do it using kube-router.

Kube-router can replace flannel/metallb, announces its routes over BGP and allows k3s to run in an unprivileged incus container. For those who can use Virtual machines, cilium should be a better shot.