General Design/Architecture Issue: Routing to the k8s Service CIDR

[kgrotel]

What the operator currently does is deploy Netbird clients into a Kubernetes cluster and route Kubernetes services (defined via HTTPRoute) out of it. The ClusterIP of each service is assigned to these deployed Netbird clients as a routable IP - and that's basically it.

You could achieve the same thing without deploying any Netbird clients into Kubernetes at all, since the service CIDR is reachable from any k8s node. Just create a Netbird network with the k8s nodes as routing peers, add the service CIDR as a resource, create some DNS entries pointing to those IPs, and you're done.

But here's the catch: as long as a subnet/CIDR is pinned to a single host and only routed locally, you can do whatever you want. The moment you turn it into a routable network across your infrastructure, you have to make sure it's well-defined and that routing is unambiguous - no loops, no collisions, and so on.

Kubernetes service CIDRs are very large - something like 10.96.0.0/12 by default. That's for a single cluster, and the range has to accommodate a unique IP for every service across the whole cluster. ClusterIPs get allocated unpredictably across the entire range, so to catch all of them you have to advertise the whole CIDR - even if only a handful of services are live. So even if you only have 100 services you want to expose (through Netbird), you still have to lock down ~1 million IPs and guarantee there are no collisions. And if you add a second cluster, there go another million IPs - and you also have to ensure the two service CIDRs don't overlap. That's why people don't actually do it this way and use load balancers instead: you use a much smaller subnet and only route the services you actually need.

Where this leaves the operator. There are really two coherent directions it can go, in my opinion. Either it becomes a full-fledged Kubernetes load balancer in its own right - owning a small, dedicated LB CIDR, assigning addresses from it to Service objects, and advertising only those addresses into the Netbird network. Or it attaches to an existing LB (Cilium LB-IPAM, kGateway, MetalLB, etc.), letting that component own IP allocation while the operator manages the overlay side: advertising the LB CIDR through a routing peer, wiring up DNS for the allocated addresses etc..

[phillebaba]

Every service is not exposed to NetBird just because you run a NetworkRouter. Only those that have a NetworkResource attached will be reachable from outside of the cluster.

I am not really following your argument about the operator becoming a load balancer. Could you give more details?

[kgrotel]

.. and i did not mean that the operator should become a loadbalance (this was bad idea on my side) but only expose services of type: loadbalancer which gets its ip from a loadbalancer or it becomes a kind of proxy that tranlate extenal IPs to Services with ClusterIP. But the first approach is by far simpler

[phillebaba]

This is more generally a NetBird problem than a operator one. The same thing happens if you have multiple networks with overlapping IPs. Sure Kubernetes makes this more apparent as most clusters use the same CIDR ranges for services. Right now using different profiles to select specific networks is the only solution. In the future there might be a better option in NetBird.

[kgrotel]

Overlapping Subnets is a general Networking Issue which you work arround with adequate Network planning and that's actually my point. Using the Kubernetes Service CIDR makes network planning kind of hard because its such a huge network. It would scale a lot better using a dedicated ingress network.

Thus instead of the operator defining its own Gateway / HTTProute implementation that routes Service CIDR, use something like kgateway/Istio and watch the status of these Gateway and route kgateway/istios Ingress IP via NetworkResource, these Gateway just need a single IP adress or the Hostport if you like, to expose all the Services you want.

What i currently prefer is even simpler as we use Cilium and Cilium provides a Loadbalancer out of the Box. Define a Subnet ( 172.16.0.0/24) and assign to CiliumLB, define a Networkressource with this Subnet and the Netbird Daemonset are the routing peer, add some DNS magic, and it work quite well with NetbirdReverseProxyService So instead of a claimed 1mio IP subnet you use a fairly small 256 IP Subnet and that makes network planning and avoid overlapping Subnets a lot easier.

[phillebaba]

This is a fundamental different approach to NetBird as a whole and removes a lot of the benefits that a wireguard based overlay network brings. It is out of scope to the operator as it aims to integrate with core NetBird functionality instead of attempting to invent new Kubernetes specific solutions.

[kgrotel]

Its actually not a so different approach, i dont see which benetits are removed as the way trafiic is routed stays the same .. to finish that for now, i just wanted to point out that the current approach has issues with scaling and routing and the longer you wait the more difficult it will get to fix it.