Cluster with kubespan and multiple disconnected private networks - possible?

[miran248]

Hey,

With 1.8.0 out i decided to work on multi network/region support on my setup.
Everything works, except for machines, located in a different network from control planes - those don't have ExternalIP, because control planes can't reach them.
Since all machines have private networks attached, cluster defaults to their internal ips, instead of the public ones.

machine.kubelet.nodeIP.validSubnets only contains the cidr of the attached private network, if i add public ips to the list, they get added to the node's InternalIP section, which then breaks cilium as ExternalIP isn't defined.
I tried filtering out private networks from kubespan but it wasn't enough.
Maybe, if i could somehow force talos-ccm to always define the ExternalIP, it might be enough, at least on talos' side.

What i'd like to achieve is to have all machines linked using kubespan (usual, as if there were no private networks)
and also have some of them linked via private network, to lower the latency.
Imagine having a cluster, spanning two regions, A and B.
All traffic on machines A would go through private network A (same for B), and all traffic between A and B would go over public internet.

Is this possible?
It seems i'd need to define multiple "regional" clusters, without kubespan, each with one private network, and then link them together using cilium cluster mesh (or keep them separated).
That, or peer the networks, but then i'd be moving the problem from one area to another.
Or, maybe, i should just remove the private networks and be done with it. It would even allow me to make the cluster single-stack.
Here is the generated machine config for one of the control planes - it's dual stack, with private ipv4 networks and public ipv6 ips.
(i use hetzner cloud, but this isn't specific to hcloud)

Thanks,
Miran

[smira]

KubeSpan requires 1-way connectivity between every pair of machines in the cluster.

You can limit KubeSpan on announced endpoints via https://www.talos.dev/v1.8/reference/configuration/v1alpha1/config/#Config.machine.network.kubespan, but that would limit it globally, for all peers.

Probably what you're asking is partially #8364

[miran248]

I believe kubespan segmentation would solve my issue, indeed!
Will leave it for now and instead make private networks optional in my setup, that way i'll bypass the issue entirely.
Then, once segmentation arrives (and kernel 6.12) we'll be golden!

Thanks!