Node Annotations are Only Evaluated on Startup and not Actively Watched During Runtime

[Feder1co5oave]

On an already existing kubernetes cluster with kube-router successfully running in route reflector mode (rr.server annotation on 3 master nodes, rr.client annotation on all the workers), my workflow for joining new worker nodes is like this (the cluster was created with kubeadm):

for each new node:

• run kubeadm join on the node
• kubectl annotate node kube-router.io/rr.client=23
• restart kube-router on the new node to put them in RR mode

Turns out the nodes previously joined to the cluster will receive routes for all the new nodes, but the new nodes will only receive routes for the "old" nodes. Restarting kube-router on the route reflector nodes solved this issue.

My troubleshooting suggested that whenever a new node joins, the rr server kube-router daemons will peer with that node right away, even before the rr.client annotation gets added to it. Thus, the new peer is assumed to be in full mesh mode even if in fact it will be in RR mode and will only peer with RR servers. BGP route reflector allows for BGP daemons in the same AS to peer in either full-mesh or RR mode. Export policies are such that RR servers will reflect advertisements:

• from RR peers to RR peers
• from full-mesh peers to RR peers
• from RR peers to full-mesh peers
• but NOT from full-mesh peers to other full-mesh peers (they're supposed to peer with each other in full-mesh, duh)

So the new nodes, assumed to be forming full-mesh, don't get advertisements about other new nodes.
Restarting RR servers forces them to reload the node list and annotations, and correctly peer with the new nodes in RR mode.

To fix this, kube-router should watch for annotation changes on nodes, and update its internal information about which nodes are forming full-mesh, and which are joined to a RR cluster.
While waiting for a fix to be implemented, I suggest the workaround to restart RR servers be documented for newcomers!

[aauren]

Closed via #677

[ofen]

Is it possible to make it automatically (without manual POD restart)?

[murali-reddy]

@ofen reopening as still below make sense to be fixed.

kube-router should watch for annotation changes on nodes, and update its internal information about which nodes are forming full-mesh, and which are joined to a RR cluster.
While waiting for a fix to be implemented, I suggest the workaround to restart RR servers be documented for newcomers!

[zhaixigui]

Why can't it automatically recognize the newly added node? And to propagate routing。Why have to do such a dangerous operation to kill RR?

[zhaixigui]

Do we have any choice but to kill the RR?

[Feder1co5oave]

@zhaixigui the new nodes are recognized, but they are assumed to be in full mesh mode, and there's no clean way to do the switch to RR mode other than restart kube-router on both RR server and client to reload the nodes' information.

Killing RR servers should not bring any disruption as long as you enabled soft-restart. I've done it several times without any repercussions.

[zhaixigui]

@Feder1co5oave a bad idea？ when the RR server watched a ”kube-router.io/rr.client=42“ annotation from
most recently joined node. it first calls method DeleteNeighbor to delete this node, and then calls method AddNeighbor to add this node as RR client, and RR will automatically propagate this route ？

[zhaixigui]

@zhaixigui the new nodes are recognized, but they are assumed to be in full mesh mode, and there's no clean way to do the switch to RR mode other than restart kube-router on both RR server and client to reload the nodes' information.

Killing RR servers should not bring any disruption as long as you enabled soft-restart. I've done it several times without any repercussions.

soft-restart is GracefulRestart ?

[zhaixigui]

Does calico have the same problem?

[Feder1co5oave]

I don't know. Calico has a great reputation as network plugin and also great features. When I considered it for my clusters I found it is more complicated and has more moving parts than kube-router, so I ended up choosing the latter because of its "simplicity".

[zhaixigui]

@Feder1co5oave a bad idea？ when the RR server watched a ”kube-router.io/rr.client=42“ annotation from
most recently joined node. it first calls method DeleteNeighbor to delete this node, and then calls method AddNeighbor to add this node as RR client, and RR will automatically propagate this route ？

@Feder1co5oave Feder， Is this a bad idea? is there a better idea or solution?

[Feder1co5oave]

@zhaixigui not at all, in fact it is exactly the solution I proposed in my first post. But I'm not a gopher and this would involve a bit of modifications to the control flow, i.e. watching nodes. I believe we'll have to wait for a volunteer to write a PR.

[sebltm]

I think it's as easy as doing something like this: #1723
I've tested it in my local environment, seems to work as expected