k8s: In control loops _3-way merge_ is the king

[AmitKumarDas]

Introduction

Control loops seem to be a viable approach to solve a problem eventually. I believe they epitomise slow is fast, fast is slow adage in its truest sense. It is a slow but sure way to get your desired intent match the real word representation. No doubt, Kubernetes has found this as an effective pattern to solve some of the most complex infrastructural challenges that modern days engineers are faced with. Kubernetes in-fact has nicely exposed control loop as a building block to design any extensions over & above Kubernetes.

Into the diff

While above was my representation of control loops, this article focuses on one of the core ingredients to let a control loop work. This ingredient is diff i.e. the difference between observed versus desired states. To a Kubernetes developer, this boils down to finding the diff and taking subsequent actions.

We have several libraries e.g. reflect.DeepEqual to flag a diff and cmp.Diff to get us the differences. However, these alone will not suffice to create an effective control loop system. In my opinion, having a 3-way merge strategy popularly understood as kubectl apply is a must have tool to design better control loops.

Need for 3-way merge

While use of control loop based tools is one thing, knowledge of current state of system is a different matter altogether. It is quite complex to get the current state of system and base all our design decisions by comparing each property of this current state against the desired property. For example, do we really need to know the current container image & version of a Pod to make subsequent decisions? In fact we might only be interested about the desired image of our Pod given its name or labels. How about a continuous stream of updates of the desired state as an approach to build control loops? Continuous updates seems to be simple but can not be scaled, will interfere with other updates, and will load the servers to the extent of them being unresponsive.

What if the designers of control loop need to bother only about their desired state to implement this reconciliation system? 3-way merge is one such strategy that can make this happen. 3-way merge makes use of observed, last applied & desired state to get to the final state. If there are no changes then final state is always equal to observed state, hence reducing the need for continuous updates to the system. On the whole designers are left to concentrate only about their desired state without bothering to calculate the diff between each property of the observed state to that of the desired state.

Conclusion

Kubernetes has understood this need and has started working on a separate working group called wg-apply (apply in K8s world implies 3-way merge). Everyone is encouraged to contribute, test & understand how its works. There is yet another kubernetes control loop library which uses this concept & has been around for a long long time now. I am talking about Metacontroller which use this effectively to create higher order controllers known as meta-controllers.