enhance(kubernetes): Add podTracker and containerTracker to use k8s API more like a k8s controller

[cognifloyd]

Part of go-vela/community#519 which is about moving where we watch the Kubernetes API (which is async) from the container-level up to the pod/build-level.

Overview

Kubernetes controllers (eg that watch Deployments to create ReplicaSets) manage their API calls very carefully to minimize load on the API servers. Though we don't need to do everything that a typical Kubernetes controller does, we can reuse the utilities built for controllers to improve how we work with the async Kubernetes APIs. (see "Why SharedInformers" below for a more verbose description).

Implementing this includes:

• Update Kubernetes Runtime's WaitContainer to use a SharedInformer
• Update Kubernetes Runtime's InspectContainer to use a SharedInformer
• Add StreamBuild method to Runtime engine interface so we can use the streamCtx introduced in enhancement: Make context handling for log streaming more explicit #317
• New podTracker and containerTracker to collect the SharedInformer-related state and methods.

Why SharedInformer?

TL;DR; to better manage the async nature of K8s APIs, reusing the tools built for k8s controllers.

Informers were created for Kubernetes Controllers to efficiently run a variety of List and Watch API calls. Shared in the name SharedInformer refers to how the same API connections can be shared for multiple Watch calls for a variety of resources, thus minimizing the load on the Kubernetes API servers.

Many controllers watch for changes in one resource to trigger add/update/delete in another resource. For example, a Deployment controller watches for changes in Deployments and then creates/modifies ReplicaSets based on the spec in a Deployment.

Vela does not need to watch for resources to create/modify other resources, but it does want to efficiently watch the Pods it creates. So, the SharedInformer works marvelously for that.

In the future, it might be nice to create a single worker-wide SharedInformer that watches for changes to all pipeline pods managed by a given worker. But, the Kubernetes clientset currently gets created for each build, so allowing a Runtime to have long-lived (outside of the build) worker goroutines would involve a much more significant architectural change. So, I'm punting that for later. Lowering the number of watches down to per-build instead of per-build-container is already much much better.

WaitContainer changes in k8s runtime

Instead of running a separate Pods().Watch() API call for each container, this:

• uses a SharedInformer to watch the pod for the entire build. The SharedInformer:
  • manages its own Watch() API calls - restarting them as needed,
  • periodically resyncs with List() calls to make sure no events were lost,
• uses a chan struct{} to signal when each container terminates using events from the SharedInformer

Now, WaitContainer just has to wait for the appropriate container's chan struct{} to be closed before returning.

InspectContainer changes in k8s runtime

This also changes InspectContainer to get the latest cached data from the SharedInformer.
Since the Infomer uses Watch and periodic List, the cached value should be up-to-date. So, we don't need a separate API call.

StreamBuild changes in executor + runtime interface

#317 improved the executor context handling so that streaming is canceled separately from the execution deadline (amongst other reasons). We want to use that higher-level context when capturing all the events for the build from Kubernetes.

So, add a StreamBuild method to the Runtime engine interface (mirroring the StreamBuild Executor engine method).

For the Docker runtime, this is a no-op.

For the Kubernetes runtime, this is where we get tell our SharedInformer to begin populating its internal cache with details from Kubernetes about our build. Then it will continue to stream those events until the end of the build (ie just after execution finishes).

New structs: podTracker, containerTracker

In order to implement go-vela/community#519, I need to move a variety of API calls from container-level methods to build-level (pod-level) methods.

The podTracker is meant to be similar to k8s.io/sample-controller.Controller encapsulating the SharedInformer and related bits.

podTracker.Containers is map[string]*containerTracker. Each containerTracker holds the signal channel (chan struct{}) that indicates when the container has terminated. In future PRs, it will also hold whatever types we need to better manage streaming k8s events and pod\logs.

ps: I lost count of how many times I rewrote this before coming up with podTracker/containerTracker 😰 .

[codecov]

Codecov Report

Merging #302 (9b50bb1) into master (cd90c6f) will increase coverage by 0.56%.
The diff coverage is 93.69%.

@@            Coverage Diff             @@
##           master     #302      +/-   ##
==========================================
+ Coverage   80.62%   81.18%   +0.56%     
==========================================
  Files          67       68       +1     
  Lines        5089     5273     +184     
==========================================
+ Hits         4103     4281     +178     
- Misses        844      848       +4     
- Partials      142      144       +2     

Impacted Files	Coverage Δ
runtime/kubernetes/kubernetes.go	89.18% <72.72%> (-1.90%)	⬇️
runtime/kubernetes/build.go	87.04% <83.33%> (-0.86%)	⬇️
runtime/kubernetes/pod_tracker.go	95.86% <95.86%> (ø)
executor/linux/build.go	74.88% <100.00%> (+0.30%)	⬆️
executor/local/build.go	77.66% <100.00%> (+0.39%)	⬆️
runtime/docker/build.go	100.00% <100.00%> (ø)
runtime/kubernetes/container.go	82.03% <100.00%> (+1.81%)	⬆️
runtime/kubernetes/volume.go	70.73% <100.00%> (+0.23%)	⬆️

[cognifloyd]

Impacted Files	Coverage Δ
runtime/kubernetes/build.go	85.71% <62.50%> (-2.42%)	⬇️
runtime/kubernetes/kubernetes.go	88.99% <72.72%> (-1.92%)	⬇️

I've reviewed each of the uncovered lines I added. They are all err check blocks, but I couldn't figure out how to reach it. So, I'm going to leave these as is.

[wass3r]

thanks for all the work on this 🤞🏼

i would call it an enhancement (to answer your question). logging was there before, but you are trying to improve it, though this is not all about logging only.

[cognifloyd]

I converted this to draft for now. Too many things are odd. I'll reopen once I'm satisfied issues are resolved (probably via other PRs).

[cognifloyd]

OK. This is ready for review now.

[wass3r]

thanks for the detailed description and work 👍🏼 i can see that the line is indeed a bit blurred between whether to call this an enhancement or feature. it's a feature within the kubernetes runtime but an enhancement to the implementation. either way, lgtm

[jbrockopp]

LGTM

[kneal]

LGTM 🐬