Support Automatic Proxy Uninjection

[ihcsim]

Feature Request

What problem are you trying to solve?

Automate the proxy uninject experience, so that a user can utilize the namespace and pod annotations to automatically uninject previously injected proxies. Currently, the proxy injector doesn't honor the linkerd.io/inject: disabled annotation during workload updates.

Continuation of #2260.
Depends on #2289.

How should the problem be solved?

The proxy-injector needs to be updated to check for the presence of the linkerd.io/inject: disabled annotation at both the namespace and pod level. If the annotation is present, the proxy-injector should issue JSON patches to the k8s api server to remove any injected proxy.

We need to consider if the absence of the annotation implies auto-uninject should happen.

(We will also need to do a quick spike to see if the spec.template.containers spec is mutable via JSON patch.)

The effect of auto-uninjecting an entire namespace also needs extra consideration. What is a user's intention when she either annotates a namespace with linkerd.io/inject: disabled or removes the annotation entirely? To trigger auto uninject on all workloads on next update, or just to pause all future auto-injection?

Perhaps we should only support auto-uninject at the workloads level, not the namespace level.

Any alternatives you've considered?

Just stick with manual uninject via the CLI.

How would users interact with this feature?

Via the linkerd.io/inject annotation.

[alpeb]

Now that #2471 has been merged, adding a linkerd.io/inject:disabled at the pod's parent workload has the effect of the pod being re-created and the auto-injector doesn't inject the proxy.

However when we deal with orphaned pods (with no owner references like ReplicaSet->Deployment) that are already injected, adding that annotation won't have the effect of removing the proxy. The auto-injector does receive the UPDATE event, but it ignores it because the workload is already injected (re shouldInject() function). But even if we changed that so that the auto-injector returned a JSON patch removing the proxy, it turns out Kuberenetes forbids adding/removing containers into an existing workload through the mutating webhook mechanism.

So the only option for uninjecting orphaned pods is by manually deleting them and creating them again with that annotation. Something that should be documented in the website's annotation docs (#222)

[grampelberg]

It feels like this works in spirit at least now. Closing out.