Support Configs Override On Meshed Workloads

[ihcsim]

Following discussion in 2471, we have agreed to re-visit the design around ways to support configs override during resource update. In general, we want the inject library to honor the config overrides annotations, even though the proxy injection process is skipped (most relevant to existing meshed workloads).

The approach currently used in #2471 causes the proxy configs to always be overridden, whether (auto or manual) proxy injection happens or not. It is uncertain if this will have unintended consequences on workloads that skipped the injection process. In addition, it also leads to different code path for supporting meshed and non-meshed workloads.

This task should commence immediately, after #2471 is merged.

[ihcsim]

With #2472, we can now override the configs of existing auto-injected workloads. It's worth pointing out that at runtime, the proxy spec and configs will not appear in the owner's pod template, because the proxy injector webhook mutates the pods directly, during the proxy injection process.

Following discussion with @grampelberg and @alpeb, we won't be supporting configs override of existing CLI-injected workloads for stable-2.3. (Configs override of new CLI-injected workloads will still work.) We will pick it up again in stable-2.4, after #2576 is resolved. This decision will be included in the 2.3 docs as part of linkerd/website#222.

As discussed in #2576, if we allowed existing CLI-injected workloads to override their proxy configs (via kubectl edit etc.) by loosening the HasExistingSidecar() check, the resulting runtime specs of the pod will be different from its owner, leading to a confusing UX. For example, if I deploy a workload with linkerd inject my-deployment.yaml | kubectl apply -f -, the proxy will inherit a set of default configs, say proxy UID 2012. If I then use kubectl edit to add the config.linkerd.io/proxy-uid: 5555 to the deployment, the annotation will be correctly added, but the deployment's YAML will remain un-edited and the pod's YAML will be edited.

The outstanding task will be tracked in a separate issue.