Support specifying kube-proxy tolerations

[discordianfish]

FEATURE REQUEST

kubeadm deploys kube-proxy with these tolerations:

      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      - effect: NoSchedule
        key: node.cloudprovider.kubernetes.io/uninitialized
        value: "true"

This allows kube-proxy to run on masters but not any other tainted hosts. In my case, I want to taint a group of hosts to reserve them for special workloads. But they still need to run kube-proxy. I could just edit the daemonset directly but would prefer a declarative way (e.g kubeadm config) and I'm worried that kubeadm might revert this change on upgrades/replacing masters.

Alternatively the toleration could also just be made to apply to all NoSchedule taints which would solve my problem without introducing a new flag/config.

[discordianfish]

I can confirm: kubeadm init seems to revert the tolerations, so this is required for prod operations.

[discordianfish]

Related #609
A simple stop-gap would be to not touch the kube-proxy ds if possible.

@luxas Both issues are today the biggest problems we have using kubeadm in a prod/HA cluster. I can help fixing them if you can tell me how you'd like to approach these things. I can also bring that up somewhere more suitable to discuss this, don't really know where though.

[timothysc]

/assign @chuckha @detiber

[discordianfish]

Any ideas already how to approach this? This hit us multiple time now and I'd be happy helping to fix this.

For something as fundamental as kube-proxy maybe just having a generic NoSchedule toleration would be fine?

[timothysc]

/assign @fabriziopandini

[timothysc]

@discordianfish - we're going to address in the 1.11 cycle, but the backlog is pretty Y'uge atm.

[discordianfish]

@timothysc That's why I'm offering my help :)
So if there is a actionable plan or route to get there, I can look into it.

[discordianfish]

So I think there are two options:

1. Tolerate all taints, that's trivial to change

2. Tolerate only specific taints but make it possible to use taints for special nodes (eg. GPU workers like in my case)

3. Would be very involved since it would require a mechanism to update the kube-proxy manifest during runtime, all things given. The best way I can imaging to solve this would be changing the taints to allow them to specify pods for which it not applies. This way I could taint my workers for all pods except kube-proxy or maybe everything in kube-system.

So I'd just go with 1., at least for now. I think it's very rare that somebody wants to prevent kube-proxy from running on their nodes. At least it's arguably much more likely that somebody tainting their nodes want kube-proxy on it than not. Since it's so trivial, I'll just submit a PR for this where this approach can be discussed.

[mxey]

This change was reverted/broken by the next commit to the manifest, kubernetes/kubernetes@d194926#diff-e3ad35b550d4fcbf99d00903a91c787e

[neolit123]

^ @dixudx WDYT?

[luxas]

@mxey Yeah 😕
Let's fix this in v1.11.1

[timothysc]

@kubernetes/sig-cluster-lifecycle-bugs - who is actively working on re-fixing this one?

[neolit123]

i pinged @dixudx earlier because he made the last change in this space.
@dixudx, do you think you have time to fix this?

[luxas]

/assign @neolit123
for fixing this on master & cherrypicking