feat(helm): Introduce priorityClassName system-node-critical

[ffuerste]

What does this PR do?

Introducing in the helm charts of all NRI plugins an optional boolean field for setting priorityClassName: system-node-critical.

As suggest here the priorityClassName is introduced as an optional helm flag analog to patchRuntimeConfig. However, in this way the feature can only be enabled with no option to define a custom priorityClassName.

Related issues

Resolves #194

[marquiz]

Thanks for the patch @ffuerste. This totally makes sense.

I'm just wondering should be the desired default be true, for at least some of the plugins 🧐 If a resource policy plugin (like balloons or topology-aware) is deployed I think it is kinda critical and should not be evicted (thus default to true). WDYT? @klihub @fmuyassarov @askervin @kad

[askervin]

I think this is a very nice improvement.

I'm fine both merging this PR as it is, or with some/all existing plugins set as system-node-critical by default.

Maybe I'd even prefer taking this PR is as it is, as it only enables configuring the priorityClassName but does not change the default behavior. And then have a separate commit that changes the default behavior for chosen plugins.

[fmuyassarov]

Thanks for the patch @ffuerste. This totally makes sense.

I'm just wondering should be the desired default be true, for at least some of the plugins 🧐 If a resource policy plugin (like balloons or topology-aware) is deployed I think it is kinda critical and should not be evicted (thus default to true). WDYT? @klihub @fmuyassarov @askervin @kad

I don't have a strong opinion about that to be frank. I'm fine with the current state as well.

[kad]

I think it makes sense.

[klihub]

LGTM.

[klihub]

Thanks for the patch @ffuerste. This totally makes sense.

I'm just wondering should be the desired default be true, for at least some of the plugins 🧐 If a resource policy plugin (like balloons or topology-aware) is deployed I think it is kinda critical and should not be evicted (thus default to true). WDYT? @klihub @fmuyassarov @askervin @kad

I'm fine with the default either way for the current plugins.

[marquiz]

Hold until comments are resolved

[marquiz]

Thanks @ffuerste. LGTM on my part