run control-plane as non-root

[neolit123]

KEP
https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/kubeadm/2568-kubeadm-non-root-control-plane
k/e issue: kubernetes/enhancements#2568

This KEP proposes that the control-plane in kubeadm be run as non-root. If containers are running as root an escape from a container may result in the escalation to root in host. CVE-2019-5736 is an example of a container escape vulnerability that can be mitigated by running containers/pods as non-root.

kubeadm feature gate is called RootlessControlPlane

ALPHA 1.22:

• code changes:
  • seccomp = runtime/default:
    runtime/default Seccomp Profile in kubeadm control-plane components. kubernetes#100234
  • add feature gate:
    Add a feature-gate to kubeadm to enable/disable rootless control-plane. kubernetes#102158
  • add utils / constants:
    Add utils to add and remove users and groups to kubeadm. kubernetes#102195
    kubeadm: add utilities to manage users and groups kubernetes#102463
    Add user and group name constants for kubeadm rootless control-plane. kubernetes#102494
    Add utils to set file/directory owners and permissions. kubernetes#102604
  • (on the side) pipe dry-run option to static pod manifest utils:
    Update CreateInitStaticPodManifestFiles, CreateStaticPodFiles and CreateLocalEtcdStaticPodManifestFile to take into account if the command was run as dry-run. kubernetes#102722
  • run CP components as non-root:
    Update kubeadm control-plane to run as non-root. kubernetes#102759
  • run etcd as non-root:
    Update etcd in kubeadm to run as non-root. kubernetes#102862
  • fix bug in "download-certs" and permissions:
    kubeadm: fix wrong check for keys/certs during "download-certs" kubernetes#103313
  • Fix incorrect user and group for kube-scheduler when it is running as non-root. kubernetes#103380
  • Remove users and groups created as part of rootless control-plane in kubeadm. kubernetes#101988
• e2e tests
  Add e2e tests for rootless control-plane. #2511
  Add e2e tests for rootless control-plane in kubeadm. test-infra#22676
  kinder: use the "auto" mode for the "rootless" workflow #2520
  Use pgrep -f to do a full match and use regular expression to make su… #2521
  kinder: include upgrade artifacts in rootless workflow #2522

BETA x.yy:
on hold until further notice we are watching the user namespaces KEP:
kubernetes/enhancements#3065

• update KEP to tag Beta
• start tracking the KEP in release spreadsheets (task for kubeadm leads)
• make code changes in kubeadm (enable FG by default, test if upgrades work as is?)
• update docs pages:
  • https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/ would need to include some details on how the feature works under the hood - e.g. the user/group add/remove magic.
  • https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init should include a section about the new feature gate. and how to turn it off. we need to remove the section when the FG is GA.
• update e2e tests?
  https://github.com/kubernetes/kubeadm/blob/main/kinder/ci/tools/update-workflows/templates/workflows/rootless-tasks.yaml
  we need to ensure that we have test coverage for upgrading from "FG off" -> "FG on" by default and then checking if CP is rootless.

[neolit123]

/assign vinayakankugoyal

[neolit123]

cc @vinayakankugoyal

[vinayakankugoyal]

/assign vinayakankugoyal

I don't think this can be assigned to me because I am not a kubernetes org member. But to anyone following this bug, I will be working on it.

[vinayakankugoyal]

Can we update the
add feature gate: link in the description above to kubernetes/kubernetes#102158

[neolit123]

@vinayakankugoyal https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/#how-to-use-a-csi-volume
if the rootless kubeadm apiserver eventually becomes ON by default, would it break CSI driver users?

[vinayakankugoyal]

@vinayakankugoyal https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/#how-to-use-a-csi-volume
if the rootless kubeadm apiserver eventually becomes ON by default, would it break CSI driver users?

no because it is not the kube-apiserver that needs to run as privileged pod, it is the csi driver that needs to run as privileged pod. --allow-privileged=true allows privileged containers it does not make kube-apiserver's container privileged. (Same for kubelet but that is anyways out of scope of this KEP.)

[vinayakankugoyal]

/assign vinayakankugoyal

[vinayakankugoyal]

Can we update e2e section above with PR: #2511

[neolit123]

our e2e for this feature started failing yesterday. i have no explanation for the time being. but i don't think it's a kubeadm problem, so maybe something in core?
#2750

[pacoxu]

our e2e for this feature started failing yesterday. i have no explanation for the time being. but i don't think it's a kubeadm problem,
so maybe something in core? #2750

Yes

kubernetes/kubernetes#113548 may fix it. (a revert of kubernetes/kubernetes#113408 that was merged hours before that. )

[neolit123]

it looks like the job has been green for a while, so maybe something else fixed it. the failures were in late august. i completely forgot about this..

https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-rootless-latest

[pacoxu]

I opened the test grid(You post months ago) and find it failed yesterday(😓).

kubernetes/kubernetes#113548 may fix it. (a revert of kubernetes/kubernetes#113408 that was merged hours before that. )

Yesterday's failure is caused by that. Not failures in August.😄

[sftim]

Is this actually important-longterm? It's been a few years.

[pacoxu]

/remove-priority important-soon

Is this actually important-longterm? It's been a few years.

This feature is an alternative way for the user namespace feature. As we prefer to use the user namespace to gain the security control plane in the future, we decided to not promote this one to beta. But we should keep this FG until user namespace kubernetes/enhancements#127 is beta.

[pacoxu]

kubernetes/enhancements#127
User Namespace is beta in v1.30. We may start the deprecation of RootlessControlPlane in v1.31.

[LyKos4]

Is this expected to be completed? For which pods it is expected to change the user to non root?

[neolit123]

Is this expected to be completed? For which pods it is expected to change the user to non root

this feature is alpha and deprecated. please use UserNamespaces instead:
kubernetes/enhancements#127

once UserNamespaces becomes GA kubeadm will enable it by default.