Kubernetes 1.22 Major Themes

[pmmalinov01]

Following the steps in the steps of release 1.21, we would like people to keep submitting content over the release cycle, and to prevent the pile-up at the end of it. In the last(and first) iteration of this process, we had quality conversations over the Major Themes.

[saschagrunert]

Proposal: New alpha feature gate for enabling seccomp RuntimeDefault profile for all workloads: kubernetes/enhancements#2413

[saschagrunert]

Proposal

A new alpha feature gate SeccompDefault has been added to the kubelet together with a corresponding command line flag --seccomp-default and kubelet configuration. If both are enabled, then the kubelet will use the RuntimeDefault seccomp profile rather than Unconfined if they're not specified in the Pod or Container SecurityContext. This allows enhancing the default cluster wide workload security of the Kubernetes deployment. Security administrators will now sleep better.

To learn more about the feature, please refer to the official documentation or the feature blog post.

[pmmalinov01]

Thanks, @saschagrunert this looks great. WIll only suggest emphasising this is in alpha.
Suggestion:
An alpha feature gate SeccompDefault is added to the kubelet together with ....

P.S.
I would love to put a comment like "Now security admins will sleep better" at the end, it sounds funny in my head :D

[pmmalinov01]

Do you know the PR for the blog post ? We can build the URL in advance.

[saschagrunert]

Thanks, added your suggestion. The PR to the blog post is kubernetes/website#28951

[pmmalinov01]

Proposal:
etcd moving to version 3.5.0

[pmmalinov01]

Suggestion:
Kubernetes default backend storage of etcd has a new release 3.5.0 and the community embraced it. The new release comes with improvements to the Security, performance, monitoring and developer experience. There are numerous bug fixes to lease objects causing memory leaks, and compact operation causing deadlocks and more. A couple of new features are also introduced like the migration to structured logging and build in log rotation. The release comes with a detailed future roadmap to implement a solution to traffic overload. A full and detailed list of changes can be read at ETCD blog post about the 3.5.0 release

[sftim]

etcd is lowercase (etcd not ETCD)

[pmmalinov01]

Thanks, fixed

[sftim]

https://www.youtube.com/watch?v=C4OojobalLQ might be relevant

[sftim]

Proposal: Lots of API removals

[liggitt]

The blurb should strongly emphasize that the beta versions of these are no longer being served in favor of the GA versions, and that all existing objects created using the beta APIs can already be accessed via the GA APIs (people have gotten very confused/anxious about that point in the past).

[sftim]

We'll almost certainly have a blog article we can link to - see kubernetes/website#28694
(I want to get that article published this week)

[pmmalinov01]

@liggitt @sftim How does the following sound?

A number of APIs are no longer serving specific Beta versions in favour of the GA version of the said APIs. All existing objects can be interacted with via general availability APIs. This removed includes and is not limited to Webhook resources, CustomResourceDefinition, APIService, TokenReview, SubjectAccessReview, CertificateSigningRequest, Lease, Ingress, IngressClass. For the full list checkout Deprecated API Migration Guide and the blog post Kubernetes API and Feature Removals In 1.22: Here’s What You Need To Know.

[liggitt]

yeah, that sounds better to me

[sftim]

• “the said” isn't current for idiomatic UK English; I'd write “those”. Not sure about US English.
• I think “This removed” should be “This removal”
• Webhook isn't an API; the APIs are ValidatingWebhookConfiguration and MutatingWebhookConfiguration. Maybe “webhook configuration”, or list both kinds explicitly.

[sftim]

Proposal: Alpha support for running nodes with swap space configured

[liggitt]

is this a major theme, or does it belong in the features section? graduation to beta or GA would be when I'd expect this to show up in major themes

[sftim]

Ah, good point.

[ehashman]

No worries! I am working on a blog for this in any case :)

[pmmalinov01]

After a short discussion on the release team meeting, I would say we should accept all alpha features as was done in the past release notes Major themes. This topic will be but on the retrospective and we should work to use strict rules on what is Major theme and what goes in it and why is it so. This is currently missing from the release notes documentation.

@ehashman We will want to add this feature to the Major theme and if you want to add more to the test feel free to so by 16.07 end of day.

[neolit123]

Alpha features should definitelly be in release notes to inform users to try them and provide feedback. Without feedback it may be hard to graduate something to beta.

[sftim]

Proposal: The replacement for PodSecurityPolicy (if included in v1.22)

[pmmalinov01]

ping @liggitt on slack if he wants to add more information as he is the author of the KEP and solution.

[liggitt]

I think a small mention of this would be good. I'm happy to collaborate on a paragraph about this.

[sftim]

This is also alpha, but I think it's worth making an exception given circumstances.

[pmmalinov01]

@liggitt I went and update my comment with a bit more details about the alpha feature. I also went and read kubernetes/website#28903 and tried to use the future documentation as a reference. Can you double-check my proposal for the Major themes and add/remove anything. Also, the URL for the documentation will be https://kubernetes.io/docs/concepts/security/namespace-controls-pod-security/, right?

[liggitt]

URL will be https://kubernetes.io/docs/concepts/security/pod-security-admission/

Suggestion:

A new PodSecurity alpha admission plugin has been introduced as a built-in mechanism to apply Pod Security Standards. Intended to replace the deprecated PodSecurityPolicy API and admission plugin, it uses namespace labels to select enforcement levels, and can operate in enforcing mode or surface warnings or audit annotations to preview the effects of a new policy level non-disruptively.

Markdown:

A new [`PodSecurity`](https://kubernetes.io/docs/concepts/security/pod-security-admission/) 
alpha admission plugin has been introduced as a built-in mechanism to apply 
[Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/).
Intended to replace the deprecated `PodSecurityPolicy` API and admission plugin, it uses namespace labels 
to select enforcement levels, and can operate in enforcing mode or surface warnings or audit annotations
to preview the effects of a new policy level non-disruptively.

[apelisse]

Server-side Apply going GA

[pmmalinov01]

@apelisse Thanks for the suggestion, @cici37 will do a Major theme on it. If you wish you can provide a review :)

[apelisse]

Of course! Thanks :-)

[cici37]

Server-side Apply graduates to GA in 1.22.
Server-side Apply is a new object merge algorithm, as well as tracking of field ownership, running on the Kubernetes api-server. It helps users and controllers manage their resources via declarative configurations. It allows them to create and/or modify their objects declaratively, simply by sending their fully specified intent. After being in beta for a couple releases, Server-side Apply graduates to GA in 1.22.

[xiaoxubeii]

Proposal: Support Memory QoS with cgroups v2 kubernetes/enhancements#2571
Suggestion: Memory QoS with cgroups v2 allows administrators to guarantee pod / container memory by using cgroups v2 memory controller. This feature is designed to improve workload and node availability in memory resource. It is first introduced as alpha in Kubernetes 1.22.

@pmmalinov01 @liggitt It would be Major theme.
/cc @bobbypage @mrunalp @ehashman

[pmmalinov01]

Yeaa, this looks great, can you write a short paragraph about it?

[Damans227]

In traditional cgroups v1 implementation in Kubernetes, pod qos class and qos manager can only limit cpu resources, such as cpu_shares. Kubernetes cgroup manager uses memory.limit_in_bytes in cgroups v1 to limit the memory capacity and uses oom_scores to determine the order of killing the container processes when OOM occurs.

This implementation has following disadvantages:

• Guaranteed memory can not be fully reserved, page cache is at risk of being recycled.
• Burstable memory allocation overcommit ( request < limit ) could increase OOM possibility when kernel memory runs out.

KEP-2570 introduces cgroups v2, which brings new capabilities for memory controllers and it would help Kubernetes enhance memory isolation quality.

[ehashman]

This one is also alpha, probably not appropriate for major themes given #1575 (reply in thread) ?

[pmmalinov01]

It should be in the Major themes, #1575 (reply in thread)

[pmmalinov01]

@mrbobbytables Suggested that it would be great to highlight the release cadence change. This is a great idea and Bob can we ask you or anyone to write something about it so we can add it to the Major Themes for the release.

[sftim]

It's likely we'll have a blog article we can link to as well: see kubernetes/website#28912

[ankeesler]

Proposal: client-go credential plugins going GA kubernetes/enhancements#541

Suggestion: client-go credential plugins have been in beta since 1.11. With the release of Kubernetes 1.22, this feature set graduates to stable. The GA feature set includes improved support for plugins that provide interactive login flows. This release also contains a number of bug fixes to the feature set. Aspiring plugin authors can look at sample-exec-plugin as a way to get started.

Related to this topic, the in-tree azure and gcp auth plugins have been deprecated in favor of out-of-tree implementations of these plugins.

[jayunit100]

For 1.22 ~ to grow the developer community, sig-windows released https://github.com/kubernetes-sigs/sig-windows-dev-tools/, which supports multiple CNI providers (antrea, calico), can run on multiple platforms (any vagrant compatible provider, such as hyper-v, virtualbox, vsphere), and provides a sandbox for running bleeding edge windows features from scratch by compiling the windows kubelet and kube proxy and running from sig-release daily builds (https://storage.googleapis.com/kubernetes-release-dev/ci/latest.txt)

[neolit123]

SIG Cluster Lifecycle themes:

In 1.22, SIG Cluster Lifecycle has added a couple of important features to kubeadm:

• A new v1beta3 configuration API. It iterates over v1beta2 by adding some long requested features and deprecating some existing ones. V1beta3 is now the preferred API version, but v1beta2 is not yet deprecated (TBD in a future release).
• A new Alpha feature for running the kubeadm control plane components as non-root users. This is a long requested security measure in kubeadm. To try it you must enable the kubeadm-specific RootlessControlPlane feature gate.

[dashpole]

APIServer tracing going alpha

[pmmalinov01]

Thanks for it. Once it gets merge we should add it.

[pmmalinov01]

the KEP is 647. Sound good will add it

[liggitt]

is this a major theme, or does it belong in the features section? it seems like graduation to beta or GA would be when I'd expect this to show up in major themes

[dashpole]

I'm not sure where it goes. I'm fine with it being a feature

[pmmalinov01]

@dashpole It can be a major theme, Lets get in it. If you have anything prepared for it we can use it will be great.

[verb]

The API used to create Ephemeral Containers changed in 1.22. The Ephemeral Containers feature is alpha and disabled by default, and the new API will not work with clients attempting to use the old API.

• Users who create ephemeral containers using kubectl debug should note that kubectl version 1.22 will attempt to fallback to the old API, but older versions of kubectl will not work with cluster versions of 1.22 or later. Please update kubectl to 1.22 if you wish to use kubectl debug with a mix of cluster versions. (This only applies to debugging using ephemeral containers.)
• Admission plugin authors should note that the AdmissionReview request will now contain the entire v1.Pod rather than just a list of ephemeral containers as before.
• Cluster administrators should note that securityContext may now be set on ephemeral containers and ensure that cluster security policies are applied to these fields prior to enabling the Ephemeral Containers feature.

[pmmalinov01]

I think it can be in the Major Theme, I have seen in past renditions alpha feature to make in the Major Themes. In my mind, it can be used to boost the popularity of the feature to gather more people using it and development.
Something I do think it can change is the bullet points. I think we can make it one paragraph.

[liggitt]

that's fine, just wanted to check on a few of these... I wasn't exactly sure the intent of the major themes section

[verb]

Ah, I think I misunderstood the intent here. These are some details I want to communicate about changes in 1.22, I agree that they don't represent a major theme that hasn't been communicated in a previous release.

@pmmalinov01 is there a place I can move this suggestion for inclusion in the features section? If you still want to include ephemeral containers in major themes I'm happy to rewrite this, but otherwise I'd hold off until beta.

[pmmalinov01]

@verb I would want to include this feature in the major themes, it sounds exciting and helpful. If you have the time to tweak it to not use bullet points it will be great.

[verb]

Looks like I can't edit the original post for whatever reason. Here's an update for ephemeral containers that's better suited for major themes:

The alpha Ephemeral Containers feature revised its API to be more useful in security-sensitive debugging scenarios. Users may now configure a different security context for ephemeral containers than the targeted workload, and admission plugins will receive additional pod context to allow or deny the ephemeral container.

[xing-yang]

From SIG-Storage:

• CSI Windows moves to GA in the 1.22 release. Windows currently does not support privileged containers. To solve this problem, CSIProxy enables CSI node plugins to be deployed as unprivileged pods and use the proxy to perform privileged storage operations on the node.
• CSI Service Account Token moves to GA in 1.22. This feature allows CSI drivers to use pods' bound service account tokens instead of a more privileged identity. It also provides a knob to re-publish volumes so that short-lived volumes can be refreshed.

cc @msau42 @saad-ali @jsafrane

[pmmalinov01]

@xing-yang Thank you for your contribution. WIll add them to the Major themes.