Kubernetes 1.26 Major Themes

[ramrodo]

This is a discussion to discuss Major Themes for the v1.26 release.

Here is the discussion from the last release cycle.

(please add links as references for the Major Theme posted)

[ramrodo]

cc @kubernetes/release-team-leads
cc @AnaMMedina21 @harshanarayana @laxmikantbpandhare @sayantani11

[saschagrunert]

The signing of Kubernetes release artifacts graduates to beta. This means that we now sign binary artifacts side by side to the container images. More information about how to verify the binary signatures can be found in the official documentation.

[logicalhan]

SIG Instrumentation:

• Metrics framework extensions to Alpha: Documentation can now be auto-generated for every metric in the Kubernetes code-base.

• Api-server tracing to Beta: Traces now include much more detail, and have parity with log-based "tracing".

• Component SLIs to Alpha: If you enable the ComponentSLIs feature flag, you can enable an additional metrics endpoint which allows you to calculate SLOs from raw healthcheck data converted into metric format.

• Feature metrics are now available for each Kubernetes component: Check the component's metrics endpoint for the kubernetes_feature_enabled metric to see the inventory of enabled Kubernetes features for the given component.

• Klog flag deprecated is completed: klog itself continues to support them, but Kubernetes components don't. This makes the command line interfaces simpler and reduces the attack surface because less functionality is exposed.

[klueska]

Dynamic Resource Allocation is a new alpha feature in Kubernetes that puts resource scheduling in the hands of 3rd-party developers. It moves away from the limited "countable" interface for requesting access to resources (e.g. nvidia.com/gpu: 2), providing an API more akin to that of persistent volumes. Under the hood it uses CDI to do its device injection. This feature is blocked by the DynamicResourceAllocation feature gate.

/sig node
/cc @pohly

[cici37]

CEL in Admission Control to Alpha: Introduced v1alpha1 API for validating admission policies, enabling extensible admission control via CEL expressions. To use, enable the ValidatingAdmissionPolicy feature gate and the admissionregistration.k8s.io/v1alpha1 API via --runtime-config.

[SergeyKanzhelev]

CRI v1alpha2 was removed from k8s 1.26: kubernetes/kubernetes#110618

For some time k8s was able to use both - v1 and v1alpha2 versions of CRI. Now as major container runtime has switched to v1, we are dropping support for v1alpha2.

Containerd 1.5, which will be EOL January 28, 2023 (containerd releases), as a result is not supported by Kubernetes 1.26. Users needs to upgrade to Containerd 1.6 before upgrading to 1.26.

There are also some CRI implementations like in stargz snapshotter: containerd/stargz-snapshotter#1005 which will also be affected. So if you are using some third party tool that may install CRI API proxy of any sort - contact the vendor of that tool

[xing-yang]

From SIG-Storage:

GA Features:

• The Delegate FSGroup to CSI Driver feature moved to GA in v1.26. This feature allows Kubernetes to supply the pod's FSGroup to the CSI driver when a volume is mounted so that the driver can utilize mount options to control volume permissions.
• CSI Migration is an on-going effort that SIG Storage has been working on for a few releases now. The goal is to move in-tree volume plugins to out-of-tree CSI drivers and eventually remove the in-tree volume plugins. In v1.26, CSI Migration for both Azure File and vSphere moved to GA.

Driver Removals

• The in-tree GlusterFS driver was deprecated in v1.25 and it is removed in v1.26.
• CSI Migration for OpenStack Cinder moved to GA in v1.24. The in-tree Cinder driver and cloud provider are removed in v1.26.

cc @saad-ali @msau42 @jsafrane

[Huang-Wei]

SIG Scheduling:

• PodSchedulingReadiness to Alpha: PodSchedulingReadiness feature introduces a
  .spec.schedulingGates field to Pod's API, to indicate whether the Pod is allowed to be
  scheduled or not. External controllers can use the gates to hold a Pod from scheduling
  based on their business needs.
• NodeInclusionPolicyInPodTopologySpread to Beta: By specifying a nodeInclusionPolicy in
  topologySpreadConstraints, you can control whether to take taints/tolerations into consideration
  when calculating pod topology spread skew.
• MatchLabelKeysInPodTopologySpread to Beta: By specifying matchLabelKeys in topologySpreadConstraints,
  you have fine-grained control over the group of Pods where the spreading skew is calculated. This
  feature provides a native way to maintain pod balance after a rolling upgrade in Deployments that
  use PodTopologySpread.

cc @ahg-g @alculquicondor

[alculquicondor]

My suggestions to explain better the usefulness of the features:

to mark a Pod's schedule readiness. Integrators can
mutate this field to signal to scheduler when a Pod is ready for scheduling.

describing whether the Pod is allowed to be scheduled. External controllers can use the gates to hold a Pod from scheduling based on their own policies.

By specifying matchLabelKeys in topologySpreadConstraints,
you have fine-grained control over the group of Pods where the spreading skew is calculated. This
feature provides a native way to maintain pod balance after a rolling upgrade in Deployments that
use PodTopologySpread.

By specifying matchLabelKeys in topologySpreadConstraints, the values of the labels are deduced from the pod being scheduled, simplifying your pod specs. You can use it in a Deployment to match only the pods of the new revision during a rolling upgrade, without the need to know its value.

[fsmunoz]

Hello,

This are the collected major themes that will be used for the Release blog article draft, collected from this discussion. That draft will be shared to all SIG leads and Release Team for additional validation of the content, but in terms of Major Themes will be structured around what I've collected here (the order of it is TBD, and the SIG indication wil be removed).

(@ramrodo @krol3 , this fulfills the "Major themes identification" steps we have, although obviously discussion is still open).

Major Themes for Kubernetes v1.26

Signing Kubernetes release artefacts graduates to Stable [SIG Release]

Metrics framework extension graduates to Alpha [SIG Instrumentation]

API server tracing graduates to Beta [SIG Instrumentation]

Components SLIs graduates to Alpha [SIG Instrumentation]

Feature metrics are now availabe for each Kubernetes component [SIG Instrumentation]

Klog flag deprecation is completed [SIG Instrumentation]

Dynamic Resource Allocation graduates to Alpha [SIG Node]

CEL in Admission Control graduates to Alpha [SIG API Machinery]

CRI v1alpha2 removed [SIG Security]

Delegate FSGroup to CSI Driver graduated to Stable [SIG Storage]

CSI migration for Azure File and vSphere graduated to Stable [SIG Storage]

In-tree GlusterFS driver removal [SIG Storage]

In-tree Cinder driver removal [SIG Storage]

PodSchedulingReadiness graduates to Alpha [SIG Scheduling]

NodeInclusionPolicyInPodTopologySpread graduates to Beta [SIG Scheduling]

MatchLabelKeysInPodTopologySpread graduates to Beta [SIG Scheduling]

[fsmunoz]

@marosset wouldn't it be appropriate to add the Support for Windows privileged containers graduation to Stable? We have dropped it from the mid-cycle but here it would make sense IMO.

[fsmunoz]

@bridgetkromhout @thockin , doesn't the graduation of the support for mixed protocols fit in here? We don't have anything from SIG Network, and I already have this one described since it looks to be relevant.