Graduate swap to Beta 1

Signed-off-by: Itamar Holder <[email protected]>

Author: harche

keps/sig-node/2400-node-swap/README.md

@@ -8,6 +8,11 @@
   - [Goals](#goals)
   - [Non-Goals](#non-goals)
 - [Proposal](#proposal)
+  - [Enable Swap Support only for Burstable QoS Pods](#enable-swap-support-only-for-burstable-qos-pods)
+    - [Set Aside Swap for System Critical Daemon](#set-aside-swap-for-system-critical-daemon)
+    - [Use Swap Throttling](#use-swap-throttling)
+  - [Steps to Calculate Swap Limit](#steps-to-calculate-swap-limit)
+    - [Example](#example)
   - [User Stories](#user-stories)
     - [Improved Node Stability](#improved-node-stability)
     - [Long-running applications that swap out startup memory](#long-running-applications-that-swap-out-startup-memory)
@@ -17,6 +22,7 @@
     - [Virtualization management overhead](#virtualization-management-overhead)
   - [Notes/Constraints/Caveats (Optional)](#notesconstraintscaveats-optional)
   - [Risks and Mitigations](#risks-and-mitigations)
+    - [Security risk](#security-risk)
 - [Design Details](#design-details)
   - [Enabling swap as an end user](#enabling-swap-as-an-end-user)
   - [API Changes](#api-changes)
@@ -30,7 +36,8 @@
   - [Graduation Criteria](#graduation-criteria)
     - [Alpha](#alpha)
     - [Alpha2](#alpha2)
-    - [Beta](#beta)
+    - [Beta 1](#beta-1)
+    - [Beta 2](#beta-2)
     - [GA](#ga)
   - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
   - [Version Skew Strategy](#version-skew-strategy)
@@ -166,6 +173,83 @@ administrators can configure the kubelet such that:
 
 This proposal enables scenarios 1 and 2 above, but not 3.
 
+### Enable Swap Support only for Burstable QoS Pods
+Before enabling swap support through the pod API, it is crucial to build confidence in this feature by carefully assessing its impact on workloads and Kubernetes. As an initial step, we propose enabling swap support for Burstable QoS Pods by automatically calculating the appropriate swap values, rather than allowing users to input these values manually. 
+
+Swap access is granted only for pods of Burstable QoS. Guaranteed QoS pods are usually higher-priority pods, therefore we want to avoid swap's performance penalty for them. Best-Effort pods, on the contrary, are low-priority pods that are the first to be killed during node pressures. In addition, they're unpredictable, therefore it's hard to assess how much swap memory is a reasonable amount to allocate for them. 
+
+By doing so, we can ensure a thorough understanding of the feature's performance and stability before considering the manual input of swap values in a subsequent beta release. This cautious approach will ensure the efficient allocation of resources and the smooth integration of swap support into Kubernetes.
+
+Allocate the swap limit equal to the requested memory for each container and adjust the proportion of swap based on the total swap memory available.
+
+#### Set Aside Swap for System Critical Daemon
+
+System critical daemons (such as Kubelet) are essential for node health. Usually, an appropriate portion of system resources (e.g., memory, CPU) is reserved as system reserved. However, swap doesn't inherently support reserving a portion out of the total available. For instance, in the case of memory, we set `memory.min` on the node-level cgroup to ensure an adequate amount of memory is set aside, away from the pods, and for system critical daemons. But there is no equivalent for swap; i.e., no `memory.swap.min` is supported in the kernel. 
+
+Since this proposal advocates enabling swap only for the Burstable QoS pods, this can be done by setting `memory.swap.max` on the cgroups used by the Burstable QoS pods. The value of this `memory.swap.max` can be calculated by:
+
+memory.swap.max = total swap memory available on the system - system reserve (memory)
+
+This is the total amount of swap available for all the Burstable QoS pods; let's call it `TotalPodsSwapAvailable`. This will ensure that the system critical daemons will have access to the swap at least equal to the system reserved memory. This will indirectly act as having support for swap in system reserved.
+
+#### Use Swap Throttling
+
+`memory.swap.high` in the cgroup v2 allows you to enable throttling for swap allocation. If a cgroup’s swap usage exceeds `memory.swap.high`, all its further allocations will be throttled.
+
+Throttling will be enabled at individual Burstable QoS pod as well as at Burstable QoS cgroup level.
+
+1. At the Burstable QoS cgroup level, it can be calculated with: memory.swap.high = `TotalPodsSwapAvailable` * swap throttling factor
+2. At individual Burstable QoS pod level, it can be calculated with: memory.swap.high = memory.swap.max * swap throttling factor
+
+The default value of the memory throttling factor is set to 0.9.
+### Steps to Calculate Swap Limit
+
+1. **Calculate the total memory requests of the pod:**
+   - Sum up the memory requests of all containers in the pod. Let's call this value `TotalMemory`.
+
+2. **Determine the swap proportion for each container:**
+   - For each container, divide its memory request by the `TotalMemory`. The result will be the proportion of requested memory for that container within the pod. Let's call this value `RequestedMemoryProportion`.
+
+3. **Calculate the total swap memory available:**
+   - `TotalPodsSwapAvailable` is the total amount of memory available for the pods. Divide the available swap memory by the total physical memory. Let's call this value `SwapMemoryProportion`.
+
+4. **Calculate the swap limit for each container:**
+   - Multiply the `RequestedMemoryProportion` of a container by its memory request and then multiply the result by the `SwapMemoryProportion`. The result will be the adjusted swap limit for that specific container within the pod.
+
+5. **Calculate the swap throttling threshold for each container:** 
+   - Multiple the swap throttling threashold (0.9) with the swap limit for each container. 
+
+
+#### Example
+Suppose we have a Burstable QoS pod with two containers:
+
+- Container A: Memory request 20 GB
+- Container B: Memory request 10 GB
+
+Let's assume the total physical memory is 40 GB and the total swap memory available is also 40 GB. Also assume that the system reserved memory is configured at 2GB, 
+
+Step 1: Calculate `TotalPodsSwapAvailable`, which denotes total available swap memory: 40 GB - 2 GB = 38 GB
+
+Step 2: Calculate the total memory requests of the pod: 20 GB + 10 GB = 30 GB
+
+Step 2: Determine the requested memory proportion for each container:
+- Container A: (20 GB) / (30 GB) = 2/3
+- Container B: (10 GB) / (30 GB) = 1/3
+
+Step 3: Calculate the total swap memory available: Since the total swap memory (38 GB) and  physical memory (40 GB), the `SwapMemoryProportion` will be 38 GB / 40 GB = 0.95
+
+Step 4: Calculate the swap limit for each container:
+- Container A: (2/3) * 20 GB * 0.95 ≈ 12.66 GB
+- Container B: (1/3) * 10 GB * 0.95 ≈ 3.16 GB
+
+Step 5: Calculate the swap throttling threshold for each container:
+- Container A: 12.66 * 0.9 = 11.39 GB
+- Container B: 3.16 * 0.9 = 2.844 GB
+
+In this example, Container A would have a swap limit of 12.66 GB, and Container B would have a swap limit of 3.16 GB.
+
+This approach allocates swap limits based on each container's memory request and adjusts the proportion based on the total swap memory available in the system. It ensures that each container gets a fair share of the swap space and helps maintain resource allocation efficiency.
+
 ### User Stories
 
 #### Improved Node Stability
@@ -300,6 +384,12 @@ and/or workloads in a number of different scenarios.
 Since swap provisioning is out of scope of this proposal, this enhancement
 poses low risk to Kubernetes clusters that will not enable swap.
 
+#### Security risk
+
+Enabling swap on a system without encryption poses a security risk, as critical information, such as Kubernetes secrets, may be swapped out to the disk. If an unauthorized individual gains access to the disk, they could potentially obtain these secrets. To mitigate this risk, it is recommended to use encrypted swap. However, handling encrypted swap is not within the scope of kubelet; rather, it is a general OS configuration concern and should be addressed at that level. Nevertheless, it is essential to provide documentation that warns users of this potential issue, ensuring they are aware of the potential security implications and can take appropriate steps to safeguard their system.
+
+Additionally end user may decide to disable swap completely for a Pod in beta 1 by making it guaranteed. This way, there will be no swap enabled for the corresponding containers and there will be no information exposure risks.
+
 ## Design Details
 
 We summarize the implementation plan as following:
@@ -487,14 +577,14 @@ Test grid tabs enabled:
 
 No new e2e tests introduced.
 
-For alpha2 [Current stage]:
+For alpha2:
 
 - Add e2e tests that exercise all available swap configurations via the CRI.
 - Verify MemoryPressure behavior with swap enabled and document any changes
   for configuring eviction.
 - Verify new system-reserved settings for swap memory.
 
-For beta [Future]:
+For beta 1:
 
 - Add e2e tests that verify pod-level control of swap utilization.
 - Add e2e tests that verify swap performance with pods using a tmpfs.
@@ -536,20 +626,25 @@ Here are specific improvements to be made:
   swap limit for workloads.
 - Investigate eviction behavior with swap enabled.
 
-
-#### Beta
-
-- Add support for controlling swap consumption at the pod level [via cgroups].
-- Handle usage of swap during container restart boundaries for writes to tmpfs
-  (which may require pod cgroup change beyond what container runtime will do at
-  container cgroup boundary).
+#### Beta 1
+- Enable Swap Support using Burstable QoS Pods only. 
+- Enable Swap Support for Cgroup v2 Only.
+- Enable Swap throttling support
 - Add swap memory to the Kubelet stats api.
 - Determine a set of metrics for node QoS in order to evaluate the performance
   of nodes with and without swap enabled.
-- Better understand relationship of swap with memory QoS in cgroup v2
-  (particularly `memory.high` usage).
-- Collect feedback from test user cases.
+- Make sure node e2e jobs that use swap are healthy
 - Improve coverage for appropriate scenarios in testgrid.
+- Remove support for setting unlimited amount of swap (including [swapBehavior](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-MemorySwapConfiguration) flag of the Kubelet) as any workload with aggressive memory allocation can bring down a node with having no limits on swap usage.
+
+#### Beta 2
+- Add support for controlling swap consumption at the container level [via cgroups] in Pod API in [container resources](https://github.com/kubernetes/kubernetes/blob/94a15929cf13354fdf3747cb266d511154f8c97b/staging/src/k8s.io/api/core/v1/types.go#L2443). More specifically add a new [ResourceName](https://github.com/kubernetes/kubernetes/blob/94a15929cf13354fdf3747cb266d511154f8c97b/staging/src/k8s.io/api/core/v1/types.go#L5522) `swap`. This will make sure we stay consistent with other resources like `cpu`, `memory` etc.
+- Publish a Kubernetes doc page encouring user to use encrypted swap if they wish to enable this feature.
+- Handle usage of swap during container restart boundaries for writes to tmpfs
+  (which may require pod cgroup change beyond what container runtime will do at
+  container cgroup boundary).
+- Deprecate the Swap Support using Burstable QoS Pods only introduced in Beta 1. 
+
 
 [via cgroups]: #restrict-swap-usage-at-the-cgroup-level
 
@@ -559,6 +654,8 @@ _(Tentative.)_
 
 - Test a wide variety of scenarios that may be affected by swap support.
 - Remove feature flag.
+- Remove the Swap Support using Burstable QoS Pods only deprecated in Beta 2. 
+
 
 ### Upgrade / Downgrade Strategy
 

keps/sig-node/2400-node-swap/kep.yaml

@@ -4,6 +4,8 @@ authors:
   - "@ehashman"
   - "@ike-ma"
   - "@SergeyKanzhelev"
+  - "@harche"
+  - "@iholder101"
 owning-sig: sig-node
 participating-sigs:
   - sig-node
@@ -18,12 +20,12 @@ approvers:
   - "@dchen1107"
 
 # The target maturity stage in the current dev cycle for this KEP.
-stage: alpha
+stage: beta
 
 # The most recent milestone for which work toward delivery of this KEP has been
 # done. This can be the current (upcoming) milestone, if it is being actively
 # worked on.
-latest-milestone: "v1.27"
+latest-milestone: "v1.28"
 
 # The milestone at which this feature was, or is targeted to be, at each stage.
 milestone: