fix(k3s): preserve Kyber disk headroom - #2126
Conversation
|
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
You do not have enough credits to review this pull request. Please purchase more credits to continue. |
📝 WalkthroughWalkthroughK3s kubelet image-pull and garbage-collection settings were updated, and activation now configures ext4 root reserved space to 1% when applicable. Required command paths, activation tests, and Kyber disk-headroom documentation were added. ChangesK3s disk headroom
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant ActivationScript
participant findmnt
participant tune2fs
participant K3sSetup
ActivationScript->>findmnt: Resolve the block device mounted at /
ActivationScript->>tune2fs: Inspect ext4 reserved-block values
ActivationScript->>tune2fs: Set reserved blocks to 1% when needed
ActivationScript->>K3sSetup: Continue k3s service and kubeconfig setup
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request optimizes disk headroom and prevents disk-pressure feedback loops on the k3s host. It reduces maxParallelImagePulls to 2, sets explicit image garbage collection thresholds, and adds a bash function to dynamically configure the root ext4 filesystem's reserved blocks to 1%. The Nix configuration, documentation, and tests are updated accordingly. Feedback was provided to append || true to the tune2fs command substitutions in the activation script to prevent premature abortion under set -e and pipefail if the commands fail.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| block_count="$(run_sudo @tune2fs@ -l "$root_source" 2>/dev/null | @awk@ -F: '/^Block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')" | ||
| # shellcheck disable=SC2016 | ||
| reserved_blocks="$(run_sudo @tune2fs@ -l "$root_source" 2>/dev/null | @awk@ -F: '/^Reserved block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')" |
There was a problem hiding this comment.
Since set -e and pipefail are enabled, if run_sudo @tune2fs@ fails (for example, if sudo requires a password in a non-interactive shell, or if the block device cannot be opened), the command substitution will return a non-zero exit status and prematurely abort the entire activation script.
To prevent this and allow the subsequent if [ -z "$block_count" ] check to handle the failure gracefully, append || true to the pipelines.
| block_count="$(run_sudo @tune2fs@ -l "$root_source" 2>/dev/null | @awk@ -F: '/^Block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')" | |
| # shellcheck disable=SC2016 | |
| reserved_blocks="$(run_sudo @tune2fs@ -l "$root_source" 2>/dev/null | @awk@ -F: '/^Reserved block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')" | |
| block_count="$(run_sudo @tune2fs@ -l "$root_source" 2>/dev/null | @awk@ -F: '/^Block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }' || true)" | |
| # shellcheck disable=SC2016 | |
| reserved_blocks="$(run_sudo @tune2fs@ -l "$root_source" 2>/dev/null | @awk@ -F: '/^Reserved block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }' || true)" |
References
- When 'set -e' is enabled in Bash scripts, ensure that individual command failures within a loop do not prematurely abort the entire script. Handle potential failures gracefully, for example by appending '|| true'.
![indent-zero[bot]](https://avatars.githubusercontent.com/in/1426746?s=60&v=4){kind=small}
| # shellcheck disable=SC2016 | ||
| block_count="$(run_sudo @tune2fs@ -l "$root_source" 2>/dev/null | @awk@ -F: '/^Block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')" | ||
| # shellcheck disable=SC2016 | ||
| reserved_blocks="$(run_sudo @tune2fs@ -l "$root_source" 2>/dev/null | @awk@ -F: '/^Reserved block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')" |
There was a problem hiding this comment.
Nit — duplicated tune2fs -l: Both block_count and reserved_blocks are extracted with independent sudo tune2fs -l "$root_source" invocations, each piped to awk. On every home-manager activation this runs tune2fs -l twice. Consider a single call whose output is parsed once, e.g.:
read -r block_count reserved_blocks < <(
run_sudo @tune2fs@ -l "$root_source" 2>/dev/null \
| @awk@ -F: '
/^Block count:/ { gsub(/[[:space:]]/, "", $2); bc=$2 }
/^Reserved block count:/ { gsub(/[[:space:]]/, "", $2); rc=$2 }
END { print bc, rc }'
)Purely cosmetic — behavior is identical.
| # by the Kyber activation script, keeping ordinary usage below the low watermark | ||
| # while kubelet remains the sole owner of image and container garbage collection. | ||
| imageGCHighThresholdPercent: 85 | ||
| imageGCLowThresholdPercent: 80 |
There was a problem hiding this comment.
Nit — explicit defaults: 85 / 80 are already the kubelet defaults per the KubeletConfiguration reference, so these two lines don't change runtime GC behavior; they only pin the values in code. The comment above ("Make the single-node disk contract explicit") makes the intent clear, but it may be worth adding # (kubelet defaults, restated for clarity) so future readers don't infer this was the fix for the July 2026 incident.
658debe to
8496917
Compare
Mesa DescriptionTL;DRPrevents K3s disk-pressure feedback loops on Kyber by reducing the ext4 root filesystem reserve to 1% and configuring explicit kubelet image garbage collection thresholds and parallel pull limits. What changed?
Root causeDuring overlapping image rollouts, root usage crossed kubelet's 85% image-GC threshold. Ubuntu's default 5% ext4 reserve hid roughly 46 GiB on the 916 GiB root filesystem. Kubelet then tried to reclaim tens of GiB from a much smaller logical image cache while containerd and Kine were I/O-bound, producing CRI timeouts and stale container-name/cgroup state. Validation
Description generated by Mesa. Update settings |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
3 issues found and verified against the latest diff
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="spec/k3s_service_activate_spec.sh">
<violation number="1" location="spec/k3s_service_activate_spec.sh:80">
P2: The grep pattern `target_reserved_percent=1` is a substring match — it also matches `target_reserved_percent=10`, `target_reserved_percent=12`, etc. If the reserved percent is ever changed to a multi-digit value, this test will silently pass even after the script loses the intended value. Adding `-w` isolates the value boundary correctly.</violation>
</file>
<file name="named-hosts/kyber/README.md">
<violation number="1" location="named-hosts/kyber/README.md:72">
P3: The diagnosis checklist says to check I/O pressure, but the commands below do not inspect it, so an operator following this recovery procedure can miss the I/O-bound condition described above. Adding an explicit PSI or `iostat` check would make the checklist complete.</violation>
</file>
<file name="home-manager/services/k3s/activate.sh">
<violation number="1" location="home-manager/services/k3s/activate.sh:74">
P2: If the `tune2fs -l` pipeline fails (e.g., sudo permission issue or transient device error), `set -euo pipefail` exits the subshell, `block_count`/`reserved_blocks` stay empty, and the script exits at line ~65 before the `[ -z "$block_count" ]` guard on line ~74 is reached. The `2>/dev/null` on the tune2fs call hides the original error, making the failure hard to diagnose. The same risk applies to the final `run_sudo @tune2fs@ -m` command on the last changed line — if that fails, the function returns non-zero and `set -e` aborts the entire activation script, skipping k3s service installation.
The ext4 reserve adjustment is a best-effort optimization; it shouldn't block the service file installation and k3s enable/start that follow. Consider guarding the function call with `|| true` so a disk-configuration failure doesn't prevent k3s from being installed. As a secondary improvement, removing `2>/dev/null` (or keeping it but adding `|| true` after each subshell) lets the existing `[ -z "$block_count" ]` warning actually fire when tune2fs fails.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| End | ||
|
|
||
| It 'keeps one percent of the ext4 root volume reserved' | ||
| When run bash -c "grep 'target_reserved_percent=1' '$SCRIPT'" |
There was a problem hiding this comment.
P2: The grep pattern target_reserved_percent=1 is a substring match — it also matches target_reserved_percent=10, target_reserved_percent=12, etc. If the reserved percent is ever changed to a multi-digit value, this test will silently pass even after the script loses the intended value. Adding -w isolates the value boundary correctly.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At spec/k3s_service_activate_spec.sh, line 80:
<comment>The grep pattern `target_reserved_percent=1` is a substring match — it also matches `target_reserved_percent=10`, `target_reserved_percent=12`, etc. If the reserved percent is ever changed to a multi-digit value, this test will silently pass even after the script loses the intended value. Adding `-w` isolates the value boundary correctly.</comment>
<file context>
@@ -75,5 +75,15 @@ It 'preserves dry-run command handling'
End
+
+It 'keeps one percent of the ext4 root volume reserved'
+When run bash -c "grep 'target_reserved_percent=1' '$SCRIPT'"
+The output should include 'target_reserved_percent=1'
+End
</file context>
| When run bash -c "grep 'target_reserved_percent=1' '$SCRIPT'" | |
| When run bash -c "grep -w 'target_reserved_percent=1' '$SCRIPT'" |
| echo "Configured $root_source ext4 reserved blocks to ${target_reserved_percent}%" | ||
| } | ||
|
|
||
| configure_root_ext4_reserve |
There was a problem hiding this comment.
P2: If the tune2fs -l pipeline fails (e.g., sudo permission issue or transient device error), set -euo pipefail exits the subshell, block_count/reserved_blocks stay empty, and the script exits at line ~65 before the [ -z "$block_count" ] guard on line ~74 is reached. The 2>/dev/null on the tune2fs call hides the original error, making the failure hard to diagnose. The same risk applies to the final run_sudo @tune2fs@ -m command on the last changed line — if that fails, the function returns non-zero and set -e aborts the entire activation script, skipping k3s service installation.
The ext4 reserve adjustment is a best-effort optimization; it shouldn't block the service file installation and k3s enable/start that follow. Consider guarding the function call with || true so a disk-configuration failure doesn't prevent k3s from being installed. As a secondary improvement, removing 2>/dev/null (or keeping it but adding || true after each subshell) lets the existing [ -z "$block_count" ] warning actually fire when tune2fs fails.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At home-manager/services/k3s/activate.sh, line 74:
<comment>If the `tune2fs -l` pipeline fails (e.g., sudo permission issue or transient device error), `set -euo pipefail` exits the subshell, `block_count`/`reserved_blocks` stay empty, and the script exits at line ~65 before the `[ -z "$block_count" ]` guard on line ~74 is reached. The `2>/dev/null` on the tune2fs call hides the original error, making the failure hard to diagnose. The same risk applies to the final `run_sudo @tune2fs@ -m` command on the last changed line — if that fails, the function returns non-zero and `set -e` aborts the entire activation script, skipping k3s service installation.
The ext4 reserve adjustment is a best-effort optimization; it shouldn't block the service file installation and k3s enable/start that follow. Consider guarding the function call with `|| true` so a disk-configuration failure doesn't prevent k3s from being installed. As a secondary improvement, removing `2>/dev/null` (or keeping it but adding `|| true` after each subshell) lets the existing `[ -z "$block_count" ]` warning actually fire when tune2fs fails.</comment>
<file context>
@@ -37,6 +37,42 @@ require_sudo() {
+ echo "Configured $root_source ext4 reserved blocks to ${target_reserved_percent}%"
+}
+
+configure_root_ext4_reserve
+
if [ -f "$SERVICE_FILE" ] && ! @diff@ -q "$SERVICE_FILE" "$SYSTEM_SERVICE" >/dev/null 2>&1; then
</file context>
| chat turns. Reducing the ext4 reserve, limiting pull parallelism, and preserving | ||
| free space prevent that loop. | ||
|
|
||
| For diagnosis, check filesystem headroom, I/O pressure, kubelet GC messages, |
There was a problem hiding this comment.
P3: The diagnosis checklist says to check I/O pressure, but the commands below do not inspect it, so an operator following this recovery procedure can miss the I/O-bound condition described above. Adding an explicit PSI or iostat check would make the checklist complete.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At named-hosts/kyber/README.md, line 72:
<comment>The diagnosis checklist says to check I/O pressure, but the commands below do not inspect it, so an operator following this recovery procedure can miss the I/O-bound condition described above. Adding an explicit PSI or `iostat` check would make the checklist complete.</comment>
<file context>
@@ -49,6 +49,42 @@ Once Tailscale is set up:
+chat turns. Reducing the ext4 reserve, limiting pull parallelism, and preserving
+free space prevent that loop.
+
+For diagnosis, check filesystem headroom, I/O pressure, kubelet GC messages,
+and CRI health before restarting services:
+
</file context>
1 participant
Summary
Root cause
During overlapping image rollouts, root usage crossed kubelet's 85% image-GC threshold. Ubuntu's default 5% ext4 reserve hid roughly 46 GiB on the 916 GiB root filesystem. Kubelet then tried to reclaim tens of GiB from a much smaller logical image cache while containerd and Kine were I/O-bound, producing CRI timeouts and stale container-name/cgroup state.
Validation
Summary by cubic
Preserves Kyber’s root disk headroom to prevent DiskPressure and CRI timeouts during overlapping image rollouts. Sets a 1% ext4 reserve on the mounted root device, caps
kubeletparallel pulls, and makes image GC thresholds explicit while keepingkubeletas the sole GC owner.findmnt/tune2fs; no-op if non-ext4.kubelet.conf:maxParallelImagePulls4 → 2;imageGCHighThresholdPercent: 85,imageGCLowThresholdPercent: 80.crictlcleanup timer; GC stays underkubelet.k3s/containerd/Kine.default.nixwiresawk,findmnt,tune2fs,diff.Written for commit 8496917. Summary will update on new commits.