Skip to content

fix(k3s): keep reserve setup best effort - #2127

Merged
shunkakinoki merged 1 commit into
mainfrom
codex/kyber-k3s-reserve-hardening
Jul 19, 2026
Merged

fix(k3s): keep reserve setup best effort#2127
shunkakinoki merged 1 commit into
mainfrom
codex/kyber-k3s-reserve-hardening

Conversation

@shunkakinoki

@shunkakinoki shunkakinoki commented Jul 19, 2026

Copy link
Copy Markdown
Owner

Summary

  • keep Kyber ext4 reserve inspection and mutation best-effort during activation
  • inspect filesystem metadata once so block counts stay internally consistent
  • make the kubelet default-threshold policy and I/O-pressure diagnosis explicit
  • tighten the reserve-policy regression assertion

Validation

  • shellcheck home-manager/services/k3s/activate.sh config/k3s/activate.sh
  • shellspec spec/k3s_service_activate_spec.sh spec/activate_k3s_spec.sh (31 examples, 0 failures)
  • git diff --check
  • make nix-format-check

Follow-up to #2126 for review findings that arrived as it auto-merged.


Summary by cubic

Make Kyber k3s activation keep the ext4 root reserve setup best-effort and consistent, and pin kubelet image GC thresholds. Improves reliability during activation and clarifies disk/IO pressure behavior.

  • Bug Fixes
    • Activation: read tune2fs -l once, parse a single snapshot, warn and skip on failures, set a 1% ext4 reserve when possible, and return success when sudo is available.
    • Kubelet: pin imageGCHighThresholdPercent: 85 and imageGCLowThresholdPercent: 80, documented in kubelet.conf to make the host headroom contract explicit.
    • Docs: add /proc/pressure/io to troubleshooting steps.
    • Tests: add a success-path check for require_sudo and tighten the reserve-policy assertion to match the exact target_reserved_percent=1 line.

Written for commit 83bfbfb. Summary will update on new commits.

Review in cubic

@indent-zero

indent-zero Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor
PR Summary

Makes the Kyber k3s ext4 reserve tuning strictly best-effort so a tune2fs failure no longer aborts the rest of home-manager activation (systemd unit install, kubeconfig sync). Also caches tune2fs -l output into a single invocation (now read via bash herestrings), tightens the spec regex that pins target_reserved_percent=1, adds explicit-return hardening to require_sudo with a matching spec assertion, adds a diagnostic (cat /proc/pressure/io) to the Kyber recovery runbook, and annotates kubelet.conf to note the pinned imageGC thresholds are the upstream defaults.

  • home-manager/services/k3s/activate.sh: wraps tune2fs -l and tune2fs -m in if ! ...; then warn; return 0 guards; caches the listing in a new filesystem_info local; extracts Block count and Reserved block count with @awk@ ... <<<"$filesystem_info" herestrings instead of printf | awk pipelines; adds an explicit return 0 to require_sudo.
  • spec/k3s_service_activate_spec.sh: replaces the loose grep 'target_reserved_percent=1' with grep -xF ' local target_reserved_percent=1'; adds a new spec that extracts the require_sudo body via sed and asserts the explicit return 0 line is present.
  • config/k3s/kubelet.conf: adds a comment clarifying that the pinned imageGCHighThresholdPercent/imageGCLowThresholdPercent values are kubelet's defaults, kept for visibility.
  • named-hosts/kyber/README.md: adds cat /proc/pressure/io to the disk-pressure recovery diagnostic snippet.

Issues

No issues found.

CI Checks

Waiting for CI checks...

@cursor

cursor Bot commented Jul 19, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@mesa-dot-dev

mesa-dot-dev Bot commented Jul 19, 2026

Copy link
Copy Markdown

You do not have enough credits to review this pull request. Please purchase more credits to continue.

@coderabbitai

coderabbitai Bot commented Jul 19, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@shunkakinoki, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 44 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 70e3bbea-8a82-4f5a-8a57-508ee5c12e06

📥 Commits

Reviewing files that changed from the base of the PR and between 411b671 and 83bfbfb.

📒 Files selected for processing (4)
  • config/k3s/kubelet.conf
  • home-manager/services/k3s/activate.sh
  • named-hosts/kyber/README.md
  • spec/k3s_service_activate_spec.sh
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/kyber-k3s-reserve-hardening

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@shunkakinoki
shunkakinoki enabled auto-merge (squash) July 19, 2026 07:33

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves the robustness of the activate.sh script by caching the output of tune2fs to avoid redundant executions, adding error handling for filesystem configuration, and updating tests and documentation. The review feedback highlights a critical bug in the require_sudo helper function, which lacks an explicit return 0 and causes the script to skip filesystem inspection when sudo is available. Additionally, it is recommended to use Bash here-strings (<<<) instead of piping printf to awk to avoid spawning unnecessary subshells.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

fi

require_sudo || return 0
if ! filesystem_info="$(run_sudo @tune2fs@ -l "$root_source")"; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

There is a critical bug in the require_sudo function (defined on lines 33-38) that prevents this entire block (and other sudo-dependent parts of the script) from executing when sudo is available.

In Bash, if a function does not end with an explicit return statement, its exit status is that of the last command executed. In require_sudo:

require_sudo() {
  if [ "${#sudo_cmd[@]}" -eq 0 ]; then
    echo "Warning: sudo not found, skipping k3s system setup" >&2
    return 1
  fi
}

When sudo is available, the if condition [ "${#sudo_cmd[@]}" -eq 0 ] evaluates to false (exit status 1), the body is skipped, and the function exits. Since the [ test was the last command executed, the function returns 1.

As a result, require_sudo || return 0 on line 55 will always evaluate to true and return early, skipping the filesystem inspection entirely. Similarly, require_sudo || exit 0 on lines 84 and 92 will cause the script to exit prematurely.

To fix this, please add an explicit return 0 at the end of require_sudo.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 83bfbfb with an explicit success return and a focused regression assertion; the native Kyber activation-package build also succeeds.

Comment thread home-manager/services/k3s/activate.sh Outdated
fi
# shellcheck disable=SC2016
block_count="$(run_sudo @tune2fs@ -l "$root_source" 2>/dev/null | @awk@ -F: '/^Block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')"
block_count="$(printf '%s\n' "$filesystem_info" | @awk@ -F: '/^Block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Since this script is executed with Bash (#!/usr/bin/env bash), you can use a here-string (<<<) instead of piping printf output to awk. This is more efficient as it avoids spawning an extra subshell/process for the pipeline.

Suggested change
block_count="$(printf '%s\n' "$filesystem_info" | @awk@ -F: '/^Block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')"
block_count="$(@awk@ -F: '/^Block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }' <<< "$filesystem_info")"

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated both parsers to Bash here-strings in 83bfbfb.

Comment thread home-manager/services/k3s/activate.sh Outdated
block_count="$(printf '%s\n' "$filesystem_info" | @awk@ -F: '/^Block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')"
# shellcheck disable=SC2016
reserved_blocks="$(run_sudo @tune2fs@ -l "$root_source" 2>/dev/null | @awk@ -F: '/^Reserved block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')"
reserved_blocks="$(printf '%s\n' "$filesystem_info" | @awk@ -F: '/^Reserved block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Similarly, you can use a here-string (<<<) here to avoid the pipeline and printf subshell.

Suggested change
reserved_blocks="$(printf '%s\n' "$filesystem_info" | @awk@ -F: '/^Reserved block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }')"
reserved_blocks="$(@awk@ -F: '/^Reserved block count:/ { gsub(/[[:space:]]/, "", $2); print $2 }' <<< "$filesystem_info")"

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the reserved-block parser to the same here-string form in 83bfbfb.

@shunkakinoki
shunkakinoki force-pushed the codex/kyber-k3s-reserve-hardening branch from 4539f19 to 83bfbfb Compare July 19, 2026 07:36
@mesa-dot-dev

mesa-dot-dev Bot commented Jul 19, 2026

Copy link
Copy Markdown

Mesa Description

TL;DR

Hardened and optimized K3s activation reserve setup by caching ext4 filesystem metadata, introducing robust error handling, explicitly documenting kubelet defaults, and tightening test specs.

What changed?

  • config/k3s/kubelet.conf: Documented default kubelet image garbage collection thresholds to keep the host headroom contract explicit and visible.
  • home-manager/services/k3s/activate.sh: Optimized filesystem reserve configuration by caching tune2fs -l output, added fallback warning handling for tune2fs failures, and ensured require_sudo returns 0.
  • named-hosts/kyber/README.md: Added instructions to diagnose I/O pressure (/proc/pressure/io) before restarting services.
  • spec/k3s_service_activate_spec.sh: Added test assertion for require_sudo and tightened reserved space checking with a stricter, line-based grep pattern.

Description generated by Mesa. Update settings

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="home-manager/services/k3s/activate.sh">

<violation number="1" location="home-manager/services/k3s/activate.sh:57">
P3: The new best-effort behavior is unverified: a future change could make an inspection or mutation failure abort activation again without CI detecting it. A focused shellspec fixture that stubs `tune2fs` should cover both failures and assert activation continues while emitting the warning.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

fi

require_sudo || return 0
if ! filesystem_info="$(run_sudo @tune2fs@ -l "$root_source")"; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The new best-effort behavior is unverified: a future change could make an inspection or mutation failure abort activation again without CI detecting it. A focused shellspec fixture that stubs tune2fs should cover both failures and assert activation continues while emitting the warning.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At home-manager/services/k3s/activate.sh, line 56:

<comment>The new best-effort behavior is unverified: a future change could make an inspection or mutation failure abort activation again without CI detecting it. A focused shellspec fixture that stubs `tune2fs` should cover both failures and assert activation continues while emitting the warning.</comment>

<file context>
@@ -53,10 +53,14 @@ configure_root_ext4_reserve() {
   fi
 
   require_sudo || return 0
+  if ! filesystem_info="$(run_sudo @tune2fs@ -l "$root_source")"; then
+    echo "Warning: unable to inspect ext4 reserve on $root_source" >&2
+    return 0
</file context>

@shunkakinoki
shunkakinoki merged commit 79df42f into main Jul 19, 2026
29 of 33 checks passed
@shunkakinoki
shunkakinoki deleted the codex/kyber-k3s-reserve-hardening branch July 19, 2026 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant