Skip to content

fix(k3s): preserve automatic crash restart - #2076

Merged
shunkakinoki merged 1 commit into
mainfrom
codex/fix-k3s-crash-restart
Jul 16, 2026
Merged

fix(k3s): preserve automatic crash restart#2076
shunkakinoki merged 1 commit into
mainfrom
codex/fix-k3s-crash-restart

Conversation

@shunkakinoki

@shunkakinoki shunkakinoki commented Jul 15, 2026

Copy link
Copy Markdown
Owner

Summary

  • remove the destructive k3s-killall.sh post-stop hook from the Kyber k3s unit
  • preserve systemd automatic restart after an unexpected k3s exit
  • add a regression check for the service lifecycle contract

Why

k3s-killall.sh stops k3s services and resets container/runtime networking state. Running it automatically after every service exit defeats Restart=always, which left Kyber k3s failed after the scheduler safety exit.

After this change, a normal systemctl stop k3s matches upstream behavior and does not perform a full container/network teardown. Operators can still run k3s-killall.sh explicitly when that full reset is intended.

Verification

  • shellspec spec/k3s_service_activate_spec.sh
  • shellcheck spec/k3s_service_activate_spec.sh
  • make nix-format-check
  • git diff --check

@indent-zero

indent-zero Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor
PR Summary

Restores k3s's automatic crash-restart behavior by removing an ExecStopPost=k3s-killall.sh hook that was tearing down containerd/pods on every stop — including the transient stop preceding a Restart=always respawn — which defeated auto-restart and worked against the unit's KillMode=process design of leaving workloads running so k3s can reattach. Aligns the unit with upstream k3s's canonical k3s.service.

  • Removed ExecStopPost=-@k3s@/bin/k3s-killall.sh from config/k3s/k3s.service; Restart=always, RestartSec=5s, and the ExecStartPre stale-socket cleanup are unchanged.
  • Added a shellspec block in spec/k3s_service_activate_spec.sh asserting Restart=always is present and that no ExecStopPost=...k3s-killall... line exists.
  • Force-push (434981f) tightened the negative assertion from The status should be failure to The status should equal 1, so a missing/unreadable service file (grep exit 2) can no longer silently pass the test.

Issues

1 potential issue found:

  • Manual systemctl stop k3s no longer performs a full teardown — because KillMode=process remains and the killall hook is gone, containerd/pods/CNI state now persist until k3s-killall.sh is run manually; worth noting in the PR body / user docs. → Autofix

CI Checks

All PR-relevant checks are green. The only non-success signal is Mesa Description (neutral), an external PR-description bot that errored with ENOSPC: no space left on device while generating its blurb — an infra issue on that bot's runner, unrelated to the k3s unit or spec changes. Re-triggerable by commenting /describe if a bot-generated description is wanted.


⚡ Autofix All Issues

@cursor

cursor Bot commented Jul 15, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@shunkakinoki
shunkakinoki enabled auto-merge (squash) July 15, 2026 20:18
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The k3s systemd unit no longer invokes k3s-killall.sh after stopping. Activation specs now verify Restart=always and ensure the removed ExecStopPost killall reference is absent.

Changes

k3s service behavior

Layer / File(s) Summary
Remove shutdown killall and verify service unit
config/k3s/k3s.service, spec/k3s_service_activate_spec.sh
The service removes the post-stop killall command, while specs verify restart behavior and the absence of the killall reference.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Poem

A rabbit hops past k3s tonight,
No killall script to end the flight.
Restarting steady, checks in place,
Clean shutdown leaves a gentler trace.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly reflects the main change: preserving automatic restart behavior for k3s.
Description check ✅ Passed The description matches the changeset by explaining the killall removal, restart behavior, and added regression check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-k3s-crash-restart

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request removes the destructive k3s-killall.sh script from the ExecStopPost directive in the k3s.service configuration and adds corresponding tests to verify this change. The feedback suggests a more robust assertion in the test suite by checking for an exit status of 1 instead of any failure status, which prevents false positives if the service file is missing or unreadable.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread spec/k3s_service_activate_spec.sh Outdated

It 'does not run the destructive killall helper after service exit'
When run grep 'ExecStopPost=.*k3s-killall' "$SERVICE"
The status should be failure

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Asserting The status should be failure can lead to false positives. If the service file $SERVICE is missing, renamed, or unreadable, grep will fail with exit status 2 (or another non-zero error status), which ShellSpec treats as a failure, causing the test to pass.

By asserting The status should equal 1, you ensure that grep successfully scanned the file (proving the file exists and is readable) but did not find the pattern.

Suggested change
The status should be failure
The status should equal 1

@shunkakinoki
shunkakinoki disabled auto-merge July 15, 2026 20:19
Comment thread config/k3s/k3s.service
@@ -19,7 +19,6 @@ Restart=always
RestartSec=5s
ExecStartPre=-@coreutils@/bin/rm -f /run/k3s/containerd/containerd.sock
ExecStart=@k3s@/bin/k3s server

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Operational behavior change on manual stop: Removing ExecStopPost=k3s-killall.sh is the right call for the auto-restart path (this is exactly what upstream k3s ships, and it lets KillMode=process + ExecStartPre=rm ... containerd.sock do their job so k3s can reattach to still-running containerd after a crash). One side effect worth mentioning in the PR description: a manual systemctl stop k3s will now leave containerd, pods, and CNI netns/mounts in place until an operator invokes k3s-killall.sh explicitly. Not a bug (matches upstream), just a UX change for anyone who was relying on systemctl stop k3s as a full teardown.

@shunkakinoki
shunkakinoki force-pushed the codex/fix-k3s-crash-restart branch from a24e7cf to 434981f Compare July 15, 2026 20:20
@shunkakinoki
shunkakinoki merged commit ab1bcbc into main Jul 16, 2026
38 checks passed
@shunkakinoki
shunkakinoki deleted the codex/fix-k3s-crash-restart branch July 16, 2026 07:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant