Skip to content

fix(k3s): authorize galactica on kyber and surface client sync failures - #1738

Merged
shunkakinoki merged 3 commits into
mainfrom
fix/k3s-client-activation-logging
May 10, 2026
Merged

fix(k3s): authorize galactica on kyber and surface client sync failures#1738
shunkakinoki merged 3 commits into
mainfrom
fix/k3s-client-activation-logging

Conversation

@shunkakinoki

@shunkakinoki shunkakinoki commented May 10, 2026

Copy link
Copy Markdown
Owner

Summary

Two bugs prevented kubectl from working on galactica after make switch:

  1. Silent client failure. config/k3s/activate-client.sh discarded scp stderr (2>/dev/null) and used if scp; then ...; fi with no else branch. Any sync failure left ~/.kube/config-kyber absent while activation looked clean.
  2. Server never authorized galactica. The scp succeeded as a design only on the assumption that galactica's pubkey was already in kyber's ~/.ssh/authorized_keys. Nothing in the repo put it there, so a fresh sync always hit Permission denied (publickey).

While fixing #2 the galactica/kyber/matic pubkeys turned out to be hardcoded in three secrets.nix files, so they got pulled into a single shared module.

Changes

  • named-hosts/pubkeys.nix (new): single source of truth for the three host pubkeys.
  • named-hosts/{galactica,kyber,matic}/secrets.nix: inherit from pubkeys.nix instead of duplicating the strings.
  • config/k3s/default.nix: read galactica's pubkey from pubkeys.nix and template it into activate.sh via pkgs.replaceVars (matching the existing k3s.service pattern).
  • config/k3s/activate.sh: new ensure_authorized_key helper appends galactica's key to kyber's ~/.ssh/authorized_keys idempotently, with a placeholder guard so the unsubstituted @galacticaAuthorizedKey@ template is a no-op.
  • config/k3s/activate-client.sh: capture scp stderr to a temp file, log a one-line success on the happy path, on failure print k3s-client: failed to fetch kubeconfig from ... plus the indented scp output. Still exits 0 so a not-yet-authorized fresh machine does not break home-manager activation.
  • Specs:
    • spec/activate_k3s_spec.sh (+10 examples): placeholder declared, helper exists, fixed-string match used, idempotency, happy-path append, preserve-existing, no-op on unsubstituted placeholder.
    • spec/activate_k3s_client_spec.sh (+2 examples): reject 2>/dev/null, require failure-log string.

Test plan

  • nix flake check --no-build (galactica, runner, default eval clean)
  • shellspec spec/activate_k3s_spec.sh spec/activate_k3s_client_spec.sh -> 25 examples, 0 failures
  • shellcheck clean on both activate scripts
  • Reran the client script on galactica - new logging surfaced the underlying Permission denied (publickey) instead of vanishing
  • On kyber: make build && make switch, confirm a single k3s-server: authorized galactica SSH key for kubeconfig sync line and that ~/.ssh/authorized_keys contains galactica's pubkey exactly once after multiple runs
  • On galactica: make switch, confirm k3s-client: kubeconfig synced from kyber.tail950b36.ts.net and kubectl get nodes succeeds

Why: scp stderr was redirected to /dev/null and wrapped in an if-then
with no else branch, so any sync failure (host key mismatch, missing
authorized key, network drop) made make switch look successful while
silently leaving ~/.kube/config-kyber absent.

Now logs success and pipes scp stderr to the activation output on
failure, while still exiting 0 so a missing key on a fresh machine
does not break home-manager activation.
@indent-zero

indent-zero Bot commented May 10, 2026

Copy link
Copy Markdown
Contributor
PR Summary

Restores visibility of scp failures in the k3s client home-manager activation so a missing key, host-key mismatch, or network drop no longer leaves ~/.kube/config-kyber absent while make switch reports success. The script still exits 0 so first-time activation on a host without the remote key does not break home-manager.

  • config/k3s/activate-client.sh: capture scp stderr to a mktemp file (cleaned via EXIT trap), log a success line on sync, and on failure echo a header plus the prefixed scp stderr to stderr while preserving exit code 0.
  • spec/activate_k3s_client_spec.sh: add a failure visibility group asserting the script no longer contains 2>/dev/null near scp and emits the failed to fetch kubeconfig message.

Issues

1 potential issue found:

  • The new "does not silently discard scp stderr" spec only inspects lines containing scp , but the original 2>/dev/null lived on the scp continuation line — so a same-shape regression would slip past this guard. → Autofix

CI Checks

Waiting for CI checks...


⚡ Autofix All Issues

@mesa-dot-dev

mesa-dot-dev Bot commented May 10, 2026

Copy link
Copy Markdown

You do not have enough credits to review this pull request. Please purchase more credits to continue.

@coderabbitai

coderabbitai Bot commented May 10, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Rate limit exceeded

@shunkakinoki has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 40 minutes and 27 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3f21a55d-e744-4e6e-b9f8-e6425c3410fc

📥 Commits

Reviewing files that changed from the base of the PR and between a3cc703 and c3b49d2.

📒 Files selected for processing (7)
  • config/k3s/activate.sh
  • config/k3s/default.nix
  • named-hosts/galactica/secrets.nix
  • named-hosts/kyber/secrets.nix
  • named-hosts/matic/secrets.nix
  • named-hosts/pubkeys.nix
  • spec/activate_k3s_spec.sh
📝 Walkthrough

Walkthrough

This PR improves error handling in the k3s client activation script. The activate-client.sh script now captures scp stderr to a temporary file, reports success or failure explicitly, and emits captured errors with a k3s-client: prefix. Corresponding tests validate that errors are not silenced.

Changes

K3s Client Error Visibility

Layer / File(s) Summary
Error Capture and Reporting
config/k3s/activate-client.sh
scp stderr is captured to a temp file via mktemp, an EXIT trap cleans it up, and on failure the script prints a failure message and pipes captured errors through sed to prefix them with k3s-client: before sending to stderr.
Failure Visibility Tests
spec/activate_k3s_client_spec.sh
New test suite confirms that scp stderr is not suppressed via 2>/dev/null and that the kubeconfig fetch failure message appears in script output.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~5 minutes

Poem

🐰 A temporary file hops into place,
Stderr captured with grace,
Errors no longer hide away,
With k3s-client: prefix on display,
Tests confirm the brave new way!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The PR title mentions 'surface client sync failures' which directly corresponds to the main change in activate-client.sh, though it also references 'authorize galactica on kyber' which is a secondary objective not directly reflected in the file changes shown.
Description check ✅ Passed The PR description provides comprehensive context about the changes, explaining the bugs being fixed and the rationale for modifications to config/k3s/activate-client.sh and related test files.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/k3s-client-activation-logging

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@mesa-dot-dev

mesa-dot-dev Bot commented May 10, 2026

Copy link
Copy Markdown

Mesa Description

TL;DR

Fixes kubectl authorization issues on galactica when interacting with kyber by centralizing public key management and improving k3s client kubeconfig sync error reporting.

What changed?

  • named-hosts/pubkeys.nix (new): single source of truth for the three host pubkeys.
  • named-hosts/{galactica,kyber,matic}/secrets.nix: inherit from pubkeys.nix instead of duplicating the strings.
  • config/k3s/default.nix: read galactica's pubkey from pubkeys.nix and template it into activate.sh via pkgs.replaceVars (matching the existing k3s.service pattern).
  • config/k3s/activate.sh: new ensure_authorized_key helper appends galactica's key to kyber's ~/.ssh/authorized_keys idempotently, with a placeholder guard so the unsubstituted @galacticaAuthorizedKey@ template is a no-op.
  • config/k3s/activate-client.sh: capture scp stderr to a temp file, log a one-line success on the happy path, on failure print k3s-client: failed to fetch kubeconfig from ... plus the indented scp output. Still exits 0 so a not-yet-authorized fresh machine does not break home-manager activation.
  • Specs:
    • spec/activate_k3s_spec.sh (+10 examples): placeholder declared, helper exists, fixed-string match used, idempotency, happy-path append, preserve-existing, no-op on unsubstituted placeholder.
    • spec/activate_k3s_client_spec.sh (+2 examples): reject 2>/dev/null, require failure-log string.

Description generated by Mesa. Update settings

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request improves error visibility in the activate-client.sh script by capturing scp errors into a temporary file and logging them upon failure, rather than discarding them to /dev/null. Corresponding tests were added to verify these changes. However, one test case was identified as ineffective because it fails to capture the multi-line scp command structure; a suggestion was made to use grep -A 1 to correctly validate the removal of the stderr redirection.


Describe 'failure visibility'
It 'does not silently discard scp stderr'
When run bash -c "grep 'scp ' '$SCRIPT'"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The test 'does not silently discard scp stderr' is currently ineffective. Because grep 'scp ' only returns the first line of the multi-line scp command, it does not see the redirection (which was on the following line due to the backslash). Consequently, this test would pass even if 2>/dev/null were still present in the script. Using grep -A 1 ensures the redirection line is included in the check.

Suggested change
When run bash -c "grep 'scp ' '$SCRIPT'"
When run bash -c "grep -A 1 'scp ' '$SCRIPT'"

Describe 'failure visibility'
It 'does not silently discard scp stderr'
When run bash -c "grep 'scp ' '$SCRIPT'"
The output should not include '2>/dev/null'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regression test is too narrow to catch a recurrence: this assertion only looks at lines containing scp (the if scp ... line), but the original 2>/dev/null was on the continuation line "${REMOTE_HOST}:..." "$LOCAL_KUBECONFIG" 2>/dev/null; then — which has no scp token. Reintroducing the bug in the same shape would leave this grep output unchanged and the test would still pass.

Consider checking the whole script instead, e.g.:

It 'does not silently discard scp stderr'
When run bash -c "! grep -n '2>/dev/null' '$SCRIPT'"
The status should be success
End

or a multi-line grep that spans the scp invocation (grep -Pzo 'scp[^;]*2>/dev/null').

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="spec/activate_k3s_client_spec.sh">

<violation number="1" location="spec/activate_k3s_client_spec.sh:50">
P2: This regression test only greps the first line of the multi-line `scp` invocation, so it can miss a reintroduced `2>/dev/null` on the continuation line. Include the following line (or scan the whole script) before asserting stderr is not discarded.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.


Describe 'failure visibility'
It 'does not silently discard scp stderr'
When run bash -c "grep 'scp ' '$SCRIPT'"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This regression test only greps the first line of the multi-line scp invocation, so it can miss a reintroduced 2>/dev/null on the continuation line. Include the following line (or scan the whole script) before asserting stderr is not discarded.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At spec/activate_k3s_client_spec.sh, line 50:

<comment>This regression test only greps the first line of the multi-line `scp` invocation, so it can miss a reintroduced `2>/dev/null` on the continuation line. Include the following line (or scan the whole script) before asserting stderr is not discarded.</comment>

<file context>
@@ -44,4 +44,16 @@ When run bash -c "grep 'chmod 600' '$SCRIPT'"
+
+Describe 'failure visibility'
+It 'does not silently discard scp stderr'
+When run bash -c "grep 'scp ' '$SCRIPT'"
+The output should not include '2>/dev/null'
+End
</file context>
Suggested change
When run bash -c "grep 'scp ' '$SCRIPT'"
When run bash -c "grep -A 1 'scp ' '$SCRIPT'"

Why: scp from galactica to kyber failed with publickey denial because
galactica's pubkey was not in kyber's authorized_keys, so the silent
sync after switch was a no-op. The pubkey was already duplicated across
three secrets.nix files, so factor it out at the same time.

- Extract galactica/kyber/matic pubkeys to named-hosts/pubkeys.nix and
  inherit from each per-host secrets.nix.
- config/k3s/activate.sh idempotently appends galactica's pubkey to
  ~/.ssh/authorized_keys on kyber, with a placeholder guard so the
  unsubstituted template is a no-op.
- config/k3s/default.nix templates the pubkey through pkgs.replaceVars,
  sourcing it from the same shared file.
@shunkakinoki shunkakinoki changed the title fix(k3s): surface scp failures during client activation fix(k3s): authorize galactica on kyber and surface client sync failures May 10, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="spec/activate_k3s_spec.sh">

<violation number="1" location="spec/activate_k3s_spec.sh:74">
P2: Behavioral checks run a duplicated helper in the spec instead of the real `ensure_authorized_key` implementation, which can hide regressions in `config/k3s/activate.sh`.</violation>
</file>

Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.

Comment thread spec/activate_k3s_spec.sh
The output should include 'grep -qxF'
End

ensure_authorized_key_function() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Behavioral checks run a duplicated helper in the spec instead of the real ensure_authorized_key implementation, which can hide regressions in config/k3s/activate.sh.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At spec/activate_k3s_spec.sh, line 74:

<comment>Behavioral checks run a duplicated helper in the spec instead of the real `ensure_authorized_key` implementation, which can hide regressions in `config/k3s/activate.sh`.</comment>

<file context>
@@ -49,4 +49,100 @@ When run bash -c "grep -A 1 '! -f' '$SCRIPT'"
+The output should include 'grep -qxF'
+End
+
+ensure_authorized_key_function() {
+  cat <<'BASH'
+ensure_authorized_key() {
</file context>

The bash -c snippets quote with single quotes intentionally so the
inner $tmp/$HOME expand at the right moment. Add SC2016 to the
existing per-file disable list to keep CI green.
@shunkakinoki
shunkakinoki merged commit fd9491a into main May 10, 2026
28 of 30 checks passed
@shunkakinoki
shunkakinoki deleted the fix/k3s-client-activation-logging branch May 10, 2026 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant