Skip to content

fix(gateway): bound permanent startup failures (#284) - #308

Merged
KrasimirKralev merged 3 commits into
betafrom
fix/gateway-restart-limiter
Aug 8, 2026
Merged

KrasimirKralev merged 3 commits into
betafrom
fix/gateway-restart-limiter

Conversation

@KrasimirKralev

Copy link
Copy Markdown
Contributor

Fixes #284. Supersedes #297 β€” same fix (@jamesachurchill's commits, authorship preserved), re-homed onto current beta and run through our flow.

The bug

clawbox-gateway.service had Restart=always and no start limit, so a permanently failing gateway restarted forever. (systemd's default 5-in-10s limiter never trips with RestartSec=5 β€” the window is too short β€” the exact footgun behind #284.)

The fix

  • StartLimitIntervalSec=3600 + StartLimitBurst=5 in the [Unit] section (not [Service], where modern systemd ignores them) + Restart=on-failure: a permanently-failing gateway now stops after 5 attempts with start-limit-hit instead of thrashing. Recovery is systemctl reset-failed && restart.
  • src/lib/gateway-health.ts: surfaces the breaker state on the authenticated gateway-offline page (unprivileged systemctl show; journalctl only when failed, scoped to the current invocation; secret redaction + HTML-escaping).

Verified on the Jetson (aarch64)

  • tsc --noEmit: clean Β· tests: 19 pass (gateway.test.ts, gateway-health.test.ts, install-post-update-units.test.ts).
  • Unit confirmed valid; limiter is in [Unit].
  • Live breaker test earlier on the box: beta config restarted forever (NRestarts climbed past 5); this config stopped at exactly 5 (failed), and reset-failed recovered β€” with the real clawbox-gateway untouched (throwaway units).

@KrasimirKralev
KrasimirKralev requested a review from a team as a code owner August 8, 2026 15:41
@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@KrasimirKralev, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 14 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
βš™οΈ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 81f3ec41-b317-4297-aca6-faad64cfde28

πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between 4e18a4d and 50b0cdf.

πŸ“’ Files selected for processing (7)
  • config/clawbox-gateway.service
  • install.sh
  • src/app/setup-api/gateway/route.ts
  • src/lib/gateway-health.ts
  • src/tests/routes/gateway/gateway.test.ts
  • src/tests/unit/gateway-health.test.ts
  • src/tests/unit/install-post-update-units.test.ts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❀️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

πŸ¦€ ClawReview

Your friendly reef crab, here with the lay of the land.

This PR fixes a systemd restart-loop bug where clawbox-gateway.service would thrash indefinitely on a permanent failure β€” the inherited 5-in-10s start-limit never fired because a single slow failure cycle already outlasted the window. The fix adds an explicit StartLimitIntervalSec=3600 + StartLimitBurst=5 breaker in the [Unit] section (the correct placement in modern systemd), switches Restart from always to on-failure, and adds systemctl reset-failed calls in install.sh so upgrades on already-crash-looping boxes don't get a refused restart. As a companion, a new gateway-health.ts surfaces the breaker state and a sanitized last-startup-error on the gateway-offline page. This is a re-landing of PR #297 onto current beta with authorship preserved.

At a glance

  • πŸ”§ Fix Β· touches systemd gateway service unit + install script + gateway offline page + new gateway-health lib
  • Base branch: beta Β· +167 source / +167 tests across 7 files
  • βœ… base beta matches the beta-first convention
  • βœ… conventional PR title
  • βœ… source changes come with test changes
  • ℹ️ touches security-sensitive paths (config/clawbox-gateway.service, install.sh) β€” review with extra care

Good to know

  • 🟑 PR fix(gateway): bound permanent startup failuresΒ #297 is still open and explicitly superseded by this one β€” it should be closed when this lands to avoid confusion.
  • 🟑 The new gateway-health.ts runs journalctl scoped to the current InvocationID and redacts secrets before surfacing log lines to the browser; the redaction regexes are a security detail worth a close look.
  • ℹ️ install.sh now calls systemctl reset-failed in three places on the gateway update/recovery path β€” important for existing devices whose gateway is in the breaker-tripped failed state.
  • ℹ️ 19 tests added across three files, including a unit that mathematically verifies the breaker window covers worst-case slow-failure cycles β€” and one that asserts XSS payloads in startup errors are HTML-escaped before rendering.

β€” ClawReview πŸ¦€, scuttling off. General info only β€” see CodeRabbit for the detailed review. Conventions: docs.

@github-actions github-actions Bot added area: install Auto-triage area area: gateway Auto-triage area area: ui Auto-triage area labels Aug 8, 2026
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

CI Summary

βœ… Tests

  • Result: passed
  • View run
  • Coverage: statements 70.13%, branches 60.31%, functions 65.95%, lines 72.06%

βœ… E2E

βœ… E2E Install

- Redaction now also covers password=/api_key=/bare sk-… provider keys, not
  just token/secret/credential/Bearer/telegram β€” the offline page is
  LAN-visible during setup, so a leaked key/password would be exposed.
- install.sh: reset-failed before every gateway restart/start in the setup +
  legacy-recovery paths, so a tripped start-limit breaker can't leave the
  gateway offline after an update (systemctl restart is refused until
  reset-failed once start-limit-hit).
- journalctl scoped by _SYSTEMD_INVOCATION_ID alone (dropping -u UNIT, which
  OR-expands and broke the AND) so the surfaced error is the current failure.
- Tighten health-probe timeouts (systemctl 3s->2s, journalctl 5s->2.5s) so the
  offline page can't block ~8s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@KrasimirKralev
KrasimirKralev merged commit 0e7dfb1 into beta Aug 8, 2026
8 checks passed
@KrasimirKralev
KrasimirKralev deleted the fix/gateway-restart-limiter branch August 8, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: gateway Auto-triage area area: install Auto-triage area area: ui Auto-triage area

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant