[automagik-genie]

Summary

recoverOnStartup iterates resumable workers and calls deps.isPaneAlive(worker.paneId) + attemptAgentResume without per-worker fault isolation. A single throw — most commonly a tmux socket that is not yet reachable when the daemon boots after a server reboot — aborts the for-loop, the outer try/catch in startDaemon logs recovery_error, and every subsequent worker stays stranded with resume_attempts=0 forever. Auto-resume is the only code path that would have revived them.

Smoking gun from a real server reboot

14:44:09.439  recovery_started
14:44:09.487  recovery_error   error="... display-message -t '%5' ...
                               error connecting to /tmp/tmux-1000/genie
                               (No such file or directory)"
(session 'genie' was created 30 s later at 14:44:39)

30-second startup race between genie serve and the tmux session. The first isPaneAlive call threw, aborted recovery for all 9 resumable workers. No retry mechanism existed.

Fix

1. Per-worker fault isolation. Extract the worker-recovery loop into runAgentRecoveryPass with a per-worker try/catch. One worker's failure no longer poisons the rest. Each failure emits a structured recovery_worker_failed log line with worker id, daemon id, and error message.
2. One-shot retry after transient failures. If any worker failed in the initial pass, schedule a single retry 60 s later via setTimeout(...).unref(). Covers the boot race where tmux is not yet listening. Retry uses the same fault-isolated pass and emits recovery_retry_scheduled / recovery_retry_completed / recovery_retry_error events.
3. Surface outcome. Add failed_agents to the existing recovery_completed log so the result of the initial pass is observable in one line.

Tests

Three new cases in scheduler-daemon.test.ts:

• one worker's isPaneAlive throwing does not stop recovery for the next
• a retry is scheduled when any initial failure occurs
• no retry is scheduled when every worker succeeds

Full file: 62/62 pass. Pre-push suite: 2479/2479 pass.

Test plan

• Unit tests cover the three new branches
• Full repo test suite passes locally
• CI on dev target passes
• Manual verification on a server with known stale worker rows: daemon restart triggers per-worker recovery with visible recovery_worker_failed + recovery_retry_scheduled events in ~/.genie/logs/scheduler.log

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: da205205-7150-4ba6-9ba6-56fa5e1d09ea

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/recover-on-startup-resilience

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0bdfa1dc2e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Cancel startup recovery retry when daemon stops

scheduleRecoveryRetry creates a one-shot setTimeout that always runs runAgentRecoveryPass after 60s, but this timer is not tied to daemon lifecycle (no running check and no handle returned for stop() to clear). In startDaemon, stop() clears other timers but cannot cancel this one, so stopping or restarting the daemon within that 60s window still lets the old instance run recovery and potentially resume agents out-of-band under a stale daemon_id, causing duplicate or conflicting recovery actions.

Useful? React with 👍 / 👎.

[gemini-code-assist]

Code Review

This pull request enhances the startup recovery process by implementing per-worker fault isolation and a one-shot retry mechanism for failed agent recoveries. These changes ensure that a single failure, such as an unreachable tmux socket, does not abort the entire recovery loop. A review comment identifies a potential issue where the retry timer is not tracked or cleared, which could lead to background tasks executing after the daemon has been shut down.

[gemini-code-assist]

The retry timer scheduled here is not tracked or cleared when the daemon stops. If startDaemon is stopped shortly after boot, this background task will still execute, potentially attempting to interact with a closed database connection or spawning processes after the daemon has gracefully shut down. Consider returning the timer handle from recoverOnStartup so it can be cleared in the stop() function of startDaemon.