Skip to content

fix(daemon): detect a stale binary and self-restart instead of running outdated in-process job logic - #466

Merged
getappz merged 3 commits into
masterfrom
task/68
Aug 12, 2026
Merged

fix(daemon): detect a stale binary and self-restart instead of running outdated in-process job logic#466
getappz merged 3 commits into
masterfrom
task/68

Conversation

@getappz

@getappz getappz commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a background watchdog so a long-running agentflare daemon self-restarts once the on-disk binary it was launched from changes, instead of silently continuing to execute stale in-process job logic forever.

Why

This is the second confirmed occurrence of the same root cause:

Because work dispatch is in-process, a rebuild/merge no longer makes dispatch fail loudly — the daemon can run arbitrarily stale logic indefinitely with zero self-awareness, and failures are silent and hard to notice, not loud crashes.

What changed

  • src/daemon.rs: BinarySnapshot captures the on-disk binary's path/mtime/size via current_exe() at daemon startup. is_stale() is a cheap stat()-based comparison (mtime/size changed, or the file is gone). respawn_from_stale() cleans up pid/lock files first (so the replacement's own is_daemon_running() gate doesn't mistake the exiting process for a live one), spawns a replacement daemon via the same serve --_foreground-daemon invocation start_daemon/the installed systemd/launchd units use, and exits. 3 unit tests cover is_stale for unchanged/replaced/deleted files.
  • src/dashboard/server.rs: spawn_binary_staleness_watchdog() ticks every BINARY_STALENESS_CHECK_INTERVAL (60s) and calls respawn_from_stale once the snapshot goes stale. Wired into run() right after the worker pool starts.
  • In-flight-job risk: respawn_from_stale deliberately does not wait for jobs the WorkerPool may have running. This is documented as safe rather than a new gap — reconcile_orphaned_jobs (item feat: add eyre + color-eyre for rich error reporting #40) already makes an abruptly-dead daemon's in-flight jobs fail loud and release their claim, and a self-triggered respawn is just another instance of that same crash/kill-9 case it was built for.

Test plan

  • cargo build --bin agentflare — clean
  • cargo test --bin agentflare — 1263 passed; 1 pre-existing unrelated failure (mcp_prompts::tests::optimize_review_returns_full_skill_body, root-caused to a sandbox artifact — a read-only ~/.local/state mount rejecting flare-code's active-mode file write, unrelated to this change and reproducible on unmodified files). All 3 new binary_snapshot_tests pass.
  • End-to-end respawn path exercised for real: started the built binary as a foreground daemon in an isolated AGENTFLARE_HOME_OVERRIDE/XDG_RUNTIME_DIR sandbox, replaced the on-disk binary via rename (same shape as update::swap::replace_binary), and confirmed the watchdog detected the change and called respawn_from_stale within one 60s tick — old process pid gone, pid file updated to the new process, cleanup performed as designed.

Closes #68

@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 7 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 80997352-2dbd-46be-8da4-3c9608f46025

📥 Commits

Reviewing files that changed from the base of the PR and between 1d7d678 and 1317909.

📒 Files selected for processing (2)
  • src/daemon.rs
  • src/dashboard/server.rs

Comment @coderabbitai help to get the list of available commands.

shiva added 3 commits August 12, 2026 20:54
…n was cut off mid-verification

Agentflare-Agent: claude-code
Agentflare-Branch: task/68
Agentflare-Item: 68
…t jobs

respawn_from_stale exits the process without waiting for any job the
WorkerPool may have running on another thread. That looks like it could
reintroduce the exact silent-failure shape item #68 exists to close, but
it doesn't: dashboard::server's reconcile_orphaned_jobs already makes an
abruptly-dead daemon's in-flight jobs fail loud and release their claim
(item #40's crash/kill-9/OOM case), and this self-triggered exit is just
another instance of that same case.

Agentflare-Agent: claude-code
Agentflare-Branch: task/68
Agentflare-Item: 68
Fixes the fmt CI gate on this PR.

Agentflare-Agent: claude-code
Agentflare-Branch: task/68
Agentflare-Item: 68
@getappz
getappz merged commit 6a9791b into master Aug 12, 2026
16 checks passed
@getappz
getappz deleted the task/68 branch August 12, 2026 15:33
getappz added a commit that referenced this pull request Aug 14, 2026
execute_work's "claim succeeded but no worktree was created" and
"failed to chdir into <worktree>" paths were structural git-state
failures (e.g. a stale prunable worktree registration, confirmed live
for items #465/#466) that retry identically every time, but went
through the normal max_retries backoff budget before the terminal-
failure recovery hook could swap the item off `dispatched`.

Add a `fatal` flag to JobFailure/WorkOutcome that Queue::fail uses to
skip straight to terminal `state=failed` regardless of retries
remaining, and set it on those two setup-failure branches. Agent-run
failures are unaffected and keep the existing retry/cooldown behavior.
dispatch_item's ready-for-work/dispatched label-flip timing is
untouched.

Agentflare-Agent: claude-code
Agentflare-Branch: task/467-fail-fast-on-structural-worktree-setup-f
Agentflare-Item: 467
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[ponytail#291] AGENTS.md as fallback for instructions, not hardcoded duplicate

1 participant