Skip to content

fix(coding-agent): preserve live workers during recovery - #675

Closed
twaldin wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
twaldin:fix/preserve-live-workers-during-recovery
Closed

twaldin wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
twaldin:fix/preserve-live-workers-during-recovery

Conversation

@twaldin

@twaldin twaldin commented Aug 6, 2026

Copy link
Copy Markdown

Summary

  • Keep identity-verified live workers alive when adoption or recovery probes time out.
  • Continue recovery in the background, including when process identity observation is temporarily unavailable.
  • Preserve destructive recovery for confirmed process death or PID reuse, without replaying uncertain operations.

Why

A restarted supervisor can adopt a worker that is still alive but too busy to answer hello, worker_subscribe, or list within the control-plane deadlines. After three retries, recovery previously treated the matching live process as safe to SIGKILL, relaunched it, and marked every hosted session's in-flight operation interrupted. In one observed incident, this interrupted 18 concurrent RLM sessions at the same timestamp.

The supervisor cannot distinguish a load-slow worker from a wedged worker from probe timeouts alone. Keeping the verified process alive avoids irreversible work loss and retains the existing no-replay rule for uncertain side effects. Recovery continues on the existing deferred-retry path and will relaunch if the process later dies or its PID identity changes.

Verification

  • npx tsx ../../node_modules/vitest/dist/cli.js --run test/daemon-supervisor-monitor.test.ts (45 tests)
  • npm run check
  • Full package build

Adversarial review identified that the first bounded adoption probe can still delay supervisor readiness when worker_subscribe is load-slow. This PR moves recovery to the background after that first probe fails. Removing the first synchronous probe would change the existing startup contract by making healthy adopted sessions temporarily absent from the first list response, so that broader availability change is left out of this safety fix.


Generated by AI using OpenAI Codex gpt-5.6-sol.


Note

Medium Risk
Touches core daemon supervisor worker recovery and process-identity safety; wrong logic could leave wedged workers unrecovered or still kill live sessions, but changes narrow destructive recovery and add tests.

Overview
Fixes daemon supervisor recovery so identity-verified live session workers are not SIGKILL’d and relaunched when adoption or reconnect probes time out on a busy but healthy process.

Adoption: If the initial adopt probe fails but the PID is still alive and processStartId still matches (or identity is temporarily unreadable), the worker is marked recovering and recoverWorker runs in the background instead of immediately entering destructive recovery.

Recovery loop: On connect/subscribe failures against a verified live process, recovery keeps probing rather than treating the last retry as grounds to kill and relaunch. recoverUncertainWorkerOperations is no longer invoked with kill permission for that path (false instead of conditional SIGKILL). If probes still fail after the bounded delays, recovery defers via deferWorkerRecovery instead of failing the worker; relaunch remains for confirmed death or PID reuse.

Changelog entry and daemon-supervisor-monitor tests cover failed adoption on live workers and repeated probe timeouts / temporary missing process identity.

Reviewed by Cursor Bugbot for commit 0372118. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Fix DaemonSupervisor to preserve verified live workers during recovery

  • adoptOrRecoverWorker now checks if a worker process is alive and its processStartId matches before recovery; if it does, it marks the descriptor as "recovering" and schedules recoverWorker instead of immediately replacing the worker.
  • handleWorkerClose tracks a keepRetryingLiveWorker flag during the retry loop; if the worker is still alive (identity confirmed or temporarily unavailable), it defers recovery via deferWorkerRecovery rather than launching a replacement.
  • recoverUncertainWorkerOperations is now always called with false (never kill the live process) instead of a computed value.
  • Behavioral Change: live workers that fail adoption or exhaust retry attempts are no longer replaced — they remain in "recovering" state until a subsequent probe succeeds.

Macroscope summarized 0372118.

zhengr pushed a commit to zhengr/prime-agent that referenced this pull request Aug 8, 2026

Copy link
Copy Markdown
Member

Hi, thanks for taking the time to contribute to Prime Agent! Since open sourcing the project, we’ve received far more pull requests than we can responsibly review and validate. Prime Agent runs directly on users’ machines, so we need to be deliberate about which changes we accept and how they are reviewed. Rather than leave a large backlog that we cannot meaningfully work through, we’re closing the current PR queue and moving to a discussion-first contribution process.

We have established new contribution guidelines to help us continue iterating on Prime Agent and better manage contributions from the community. Going forward, we won’t review unsolicited pull requests. Instead, please start with a GitHub Discussion. We’ll identify recurring bugs and feature requests, create Issues for work we want to pursue, and invite pull requests from maintainers or vouched contributors when implementation is ready. Please read the full process documented in our contribution guidelines.

While we’re closing this backlog, we’re still reviewing it at a high level to identify recurring bugs, useful ideas, and important problems that we should address ourselves. Thanks again for the time you put into this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants