Skip to content

fix(coding-agent): restore externally deleted supervisor owner records - #1225

Open
irl-dan wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
irl-dan:fix/daemon-supervisor-owner-record-restore
Open

fix(coding-agent): restore externally deleted supervisor owner records#1225
irl-dan wants to merge 1 commit into
PrimeIntellect-ai:mainfrom
irl-dan:fix/daemon-supervisor-owner-record-restore

Conversation

@irl-dan

@irl-dan irl-dan commented Aug 11, 2026

Copy link
Copy Markdown

Summary

A daemon that runs for several days can stop accepting work. Every new CLI launch then fails within seconds. The daemon looks healthy from the outside, so nothing replaces it. Only a manual shutdown clears the state. This change lets the supervisor repair itself and keep serving.

Crash signature

Every CLI launch exits with code 1 after 1-2 seconds:

Error: Daemon supervisor generation <uuid> no longer owns its registry entry
    at deserializeDaemonError (packages/coding-agent/src/modes/daemon/daemon-errors.ts:34:9)
    at createDaemonClientConnection (packages/coding-agent/src/main.ts:1002:10)

Root cause

The supervisor writes an owner record to the registry. The registry is in the OS temporary directory: os.tmpdir()/prime-agent-<uid>/supervisor-owners/<generation>.owner/. The owner record has two files: owner.json and scope.json.

OS cleaners delete old files in temporary directories. The macOS /var/folders cleaner removes files after about three days. It deletes the two record files in place. It keeps the record directory, the socket, and the lockfile directories. We observed this on a supervisor with four days of uptime. The same sweep emptied hundreds of unrelated temporary directories.

The supervisor validates the owner record for each command (daemon-supervisor-ownership.ts:130-138). After the deletion, validation always fails. The supervisor rejects every command with the error above.

The supervisor keeps its socket. It still sends a current-version daemon_hello. Client probes report the daemon as current. So no client starts a replacement. The daemon stays in this state until a person stops it.

Workers validate the same owner record every 250 ms (daemon-mode.ts). They drop their supervisor connections for the same reason.

Reproduction

The steps are deterministic and need no API keys:

SANDBOX=$(mktemp -d /tmp/pa-repro.XXXX)
export PRIME_AGENT_INTERNAL_DAEMON_SUPERVISOR_REGISTRY_DIR=$SANDBOX/registry
export PRIME_AGENT_CODING_AGENT_DIR=$SANDBOX/agent
# 1. Start a sandbox supervisor.
prime-agent --mode daemon --daemon-socket $SANDBOX/daemon.sock --offline &
sleep 2
# 2. Delete the owner record files in place, as the cleaner does.
rm $SANDBOX/registry/*.owner/owner.json $SANDBOX/registry/*.owner/scope.json
# 3. Each client command now fails with the signature.
prime-agent --daemon-socket $SANDBOX/daemon.sock -p --no-skills --no-context-files -- 'hi'

On main (14d6e74), 15 of 15 concurrent CLI launches fail with the signature. With this fix, 0 of 15 fail. The supervisor logs one line: Restored supervisor owner record for generation <uuid> after external removal.

Fix

The fix has two goals. A live supervisor must survive the loss of its owner record. A superseded generation must not replace its live successor.

  • New method DaemonSupervisorOwnership.restoreIfUnowned(). It runs under the registry guard. It confirms that the owner record is absent. It then runs the same conflict scan as acquisition. If no live conflicting owner exists, it writes the owner record again with the same generation and token. If a live conflicting owner exists, it refuses. If a different record is present, it also refuses.
  • DaemonSupervisor.assertCurrentOwnership() calls the restore when validation fails with supervisor_generation_stale. It then validates again. The restore is single-flight: concurrent commands share one scan. If the restore refuses, the original error propagates.
  • The idle-eviction sweep runs the same ownership check. An idle supervisor heals without client traffic. Workers re-fence within 250 ms after the owner record returns.

Tests

Three new cases in test/suite/regressions/4600-supervisor-singleton.test.ts:

  • Restore succeeds after the record files are deleted in place. Restore also succeeds after the record directory is removed. A released ownership refuses to restore.
  • A generation with a deleted record does not restore over a live successor.
  • End to end: a supervisor with a deleted record continues to serve list. It writes the owner record again with the same generation and token.

Results: 4600-supervisor-singleton 18/18 pass. daemon-supervisor-process 8 pass, 8 tag-skipped. daemon-supervisor-eviction 13/13 pass. daemon-socket 7/7 pass. npm run check passes.

Generated with Claude Code

A daemon supervisor's durable owner record (owner.json / scope.json) lives
in a registry directory under the OS temporary directory. System temp
cleaners - notably the macOS /var/folders reaper, which deletes regular
files it considers old enough - can remove those files out from under a
long-running supervisor while leaving its socket, lockfiles, and process
intact.

The supervisor re-validates its registry entry on every daemon command, so
after such a removal it permanently refuses every command with

  Daemon supervisor generation <uuid> no longer owns its registry entry

while still holding the socket and greeting clients with a current-version
hello. Clients therefore neither succeed nor replace it: every new CLI
launch connects, sends its first command, and exits 1. The daemon is
poisoned until manually shut down.

Teach the supervisor to repair this state: when an ownership assertion
fails because the durable record is gone, re-run the acquisition conflict
scan under the registry guard and rewrite the record (same generation and
token) if - and only if - no other live supervisor owns a conflicting
scope. A superseded generation still cannot resurrect itself past a live
successor. The restore is single-flight across concurrent commands, and the
periodic idle-eviction sweep piggybacks the same check so an idle
supervisor heals without waiting for the next client command.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@irl-dan
irl-dan force-pushed the fix/daemon-supervisor-owner-record-restore branch from 7685d08 to 2dc9224 Compare August 11, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant