fix(coding-agent): finalize timed-out worker stops instead of stranding registrations

[snimu]

Problem

When a worker refused to exit in time (say it was suspended), the stop gave up halfway. That left behind a dead worker that was still registered, with a stop marker saved on disk. The session was then stuck with "Session worker is not connected" until someone repaired it by hand. This is the incident behind #836.

On top of that, the liveness check counted zombie processes as alive, so cleanup could wait forever on a process that was already dead.

Fix

• If a stop times out, the supervisor now keeps watching in the background: after 5 seconds it force-kills the process, waits for it to die, and then finishes the cleanup it started (archive if requested, remove the registration).
• Liveness checks now treat zombies as dead (isProcessAlive / new isZombieProcess in shared utils).

What this does not change

Only the timed-out case gets new behavior. If the user retries the worker in the meantime (#836), the retry wins and the cleanup backs off. Normal stops are untouched.

Validation

• 3 unit tests: cleanup after the process dies, force-kill escalation, backing off after a retry.
• Zombie detection tested against a real zombie process.
• A real end-to-end test: suspend a worker, let the stop time out, and watch the supervisor clean it up by itself.
• npm run check passed.

---

Part 2 of 3, on top of #850. Next: #852.

---

Note

High Risk
Changes how the daemon supervisor kills processes and removes worker registrations, including PID recycling and identity edge cases—core session infrastructure with safety-sensitive signaling behavior.

Overview
When a session worker does not exit within the stop deadline, the supervisor no longer leaves a tombstoned registration stranded forever. It schedules background finalization that keeps polling (with throttled identity checks), escalates to SIGKILL after a grace period, and retries archive/cleanup until the registration is removed or the stop is rescinded.

Worker stops are bound to the process generation (pid + processStartId) for the whole stop path: signals go out only when identity is current, recycled pids are treated as gone, and transient identity failures are treated conservatively so live workers are not cleaned up or unrelated processes are not killed. Relaunches and rescinded tombstones abort in-flight stop cleanup.

Shared liveness helpers in child-process.ts split cheap kill(0) existence from zombie-aware isProcessAlive (zombies count as dead). Adoption of an intentional stop no longer SIGKILLs before the normal stop flow.

Daemon shutdown treats WorkerStopTimeoutError as non-fatal so shutdown can finish while tombstoned workers are finalized in the background.

^{Reviewed by Cursor Bugbot for commit 0b0c43c. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Finalize timed-out worker stops via background SIGKILL instead of stranding registrations

• Timed-out worker stops now schedule background finalization (scheduleWorkerStopFinalization) that periodically checks process liveness, escalates to SIGKILL once identity is confirmed, and retries cleanup until the registration is removed.
• Adds process identity tracking (processStartId) to avoid signalling recycled PIDs — SIGKILL is only sent when the PID's start ID matches the stopped worker's recorded identity.
• Introduces effectiveWorkerState so disconnected "ready" workers are reported as "recovering" and tombstoned workers as "stopping", with handleList returning honest states for busy-daemon checks.
• Bumps DAEMON_SCHEMA_REVISION to 16 and adds "stopping" to the DaemonWorkerLifecycle and SessionSummary unions to surface the new state to clients.
• Shutdown now catches WorkerStopTimeoutError and leaves tombstoned workers for recovery rather than failing the shutdown sequence.
• Risk: Workers that time out during stop are now cleaned up asynchronously; if the supervisor restarts before finalization completes, recovery relies on the tombstone left in the descriptor.

^{Macroscope summarized da89f2d.}

[snimu]

Fixed in a096b3d. The finalizer now snapshots pid, processStartId, and stopRevision at schedule time and rechecks the stop generation inside the loop before every poll and before SIGKILL escalation, so a rescinded/retried stop aborts it immediately and a relaunched worker's new pid is never followed. A pid whose observed processStartId no longer matches is treated as gone and never signalled; stopWorker's signalling is identity-aware too. Added regression tests for the retry-relaunch race and the recycled-pid case.

[snimu]

Addressed the three remaining findings in 62a5265:

• Fail-closed identity: with a recorded processStartId, an unobservable identity now counts as a different process, so a recycled pid is never signalled even when /proc/ps observation fails.
• Missing start id: when a worker never had a recorded identity, the finalizer records one at schedule time (the process was provably alive moments earlier), so the eventual stopWorker is identity-guarded too.
• Transient cleanup failure: finalization now retries (5s backoff) until the registration is gone or the stop is rescinded, instead of stranding the tombstone permanently.
• Polling cost: every poll now uses a cheap kill(0) existence probe; the ps-backed zombie/identity checks are throttled to once per 500ms in both the stop wait loop and the finalizer.

Regression tests added for the transient-failure retry; existing recycled-pid and retry-relaunch tests updated for the new probe.

[snimu]

Fixed in ee9e59d (and a6e481d on #852). The identity check is now directional instead of a single boolean:

• workerProcessIdentity() returns current / replaced / gone / unknown.
• Signalling (SIGTERM/SIGKILL) requires confirmed current — a recycled pid is never signalled.
• Cleanup (registration removal, archive, reclaim) requires confirmed gone/replaced — an unknown verdict from a transient getProcessStartId failure keeps waiting and never orphans a possibly-live worker.

Regression test added: with a live pid and an unobservable identity, the finalizer neither signals nor cleans up.

[snimu]

Fixed in 22dbec4. stopWorker's cleanup tail now verifies the registered process is still the one it stopped after every await (including catalog archival): if a retry rescinded the stop and relaunched the worker mid-await, the stale invocation aborts with an error instead of deleting the successor's registration and descriptor. Regression test covers the relaunch-during-archival race.

Also merged the latest main (including #836) through the stack.

[snimu]

Fixed in 46c8241. The throttled cache is now used only for read-only wait-loop polling; both SIGKILL sites (stopWorker's force escalation and the finalizer) run a fresh, unthrottled identity check immediately before signalling, so a pid recycled inside the 500ms window is never killed. Regression test flips the observed identity between the last throttled poll and the SIGKILL deadline and asserts the signal is withheld.

[alexanderkjeldaas]

This is causing me endless issues. Thanks for the fix.

[snimu]

Fixed in c864b10. killed is now only set when the SIGKILL was actually sent: a transiently unobservable identity at the deadline skips that attempt but keeps escalation armed, so a later pass that re-verifies the original process still kills it. Regression test covers an identity outage spanning the deadline followed by recovery.

Also merged the latest main through the stack; the schema bump moved to revision 15 because main took revision 14 for the telemetry opt-out.

[snimu]

Both fixed in 37345d5:

• Mid-stop relaunch SIGKILL: all of stopWorker's polling and signalling (SIGTERM gate, throttled wait-loop verdict, fresh pre-SIGKILL check) now bind to entryPid/entryStartId captured at entry instead of reading the mutable worker.descriptor, so a retry that swaps in a successor process mid-stop can never be signalled by the stale stop. The shared helper is now pid-based (processIdentity(pid, startId)), which fix(coding-agent): self-heal stale worker registrations on resume #852's reclaim reuses unchanged.
• Rescinded-before-relaunch cleanup: the cleanup guard (assertStopStillApplies) now also aborts when a removeDescriptor stop has lost its tombstone (stopRequestedAt cleared), catching a rescission that lands before recoverWorker assigns the successor pid. Regression test covers exactly that window.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a960653. Configure here.}

The merge-base changed after approval.