fix(kanban): reap a superseded run's live worker before respawn

[tahleen]

What this fixes

A kanban run can reach a terminal handoff (kanban_request_review, kanban_request_changes, kanban_complete) and close in the DB while its worker process is still alive — the turn loop keeps calling the model against a run that no longer exists, and the process keeps holding tool calls. When the dispatcher then claims the same card again (review lane, then the fix lane), the previous process is still there: two workers mutate one card.

Measured on 2026-09-07: a worker whose kanban_request_review had already succeeded kept emitting tool calls for ~30 minutes (heartbeats, diagnostics, board calls rejected with expected_run_id mismatch). The dispatcher stacked runs behind it — 4–6 concurrent workers on one card (t_e46a5fea, runs 30 then 32-36), each one holding a claim slot. The reap closes that window at the source: before respawning a task, kill whatever is left over from its already-closed runs.

What it changes (dispatcher side only)

hermes_cli/kanban_db_dispatch.py:

• _spawned_event_pids(conn, task_id) — (run_id, pid) pairs read back from the task's append-only spawned events, newest first. This is the only durable trace of a run's PID: _end_run() clears task_runs.worker_pid when the run closes, so a monitor cannot find the stale process from the run row.
• _worker_query(task_id) — single source of the task-scoped query _worker_argv() passes to chat -q, so the guard below cannot drift from what is actually spawned.
• _cmdline_belongs_to_kanban_worker(pid, task_id) — PID-recycling guard: signal only a PID whose live cmdline still carries that task-scoped query, never a process that merely inherited the number.
• _reap_superseded_run_workers(conn, task_id, keep_run_id=...) — SIGTERM then SIGKILL, only for runs that are already closed (task_runs.outcome IS NOT NULL), emitting a reaped_superseded_worker event.
• Call site: _dispatch_lane_task(), immediately before the spawn, inside a try/except logged at debug level, so a failing reap can never block the dispatch path.

Existing termination paths on main (claim-expiry reclaim, operator reclaim, enforce_max_runtime, crash detection, archive_task) all key off the task leaving/staying in a state they can see; none of them covers "a new run of this card is about to spawn while the previous process is still alive", which is exactly the goal-mode review/fix cycle.

Tests

tests/hermes_cli/test_kanban_reap_superseded_workers.py (new, 4 tests):

• a closed run's live worker is SIGTERMed; an open run's worker is not;
• an innocent live PID (alive, but not this task's worker) is never signalled — asserted by making the terminate helper fail the test if it is called;
• no-op when the task has no spawned events;
• test_full_review_fix_cycle_keeps_single_live_worker: replays implementation → request_review → request_changes → request_review → complete through the real dispatch_once, with fake workers that do not exit after their terminal tool (the bug), asserting one live worker after every tick.

Local evidence on base origin/main b6b53c69a6:

• scripts/run_tests.sh tests/hermes_cli/test_kanban_reap_superseded_workers.py → 4 passed
• scripts/run_tests.sh tests/hermes_cli/test_kanban*.py → 53 files, 342 passed, 0 failed, 1 skipped (the skip is a windows_only test on Linux)
• uvx ruff check on both files → clean; py_compile clean

Related prior art

• fix(kanban): stop zombie workers after complete (#61923) #61967 / Kanban worker zombie process on Windows: complete_task doesn't terminate OS process, no process-state sync #61923 (open, mergeable_state: dirty since July) — "stop zombie workers after complete". Same symptom class, different mechanism: it stops the worker's own turn after a terminal tool, terminates the worker on manual lifecycle exits, and adds a tick-wide reap_orphaned_terminal_workers() backed by a spawn registry. This PR is spawn-time and per-card: it reaps only the task that is about to be respawned, reads the durable spawned events instead of a process-wide registry, and needs no assumption that the task left running (a review_requested card keeps being re-dispatched on the same board state). If fix(kanban): stop zombie workers after complete (#61923) #61967 lands, this stays the narrow guarantee that two runs of one card never overlap.
• fix(kanban): require one durable terminal handoff #91220 (open) — "require one durable terminal handoff" — covers the worker-side half (stop the turn on terminal success, plus a per-run durable handoff receipt). Deliberately not reimplemented here: this PR is dispatcher-only, and that one does it better.
• fix(kanban): trust a worker PID only inside the claim's PID namespace #104125 — trust a worker PID only inside the claim's PID namespace — about reclaiming workers believed dead seen from another PID namespace (containers). Here the target is a live process of an already-closed run, at respawn time.
• fix: drain embedded Kanban workers before gateway restart #103961 — drain embedded Kanban workers before gateway restart — restart-time drainage, unrelated to the respawn of one task.
• kanban: fail-closed reconciliation of running tasks against real OS processes #100529 — fail-closed reconciliation of running tasks against real OS processes (RFC-sized, unmerged) — reconciles running tasks against the OS on a schedule. Here nothing needs reconciling: the run is already closed in the DB, and the reap has to happen at the moment of respawn, per card.
• Same family, other levers, none of them spawn-time: fix(kanban): scope zombie reaper to own worker pids (fixes embedded-dispatcher MCP spawn race) #63728 (scope the zombie reaper to own worker PIDs), fix(kanban): classify zero-turn clean exits as route failures and reap before classifying #109468 (reap before classifying a zero-turn exit). Symbol-level search on the code this PR adds (_reap_superseded, reaped_superseded_worker, _reap_superseded_run_workers) returns 0 results across all PR states, re-checked immediately before opening this PR.

Notes

Opened from a fork (tahleen/hermes-agent), so there is no CI on the branch — the evidence above is local, on a fresh rebase of origin/main.