Skip to content

🐛 fix(kanban): reaper clears stale claims on dead workers in non-running lanes - #16

Merged
cwest merged 1 commit into
cwest/integrationfrom
topic/reaper-clear-stale-claim-nonrunning
Jun 29, 2026
Merged

🐛 fix(kanban): reaper clears stale claims on dead workers in non-running lanes#16
cwest merged 1 commit into
cwest/integrationfrom
topic/reaper-clear-stale-claim-nonrunning

Conversation

@cwest

@cwest cwest commented Jun 29, 2026

Copy link
Copy Markdown
Owner

Summary

A worker can die or be killed while its kanban card sits in a non-running
lane — most commonly review, after the implementer opened a PR and the card
moved on. The dead worker leaves claim_lock + worker_pid populated, and the
review-column dispatch query gates on claim_lock IS NULL, so the next worker
(the reviewer) cannot spawn until the 1h stale-claim TTL in
release_stale_claims eventually frees it. The lane wedges in the meantime.

Root cause

detect_crashed_workers scanned only WHERE t.status = 'running' and its
release UPDATE was guarded status = 'running'. A dead worker whose card had
already moved to review (or any non-running lane) was invisible to the fast
reaper — it never inspected the card, never cleared the stale lock. The TTL was
the only path that ever freed it. The reaper already keys liveness on
worker_pid (correct), but the status filter excluded the card before that
check ran.

Fix

Widen the scan to any card with a non-NULL worker_pid (status added to the
projection). After the existing host-local, launch-grace, and dead-PID checks
pass, branch on status:

  • non-running card → clear the dead claim in place
    (claim_lock/claim_expires/worker_pid = NULL) with a
    stale_claim_cleared event and no lane change — a dead worker in review
    stays in review so the reviewer re-spawns, rather than being yanked back to
    ready. No run is opened/closed, no crashed event is emitted, and the
    failure counter / circuit breaker is not touched (those are running-crash
    semantics).
  • running card → unchanged. The crash/requeue/breaker machinery runs
    exactly as before; its UPDATE stays guarded status = 'running' and only ever
    fires for running cards (non-running ones continue out above).

Launch-grace (measured from the active task_runs row) and the host-local claim
check still apply, so a freshly-spawned worker is not reaped mid-init and a
foreign-host claim is left alone.

Tests

4 new regression tests in tests/hermes_cli/test_kanban_db.py, E2E-style
against a temp HERMES_HOME with the real kanban_db import:

  • dead host-local worker_pid on a review card → claim cleared, status stays
    review, no failure counted;
  • after the clear, the card matches the review-dispatch query (fresh spawn
    permitted);
  • a within-launch-grace claim on a review card is not cleared;
  • a foreign-host claim on a review card is left untouched.

Full tests/hermes_cli/test_kanban_db.py green (230 passed); the 35
crash/reaper/stale/grace/rate-limit tests — including the breaker-trip and
systemic-block paths — confirm the running-crash semantics are unchanged.

Scope

Single-file change in hermes_cli/kanban_db.py + tests + a PATCHES.md
upstream-pending row. A reaper that ignores non-running lanes is a general
dispatcher bug, not homestead-specific — a clean upstream candidate.

…ing lanes

A worker can die or be killed while its card sits in a non-`running` lane —
most commonly `review`, after the implementer opened a PR and the card moved
on. The dead worker leaves `claim_lock` + `worker_pid` populated, and because
the review-column dispatch query gates on `claim_lock IS NULL`, the next worker
(the reviewer) cannot spawn until the 1h stale-claim TTL in
`release_stale_claims` eventually frees it — the lane wedges.

Root cause: `detect_crashed_workers` scanned only `WHERE t.status = 'running'`
and its release UPDATE was guarded `status = 'running'`, so a dead worker whose
card had already moved to `review` was invisible to the fast reaper. The TTL was
the only path that ever cleared it.

Widen the scan to any card carrying a non-NULL `worker_pid` (status added to the
projection). After the existing host-local, launch-grace, and dead-PID checks
pass, branch on status: a non-`running` card has its claim cleared in place
(`claim_lock`/`claim_expires`/`worker_pid` = NULL) with a `stale_claim_cleared`
event and NO lane change — a dead worker in `review` stays in `review` so the
reviewer re-spawns, rather than being yanked back to `ready`. The clear opens no
run, emits no `crashed` event, and does not touch the failure counter or circuit
breaker; those are `running`-crash semantics and stay byte-for-byte unchanged
(non-`running` cards `continue` out before that machinery).

Launch-grace (measured from the active `task_runs` row) and the host-local claim
check still apply, so a freshly-spawned worker is not reaped mid-init and a
foreign-host claim is left alone.

Adds 4 regression tests: a dead worker in `review` has its claim cleared with
status staying `review` and no failure counted; the card is then eligible for
review dispatch; a within-grace claim is NOT cleared; a foreign-host claim is
untouched. Recorded in PATCHES.md as upstream-pending — a reaper that ignores
non-running lanes is a general dispatcher bug, a clean upstream candidate.
@cwest
cwest force-pushed the topic/reaper-clear-stale-claim-nonrunning branch from 22962ba to dc458d0 Compare June 29, 2026 01:48

@cwest cwest left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix lands where the wedge actually was. detect_crashed_workers only ever looked at running cards, so a dead worker whose card had already moved to review sat there holding claim_lock until the 1h TTL — and the review-column dispatch gates on claim_lock IS NULL, so no reviewer could spawn in the meantime. Widening the scan to any non-NULL worker_pid and branching on status is the right shape.

The branch ordering is correct: host-local, launch-grace, and dead-PID checks all run before the new code, so a freshly-spawned worker still gets its grace window and a foreign-host claim is left alone. The non-running clear sits behind a continue, so it never reaches the run open/close, the crashed event, or the failure-counter and breaker — those stay running-only. The running path's UPDATE is unchanged, still guarded on status = 'running'.

The in-place UPDATE is guarded on id, status, and worker_pid together, so if the row moved between the SELECT and the UPDATE the rowcount won't be 1 and no stale_claim_cleared event fires. No lane change means a dead worker in review stays in review and the reviewer re-spawns instead of getting yanked back to ready.

Verified by running the suite in a throwaway clone at the head commit: the four new tests pass and the full tests/hermes_cli/test_kanban_db.py is 230 passed. ruff is clean on both changed files. mergeable is MERGEABLE, every check is success, and there are no unresolved threads.

No changes needed.

@cwest
cwest marked this pull request as ready for review June 29, 2026 01:51
@cwest
cwest merged commit 3a06537 into cwest/integration Jun 29, 2026
8 checks passed
@cwest
cwest deleted the topic/reaper-clear-stale-claim-nonrunning branch June 29, 2026 01:54
cwest added a commit that referenced this pull request Jul 1, 2026
…g lanes

Clear a dead worker's stale claim regardless of lane, so a worker that dies
while its card sits in a NON-running lane (most commonly review, after the
implementer opened a PR and the card moved on) no longer wedges that lane for
the full 1h stale-claim TTL. Widens the crash scan to any card with a non-NULL
worker_pid and, for a non-running card, does an in-place claim clear with a
stale_claim_cleared event and NO lane change; the running path is byte-for-byte
unchanged.

upstream-pending: fork PR #16
cwest added a commit that referenced this pull request Jul 26, 2026
…g lanes

Clear a dead worker's stale claim regardless of lane, so a worker that dies
while its card sits in a NON-running lane (most commonly review, after the
implementer opened a PR and the card moved on) no longer wedges that lane for
the full 1h stale-claim TTL. Widens the crash scan to any card with a non-NULL
worker_pid and, for a non-running card, does an in-place claim clear with a
stale_claim_cleared event and NO lane change; the running path is byte-for-byte
unchanged.

upstream-pending: fork PR #16
(cherry picked from commit b5420f3)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant