fix(kanban): hold reclaim while the worker is still alive

[Sahil-SS9]

Problem

release_stale_claims and detect_stale_running call _terminate_reclaimed_worker(...) and then release the task claim unconditionally, regardless of whether the kill actually succeeded. _terminate_reclaimed_worker already reports this via its terminated flag, but both callers ignore it.

When a worker is parked in uninterruptible (D) state — for example throttled by a cgroup memory.high limit — a pending SIGTERM/SIGKILL cannot be delivered until the throttle lifts, so the termination is a no-op. The dispatcher then frees the claim and spawns a fresh worker beside the still-alive one. Repeated every dispatch tick, this accumulates duplicate workers without bound and deepens the memory pressure that caused the throttle in the first place — a self-reinforcing runaway. (Observed in production: a single board grew to 100+ live workers, load average 70+, swap exhausted.)

The task-claim CAS is correctly atomic, so this is not a double-claim — it is the reclaim path trusting a kill that never landed.

Fix

Gate both automatic reclaim paths on a new _worker_survived_termination() helper. When we attempted to kill our own host-local worker and it is still alive, defer the reclaim (_defer_reclaim_for_live_worker() extends the claim a short grace and emits a reclaim_deferred event) instead of releasing.

This guarantees at most one live worker per task and is self-correcting: not spawning a duplicate is what relieves the pressure so the pending signal lands and the worker dies, after which the next tick reclaims cleanly.

• Non-host-local claims (host_local=False) keep the existing release behaviour — we cannot manage a worker on another host, so stranding the claim would be worse.
• The operator-driven reclaim_task() path is intentionally not gated: an operator hitting "abort" wants the force-release, and it is a single deliberate action, not the automatic tick loop.

Tests

Adds four tests to tests/hermes_cli/test_kanban_db.py:

• defer when a host-local worker survives termination (claim held, reclaim_deferred emitted, no duplicate)
• still reclaims when termination succeeds
• still releases when the worker is not host-local
• the same defer behaviour in detect_stale_running

All existing reclaim/stale tests pass.

Related

• [Bug]: kanban dispatcher: gateway embeds dispatcher without singleton guard — two gateway instances cause claim-reclaim churn #41448 — concurrent dispatchers amplify this by doubling reclaim frequency (orthogonal; a singleton guard plus this fix together close the loop).
• [kanban] archive_task: kill the worker process instead of leaving it orphaned #42858 — kill the worker rather than orphan it on archive (same "do not abandon a live worker" principle, different path).

[liuhao1024]

Reviewed the diff — this correctly addresses the worker duplication race condition. When release_stale_claims or detect_stale_running tries to reclaim a TTL-expired claim, the new _worker_survived_termination guard prevents releasing the claim while the host-local worker is still alive (e.g., stuck in D-state under cgroup memory.high throttle). The claim is extended by RECLAIM_DEFER_GRACE_SECONDS (120s) and retried next tick — self-correcting because not spawning a duplicate is what lets the throttled worker finally receive the pending SIGKILL.

The guard is correctly scoped: termination_attempted AND host_local AND NOT terminated. Foreign-host claims (where we can't manage the worker) fall through to normal release. Both release_stale_claims and detect_stale_running paths are covered. Test coverage is thorough — 4 tests covering worker-survives, termination-succeeds, non-local-worker, and stale-running scenarios.

CI hasn't started yet. Clean implementation.

[tonydwb]

Code Review Summary

Verdict: Approved

Excellent fix with outstanding test coverage. This prevents a kanban worker duplication race condition when a cgroup-throttled worker survives termination attempts.

Looks Good

• Well-documented: detailed comment explaining the cgroup D-state scenario
• New helper functions are focused and well-named: _worker_survived_termination, _defer_reclaim_for_live_worker
• RECLAIM_DEFER_GRACE_SECONDS = 120 provides enough time for throttled workers to die
• The fix is applied in both release_stale_claims and detect_stale_running paths
• Comprehensive tests covering: surviving worker defer, successful termination reclaim, non-host-local bypass, detect_stale interaction
• Uses _append_event with reclaim_deferred kind for observability in hermes kanban tail

---

Reviewed by Hermes Agent

[teknium1]

Merged via #49064 — your commit was cherry-picked onto current main with authorship preserved (b9e521d). We added a small follow-up on top: ProcessLookupError from the kill now sets terminated=True (an already-gone process is terminated, not survived) — otherwise the new defer guard would hold a dead worker's claim forever. Thanks!