fix(kanban): hold reclaim while the worker is still alive - #49064
Merged
Conversation
Contributor
🔎 Lint report:
|
| Rule | Count |
|---|---|
invalid-assignment |
1 |
First entries
tests/run_agent/test_credits_notices_toggle.py:76: [invalid-assignment] invalid-assignment: Object of type `None` is not assignable to attribute `_credits_session_start_micros` of type `int`
✅ Fixed issues (2):
| Rule | Count |
|---|---|
unresolved-attribute |
2 |
First entries
run_agent.py:2971: [unresolved-attribute] unresolved-attribute: Object of type `Self@get_credits_spent_micros` has no attribute `_credits_session_start_micros`
tests/run_agent/test_credits_notices_toggle.py:76: [unresolved-attribute] unresolved-attribute: Unresolved attribute `_credits_session_start_micros` on type `AIAgent`
Unchanged: 5793 pre-existing issues carried over.
Diagnostics are surfaced as warnings — this check never fails the build.
alt-glitch
added
type/bug
release_stale_claims and detect_stale_running call _terminate_reclaimed_worker and then release the task claim unconditionally, even when the termination did not actually kill the worker. _terminate_reclaimed_worker already reports this via its "terminated" flag, but the callers ignore it. When a worker is parked in uninterruptible (D) state — for example throttled by a cgroup memory.high limit — a pending SIGTERM/SIGKILL cannot be delivered until the throttle lifts, so the kill is a no-op. The dispatcher then frees the claim and spawns a fresh worker beside the still-alive one. Repeated every dispatch tick this accumulates duplicate workers without bound, deepening the memory pressure that caused the throttle in the first place — a self-reinforcing runaway. Fix: gate both automatic reclaim paths on _worker_survived_termination(). When we attempted to kill our own host-local worker and it is still alive, defer the reclaim (_defer_reclaim_for_live_worker extends the claim a short grace and emits a reclaim_deferred event) instead of releasing. This guarantees at most one live worker per task and is self-correcting: not spawning a duplicate is what relieves the pressure so the pending signal lands and the worker dies, and the next tick reclaims cleanly. Non-host-local claims and the operator-driven reclaim_task() path keep their existing force-release behaviour. Related: #41448 (concurrent dispatchers amplify this by doubling reclaim frequency); #42858 (kill the worker rather than orphan it on archive). Tests: defer-when-worker-survives, reclaim-when-killed, release-when-not-host-local, and the detect_stale_running path.
_terminate_reclaimed_worker early-returned on ProcessLookupError with terminated=False. The new reclaim-defer guard reads that as 'worker survived the kill' and defers the reclaim forever, so a stale task whose worker is already dead never lands in result.stale. ProcessLookupError means the process is gone — that IS a successful termination. Split it from the generic OSError branch and set terminated=True.
teknium1
force-pushed
the
salvage/44909-kanban-reclaim-hold
branch
from
June 19, 2026 14:26
b8ef9ea to
2c489e1
Compare
1 task
This was referenced Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The kanban dispatcher no longer spawns a duplicate worker beside one that's still alive — fixing the self-reinforcing runaway observed at 100+ workers on a single board.
Salvage of #44909 by @Sahil-SS9 onto current
main(authorship preserved via cherry-pick).Root cause
release_stale_claims/detect_stale_runningreclaimed a TTL-expired task unconditionally, even when the kill we issued didn't land. A worker parked in uninterruptible D-state by a cgroupmemory.highthrottle can't receiveSIGKILLuntil the throttle lifts — so the claim got released and the dispatcher spawned a second worker beside the still-alive first. More workers → more memory pressure → more throttling → more duplicates.Changes
hermes_cli/kanban_db.py:_worker_survived_termination(termination)— true only when we attempted to kill our own host-local worker and it's still alive (attempted ∧ host_local ∧ ¬terminated). Foreign-host claims and no-op attempts fall through to normal release._defer_reclaim_for_live_worker(...)— extendsclaim_expiresbyRECLAIM_DEFER_GRACE_SECONDS(120s), keeps the taskrunning(no duplicate spawn), and records areclaim_deferredevent visible inhermes kanban tail.release_stale_claimsanddetect_stale_runningnow defer instead of release when the worker survived termination. Self-correcting: not spawning a duplicate is what finally lets the throttled worker die, after which the next tick reclaims cleanly.Validation
scripts/run_tests.sh tests/hermes_cli/test_kanban_db.py→ 218/218 pass, including 4 new cases: defer-on-survival, reclaim-on-kill-success, release-when-not-host-local, detect_stale defers.py_compileclean.Closes #44909.
Infographic