fix(kanban): has_spawnable_ready excludes respawn-guarded tasks - #44899
Open
Zazzles2908 wants to merge 1 commit into
Open
fix(kanban): has_spawnable_ready excludes respawn-guarded tasks#44899Zazzles2908 wants to merge 1 commit into
Zazzles2908 wants to merge 1 commit into
Conversation
The dispatcher 'stuck' detection uses has_spawnable_ready() to
decide whether '0 spawned' is a real stuck condition (genuinely
spawnable work waiting) or a correctly-idle condition (only
control-plane lanes waiting). But has_spawnable_ready() previously
returned True for tasks that the dispatch path would correctly
skip via check_respawn_guard — so the gateway fires a false
'kanban dispatcher stuck' warning every 6 ticks (60s) when:
- the ready queue has tasks (non-empty)
- those tasks are all respawn-guarded (rate_limit_cooldown,
blocker_auth, recent_success, active_pr)
- dispatch_once correctly defers them
- but the health check thinks they're spawnable work
Root cause: has_spawnable_ready() and dispatch_once() use
DIFFERENT criteria for 'spawnable'. has_spawnable_ready() checks
ready+assigned+unclaimed+valid-profile. dispatch_once() checks
the same AND NOT respawn-guarded. The mismatch causes the
false warning.
Fix: call check_respawn_guard for each candidate task and return
True only if at least one is NOT guarded. Same posture as the
patch 4 fix to check_respawn_guard itself (result IS NOT NULL
early-return). Fail-open: if check_respawn_guard raises, assume
spawnable — same behavior as the legacy code path.
has_spawnable_review(): mirror the function signature for
compatibility but deliberately do NOT filter on guard (the
dispatch path doesn't call check_respawn_guard on review tasks
— that's a different layer of the system).
Repro before fix:
Single task in 'ready' with completed task_runs row (recent_success)
has_spawnable_ready() returns True
dispatch_once() defers the task
Gateway logs: 'kanban dispatcher: tick failed on board default'
Repro after fix:
Same task, has_spawnable_ready() returns False
No 'stuck' warning. Dispatcher correctly idle.
Production evidence: WSL2 hook
~/.hermes/hooks/wsl2-kanban-monkeypatches/handler.py:241-367
implements this exact filtering. Live dispatch health: 5 active
workers, 0 failed ticks since 2026-06-12.
Closes: bead prod-audit-wr2yw
Tested: 6-scenario test (unguarded, recent_success guard,
blocker_auth guard, mixed, empty, mixed with blocker_auth) —
all 6 pass
tonydwb
approved these changes
Jun 13, 2026
tonydwb
left a comment
There was a problem hiding this comment.
Code Review Summary
Verdict: Approved
Good telemetry fix. has_spawnable_ready now also checks check_respawn_guard so the gateway health telemetry does not false-positive a "kanban dispatcher stuck" warning every 6 ticks for respawn-guarded tasks.
Looks Good
- Uses
globals().get("check_respawn_guard")for safe fallback in partial-install environments - The check is applied to each individual task row rather than batch-filtered, allowing the loop to short-circuit
- Clear documentation distinguishing ready-task guard checking from review-task (which does not use
check_respawn_guardin the dispatch path) - Documented note explaining why review tasks intentionally skip the guard filter
Reviewed by Hermes Agent
4 tasks
Contributor
|
Thanks for tracing the dispatcher-health mismatch. The current-head premise is confirmed: Problems
Suggested changes
Automated hermes-sweeper review. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The dispatcher "stuck" detection uses
has_spawnable_ready()to decide whether0 spawnedis a real stuck condition (genuinely spawnable work waiting) or a correctly-idle condition. Buthas_spawnable_ready()previously returned True for tasks that the dispatch path would correctly skip viacheck_respawn_guard— so the gateway fires a false "kanban dispatcher stuck" warning every 6 ticks (60s) when:rate_limit_cooldown,blocker_auth,recent_success,active_pr)dispatch_oncecorrectly defers themThis is a 42-line addition to
hermes_cli/kanban_db.py:5968-6022that callscheck_respawn_guardfor each candidate and returns True only if at least one task is not guarded.Root cause
has_spawnable_ready()anddispatch_once()use different criteria for "spawnable":has_spawnable_ready()(telemetry)ready AND assignee IS NOT NULL AND claim_lock IS NULL AND profile_exists(assignee)dispatch_once()(actual dispatch)The mismatch causes the false warning.
Repro
Before fix:
After fix:
Tested 6 scenarios:
True✓False✓False✓True(at least one spawnable) ✓False✓True✓Fix
def has_spawnable_ready(conn: sqlite3.Connection) -> bool: rows = conn.execute( - "SELECT DISTINCT assignee FROM tasks " + "SELECT id, assignee FROM tasks " "WHERE status = 'ready' AND assignee IS NOT NULL " " AND claim_lock IS NULL" ).fetchall() if not rows: return False try: from hermes_cli.profiles import profile_exists except Exception: return True + check_respawn_guard = globals().get("check_respawn_guard") for row in rows: - if profile_exists(row["assignee"]): - return True + if not profile_exists(row["assignee"]): + continue + if check_respawn_guard is not None: + try: + if check_respawn_guard(conn, row["id"]) is not None: + continue + except Exception: + pass + return True return Falsehas_spawnable_review()also gets theidcolumn added to the SELECT (consistency) and usescontinueinstead of early-return, but does NOT filter on guard —dispatch_oncedoesn't callcheck_respawn_guardon review tasks, so the review path stays consistent with the dispatch path.Why fail-open?
If
check_respawn_guardraises or is unavailable, we return True. This is the same posture as the existingprofile_existsfallback (line 5999) and the existingdispatch_oncepath. We never wanthas_spawnable_readyto silently return False when a guard check transiently fails — that would suppress legitimate "stuck" warnings and hide real dispatch bugs.Production evidence
The WSL2 consolidation hook at
~/.hermes/hooks/wsl2-kanban-monkeypatches/handler.py:241-367implements this exact filtering logic. That hook has been live in the user'shermes-agentdeployment since 2026-06-12 with 5 active workers and 0 false "stuck" warnings. Filing this PR retires the hook dependency for this specific fix.Risk
dispatch_onceinvariant (which already skips guarded tasks viacheck_respawn_guard).Test plan
tests/hermes_cli/test_kanban_db.pytests forhas_spawnable_readyandhas_spawnable_reviewBead
prod-audit-wr2yw(parent epic for upstream PRs in this batch)Note: this PR is the third of a 3-PR series that retires the WSL2 consolidation hook dependency:
recompute_readycancelled-parent fix (1-line)check_respawn_guardresult-skip (9-line)has_spawnable_readyexclude respawn-guarded (42-line)Each PR is independently mergable. They fix 3 separate bugs, not a chain.