…t dispatcher auto-promotion as a legitimate unblock (t_e2b1f62a) (#25)
Kanban card: t_e2b1f62a (rhea-ramos) — second, distinct dispatch-routing
defect discovered while verifying PR #23's fix for t_21f59f6d.
Linear issue: none exists for this card (kanban-native dispatcher infra
bug; no user-facing Linear-tracked feature/fix maps to it — same
convention as PR #23/t_21f59f6d).
Root cause (a): block_task()'s unblock-loop breaker incremented
block_recurrences on any same-kind re-block, with no check on WHAT put
the task back in running/ready. A dispatcher-side auto-promotion
(recompute_ready's parents_terminal trigger, or any other non-
unblock_task exit from blocked) looked identical, at this layer, to a
genuine human/cron unblock_task -> worker re-block ping-pong. This
inflated t_342c4c9f's counter to 3 (limit=2) purely from residual churn
caused by the now-fixed t_21f59f6d bug, tripping block_loop_detected and
routing an already-reviewed, correctly-blocked card to triage instead of
blocked.
Fix: block_task now checks the most recent status-exit event kind before
counting a same-block_kind re-block as the same-cause loop signal. Only
an explicit unblock_task call (which emits "unblocked") re-arms the
counter; a dispatcher "promoted" event resets it to a fresh 1.
Root cause (b): decompose_task() (and the gateway's auto-decompose
sweep via list_triage_ids()) treated every triage card identically,
including ones routed there by block_loop_detected specifically to
force a human decision. auto-decomposer picked such a card up, made a
cosmetic title/body edit, and immediately promoted it straight back to
ready via parents_terminal (with REAL, all-done parent ids this time —
a distinct defect from t_21f59f6d) — defeating the loop breaker's whole
purpose within ~90 seconds.
Fix: decompose_task() now checks whether the triage card's most recent
event is block_loop_detected and refuses to specify/promote it (no LLM
call made), leaving it for an explicit human/operator action.
Ask (c) confirmed structurally: the active_pr respawn guard
(_RESPAWN_GUARD_PR_WINDOW = 86400s) is NOT indefinite — without fixes
(a)/(b) the wasted-spawn loop would have resumed after 24h at a slower
cadence, not stayed blocked forever. This makes (a)/(b) the real fix.
## Tests
- test_kanban_blocked_sticky.py: added
test_dispatcher_repromotion_does_not_inflate_block_recurrences
(reproduces the t_342c4c9f loop across 4 dispatcher-style
re-promotion cycles; asserts recurrences stay at 1 and status stays
blocked) and test_genuine_unblock_reblock_loop_still_trips_breaker
(sanity check the original Dale's-Type-1 loop-breaker behavior is
preserved: BLOCK_RECURRENCE_LIMIT=2 still trips on the 2nd genuine
same-cause unblock->reblock).
- test_kanban_decompose.py: added
test_decompose_skips_triage_card_from_block_loop_detected (asserts
ok=False, no LLM call, card stays in triage untouched).
All new/modified tests pass (12/12). Ran full tests/hermes_cli -k kanban
suite on this branch (276 passed, 9 pre-existing failures) and diffed
1:1 against the same suite on unmodified fork/main (273 passed, same 9
failures) — confirmed the 9 failures are pre-existing test-isolation
issues unrelated to this change, and this diff adds exactly 3 new
passing tests with zero regressions.
🤖 Generated with Hermes Agent (rhea-ramos)
Co-authored-by: SSC-ENG <225143396+SSC-ENG@users.noreply.github.com>
Summary
Kanban card: t_21f59f6d — "DISPATCH BUG: blocked cards spuriously re-promoted to ready via empty-parents 'parents_terminal' trigger (t_342c4c9f loop, 100+ wasted spawns)".
Linear issue: none exists for this card (kanban-native dispatcher infra bug; no user-facing Linear-tracked feature/fix maps to it).
Root cause
_record_task_failure's breaker-trip path (hermes_cli/kanban_db.py) persisted the raw per-callfailurescounter onconsecutive_failuresinstead of flooring it at theeffective_limitthat actually tripped the breaker.For
force_trip=Truecallers (protocol-violation streak, systemic crash fingerprint), the streak that trips the breaker is often unrelated to the unifiedconsecutive_failurescolumn. If that column was reset to 0 by a priorunblock_task, the very first force-trip call computesfailures=1even when tripping on, say,failure_limit=3.Storing that raw
1letrecompute_ready— invoked later in the SAME dispatch tick, re-resolving its own (often lower/unrelated)effective_limit— see1 < 2and immediately promote the task straight back toready, undoing the trip within the same tick.Observed in production:
t_d1994b5bhit this 96+ times, feeding a BEL-ESCALATION storm of wasted dispatch/spawn/crash cycles.t_342c4c9fflappedblocked -> ready -> crashedrepeatedly with the identical signature:{"trigger": "parents_terminal", "satisfied_parent_ids": []}on a task with zero parents (a zero-parent task's parent list is vacuously "all done", so anyrecompute_readyre-check trivially satisfies the promotion condition once the under-reported counter clears whatever limit it re-resolves).t_21f59f6d, reproduced it live during investigation:gave_upatfailures=2/effective_limit=2, immediately re-promoted viaparents_terminalwithsatisfied_parent_ids=[].Fix
Persist
max(failures, effective_limit)on the stored counter at trip time. Any later re-check computing the same-or-lower limit now correctly sees the task as still over threshold. A genuinely higher configured limit (e.g. a raisedmax_retries) can still recover the task — unified counter semantics preserved, just no longer allowed to under-report.Tests
New regression test
tests/hermes_cli/test_kanban_blocked_sticky.py::test_force_trip_floors_consecutive_failures_at_effective_limit:failure_limit=3on a freshly-created task (counter starts at 0).consecutive_failuresis floored at 3, not the raw per-call count of 1.recompute_ready(failure_limit=2)call and asserts it does NOT re-promote (would have, pre-fix, since1 < 2).Verified the test fails without the source fix (
assert 1 == 3— got 1) and passes with it.Full kanban test slice (
tests/hermes_cli/ -k kanban): 269 passed / 9 pre-existing failures, identical on the unpatchedfork/mainbaseline (environment-only: missinghttpx2, decompose-message-format drift, lifecycle-hook fixture, write-guard real-root fixture) — zero net-new failures introduced by this change.Not done / explicitly out of scope
_record_task_failure/recompute_readybefore this merge will keep exhibiting the bug until it is restarted/replaced after merge. Whoever merges this should schedule that restart.996ac78c4).Card: t_21f59f6d.