🐛 fix(kanban): restore crashed reviewer to review lane, not build lane - #17
Conversation
cwest
left a comment
There was a problem hiding this comment.
The fix itself holds up. I read the source at the head SHA and confirmed the premise: check_respawn_guard gates both recent_success and active_pr on if not is_review (where is_review keys off status == 'review'), so restoring a crashed reviewer to review instead of ready is exactly what frees those guards. The crash path calls _record_task_failure with release_claim=False, which takes the branch whose WHERE-IN you widened to include 'review', so a repeatedly-crashing reviewer still trips the breaker to blocked rather than looping. Reading source_status off the run's claimed event is the right durable signal; no new column. The CAS guard on status='running' and the crash bookkeeping are untouched.
Verified the tests independently: reverted only the production hunk to the parent commit and the new tests went 3 failed / 1 passed (the build-to-ready case passing as designed); restored, and the full file runs 234 passed. ruff clean. Commit is signed and follows the convention.
One thing to settle before this lands, on the PATCHES.md row.
A card in `review` is claimed by `claim_review_task`, which CAS-transitions
`review -> running` and stamps the run's `claimed` event with
`source_status: "review"`. While the reviewer works, the row status is
`running` — indistinguishable from a build run. When that reviewer worker
crashed, `detect_crashed_workers` took the running-crash path and reset the
card to `ready`, dropping the review lane:
1. the normal `ready` dispatch re-ran the implementer instead of
respawning the reviewer — the PR under review silently fell back into
the build lane; and
2. `check_respawn_guard` recomputed `is_review = False`, re-tripping the
`recent_success` and `active_pr` guards (the original build run is a
recent `completed` run that left a PR-URL comment), deferring the
respawn for the full guard window.
A prior fix only cleared a stale claim on a card parked in a NON-running
lane; it did not cover a reviewer that dies while actively `running`.
Fix: read the durable `source_status: "review"` signal off the crashed
run's `claimed` event (scoped to `current_run_id`; no new schema column)
and, on a genuine crash, restore the card to `review` instead of `ready`.
The claim is still cleared, the UPDATE is still CAS-guarded on
`status = 'running'`, and the crash event / run outcome / failure counter /
circuit breaker bookkeeping is unchanged — only the lane a crash returns to
changes. To stop a flaky reviewer looping forever in `review`, the
crash-path breaker-trip UPDATE in `_record_task_failure` now matches
`status IN ('ready','running','review')`, so a repeatedly-crashing reviewer
still trips to `blocked` via the normal failure-count path. Rate-limited
and protocol-violation sub-cases are untouched.
Adds 4 regression tests (crashed reviewer -> review; crashed build run ->
ready unchanged; repeatedly-crashing reviewer -> blocked; restored review
card free of the recent_success/active_pr guards) and a PATCHES.md row.
7cc4fae to
355693e
Compare
cwest
left a comment
There was a problem hiding this comment.
The PATCHES.md bucket is reclassified to permanent-local and the row now reads consistently with it (no upstream PR, never auto-retires, not surfaced upstream). That was the only open item; the kanban_db.py fix and its four regression tests are unchanged from the prior pass and verify clean.
Re-ran at this head in a throwaway clone: 234 tests pass (the 4 new crash-recovery cases included), ruff clean on hermes_cli/kanban_db.py. The merge-readiness picture is green end to end — mergeable, all required checks succeeded, no unresolved threads.
No changes needed. Ready to merge.
…wns through chokepoint (NousResearch#53829) Follow-up to NousResearch#53791 addressing review feedback: the footgun checker treated capture_output=/stdout=/stderr=/check_output as proof a subprocess can't pop a Windows console. That invariant is false — stream redirection controls where a child's output goes, not whether a console is allocated. From a console-less parent (Desktop/Electron, pythonw.exe, detached gateway/cron) a console-subsystem child still flashes a window even when fully captured. - check-windows-footguns.py: capture/redirect/check_output is no longer a blanket safe-pass. Added _WINDOWS_FLASHING_PROGRAMS (git/gh/npm/node/python/uv/ffmpeg/ docker/powershell/…); calls to those are flagged even when captured. Non-flashing programs keep the capture exemption (no 271-site noise). _subprocess_compat.run/ popen calls are inherently safe (wrapper injects CREATE_NO_WINDOW). - Routed the 35 genuine flashing git/gh/npm/uv/ffmpeg/docker spawns through the _subprocess_compat.run/popen chokepoint (Brooklyn's wrapper from NousResearch#53810) — the durable fix, not per-site annotations. cmd.exe /c start stays # ok (intentional). - Updated tests + CONTRIBUTING.md rule #17 to the corrected invariant.
…t pattern disconnect() reads self._post_connect_task, but several tests build a bare TelegramAdapter via object.__new__() without calling __init__ (which sets the attr). Use getattr(..., None) so disconnect() works on those instances too (pitfall #17).
… fail-closed flip Follow-up to the salvaged fail-closed defaults. The own-policy default flip (open -> pairing) and the email dispatch-level deny broke sibling tests across the suite that relied on the old fail-open behavior: - test_email.py: dispatch-mechanics tests now opt into EMAIL_ALLOW_ALL_USERS (they test formatting/attachments/threading, not authz); the two auth contract tests are rewritten to assert the new fail-closed behavior (no allowlist + no allow-all => sender dropped at the adapter). - test_whatsapp_cloud.py / test_whatsapp_formatting.py / test_whatsapp_from_owner.py: autouse fixture opts into WHATSAPP_ALLOW_ALL_USERS so dm_policy: open dispatch-mechanics tests still flow (open now requires an explicit allow-all opt-in, SECURITY.md 2.6). - _adapter_for_source: use getattr for source.platform/profile so bare SimpleNamespace test fixtures without .profile don't crash the busy/queue ingress path (AGENTS.md pitfall #17). Full tests/gateway/ + yuanbao pipeline: 8555 passed, 0 failed.
A card in review is claimed by claim_review_task, which CAS-transitions review->running, so while the reviewer works the row status is running, indistinguishable from a build run. When detect_crashed_workers reaped such a crash it ran SET status='ready', losing the review lane (the implementer re-ran instead of the reviewer respawning). Reads the durable source_status='review' signal off the crashed run's claimed event and restores the card to review instead of ready; the breaker-trip WHERE-IN widens to include 'review' so a repeatedly-crashing reviewer still trips to blocked via the failure-count path. upstream-pending: fork PR #17
…_adapter_for_source The routing sweep sends these paths through _adapter_for_source, which reads source.profile. A bare MagicMock auto-attribute is truthy, so the fixtures looked like stamped secondary profiles and hit the new fail-closed branch. Real SessionSource.profile is None or str (AGENTS.md pitfall #17).
…t doubles The dedup-reset calls assumed a full AIAgent; gateway/loop test doubles built via object.__new__ lack _clear_context_overflow_warn and crashed in build_turn_context (caught by test_api_content_sidecar on CI slice 3). getattr-guard all four call sites per the established test-double pitfall pattern (AGENTS.md #17).
A card in review is claimed by claim_review_task, which CAS-transitions review->running, so while the reviewer works the row status is running, indistinguishable from a build run. When detect_crashed_workers reaped such a crash it ran SET status='ready', losing the review lane (the implementer re-ran instead of the reviewer respawning). Reads the durable source_status='review' signal off the crashed run's claimed event and restores the card to review instead of ready; the breaker-trip WHERE-IN widens to include 'review' so a repeatedly-crashing reviewer still trips to blocked via the failure-count path. upstream-pending: fork PR #17 (cherry picked from commit 4b515a8)
What & why
A card moved to
review(a worker opened a PR and parked it) is claimed byclaim_review_task, which CAS-transitionsreview → runningand records aclaimedevent withsource_status: "review"in its payload. While the reviewer works, the card's row status isrunning— indistinguishable at the row level from a build run.If that reviewer worker crashes (signal / OOM / host hiccup),
detect_crashed_workersfinds the card inrunningand takes the running-crash path, which reset the card toready. That dropped the review lane entirely, with two consequences on the next dispatch tick:ready, notreview, so the review-column dispatch (WHERE status = 'review') does not pick it. The normalreadydispatch claims it as a fresh build run — re-running the implementer instead of the reviewer. The PR under review silently falls out of the review lane back into the build lane.check_respawn_guardrecomputesis_review = (status == "review")becomesFalse, so therecent_successguard (the original build run is a recentcompletedrun) andactive_prguard (the build handoff left a PR-URL comment) both fire, deferring respawn for the full guard window.A prior change (the non-running-lane stale-claim clear) only handled a dead worker on a card parked in a non-
runninglane; it did not cover a reviewer that dies while activelyrunning. This is that gap.The fix
In
hermes_cli/kanban_db.py:_crashed_run_was_review(conn, task_id, current_run_id)which reads thesource_status: "review"field off the crashed run'sclaimedevent payload, scoped tocurrent_run_id(falls back to the task's latestclaimedevent). This reuses the existing signalclaim_review_taskalready writes — no new column.was_review_runand set the reset UPDATE's status to'review'when true (else'ready'). The UPDATE still clearsclaim_lock/claim_expires/worker_pidand is still CAS-guarded onstatus = 'running'. The crash event, run outcome, failure counter, and circuit breaker bookkeeping are unchanged — only the lane a crash returns to changes._record_task_failurewidened itsWHERE ... status IN (...)from('ready','running')to('ready','running','review'). Without this, a card restored toreviewwould never flip toblockedand a flaky reviewer would loop forever. Now a repeatedly-crashing reviewer still trips toblockedvia the normal failure-count path.Tests — RED to GREEN
Added 4 regression tests in
tests/hermes_cli/test_kanban_db.py:test_crashed_reviewer_run_restored_to_review_lane— reviewer run (claimed viaclaim_review_task) whose PID is dead → card back inreview, claim cleared.test_crashed_build_run_restored_to_ready_lane— build run (claimed viaclaim_task) whose PID is dead → card back inready(no regression).test_crashed_reviewer_still_trips_breaker_to_blocked— reviewer crashing up to the breaker limit →blocked(no infinite review-lane loop).test_respawn_guard_frees_restored_review_card— end-to-end: restoredreviewcard returnsNonefromcheck_respawn_guardfor therecent_success/active_prcases (reviewer is free to respawn).RED (4 tests against the pristine
kanban_db.py, implementation stashed):(Test 2 passes on pristine code by design — it asserts the unchanged build-crash behavior.)
GREEN (with the fix):
Verification
pytest tests/hermes_cli/test_kanban_db.py→ 234 passed (230 baseline + 4 new).test_kanban_db.py,test_kanban_core_functionality.py,test_signal_handler_kanban_worker.py,test_kanban_cli.py,test_kanban_diagnostics.py,test_kanban_goal_mode.py,test_kanban_boards.py,test_kanban_db_init.py,test_kanban_tools.py) → 671 passed, 1 skipped, 1 pre-existing failure (test_sigterm_with_kanban_task_env_terminates_quickly, a SIGTERM timing test that fails identically on the pristine base3a065375d— unrelated to this change, verified by stashing).ruff check hermes_cli/kanban_db.py→ All checks passed!PATCHES.md
Added an
upstream-pendingrow (no upstream PR — fork-internal review-lane semantics, not surfaced to NousResearch) documenting the carry and its retire trigger.Notes
cwest/integration(3a065375d). Branched off integration, notmain.hermes_cli/kanban_db.py.