fix(kanban): stop miscounting forced kills and init failures as protocol_violation - #22
Merged
Merged
Conversation
Collaborator
Author
|
Linear: HEL-3145 / tech-scope HEL-3146 (CPTC predicted CP=3). Kanban t_5229e0ea. Exact head d5594c000052a9783b20db7762e40c02d197227d. TRC gate + Rhea merge lane — do not self-merge. |
…col_violation Root cause of the t_fe90ab52 50× ~60s crash loop: 1. Non-quiet single-query (`chat -q`) init/run failures printed Goodbye and exited rc=0. Dispatcher reaped that as clean-exit protocol_violation after one dispatch_interval_seconds tick (~60s on default gateway). 2. Kanban SIGTERM handler called os._exit(0), collapsing forced kills. Fix (rebased onto current fork/main; preserves terminal-loop exit helper): - Human -q path: sys.exit(1) when chat() returns None - Kanban SIGTERM: os._exit(128+signum) → 143 - Reaper: forced_signal kind is not protocol_violation Kanban: t_5229e0ea / Linear HEL-3145
SSC-ENG
force-pushed
the
fix/kanban-worker-rc0-protocol-violation-60s
branch
from
July 31, 2026 02:56
4b691bc to
bca834c
Compare
…rced_signal _classify_worker_exit must still return terminal_loop_error for exit 76 (#17) before shell 128+N forced_signal. Restores CI slice failures from tests/hermes_cli/test_kanban_terminal_loop_error.py without losing the SIGTERM/rc=0 protocol_violation distinction.
Collaborator
Author
CEA handoff for TRCExact head: Root cause + fix summarized in PR body + HEL-3145. GATEWAY-VERDICT: CEA=READY_FOR_TRC head=742cce3cad1fcc47b92d783694900308a6356791 Please post: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the P0 kanban reliability crash loop (t_fe90ab52 / t_5229e0ea): workers exit at ~60s with
exit_code=0and the dispatcher recordsprotocol_violationeven when the run never legitimately finished.Root cause (file:line)
Primary — init/run failure exits rc=0 on human
-qpathcli.pynon-quiet single-query path calledcli.chat(...)then returned without checking the result.chat()returnsNone), CLI printsFailed to initialize agent…+Goodbye!and exits 0./Users/danieldezago/.hermes/kanban/logs/t_fe90ab52.logshows 50+ segments of exactly that pattern.dispatch_interval_seconds: 60on the gateway/01-max-headroom profile) → surface duration clusters at exactly ~60s withprotocol_violation: true.Secondary — SIGTERM handler exits 0
cli.py_signal_handler_q(kanban branch) calledos._exit(0)on SIGTERM so forced kills looked identical to clean successful exits.hermes_cli/kanban_db.py_classify_worker_exitonly special-cased 0 / rate-limit / generic nonzero, so kill=0 became protocol_violation.Fix
cli.pyhuman-qpathsys.exit(1)whenchat()returnsNonecli.pykanban SIGTERM handleros._exit(128+signum)(143 for SIGTERM)kanban_db._classify_worker_exitforced_signalkind for shell 128+Ndetect_crashed_workersLegitimate yellow-field clean exits (true narrated stop, rc=0, no terminal tool) still classify as protocol_violation.
Minimal repro (pre-fix)
Tests
test_forced_sigterm_exit_is_not_protocol_violationtest_clean_exit_zero_still_protocol_violationtest_classify_worker_exit_forced_signal_vs_cleantest_sigterm_with_kanban_task_env_terminates_quickly(now asserts rc=143)test_human_single_query_exits_1_when_chat_returns_noneLocal: 8/8 targeted tests green.
Deployment impact
Rollback
Revert this commit. Pre-fix behavior (rc=0 on init fail / SIGTERM) returns; boards can resume internal retry budgets but the crash loop habit returns.
Test plan
Kanban: t_5229e0ea (parent t_fe90ab52)