Skip to content

fix(kanban): stop miscounting forced kills and init failures as protocol_violation - #22

Merged
SSC-ENG merged 2 commits into
mainfrom
fix/kanban-worker-rc0-protocol-violation-60s
Jul 31, 2026
Merged

fix(kanban): stop miscounting forced kills and init failures as protocol_violation#22
SSC-ENG merged 2 commits into
mainfrom
fix/kanban-worker-rc0-protocol-violation-60s

Conversation

@SSC-ENG

@SSC-ENG SSC-ENG commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes the P0 kanban reliability crash loop (t_fe90ab52 / t_5229e0ea): workers exit at ~60s with exit_code=0 and the dispatcher records protocol_violation even when the run never legitimately finished.

Root cause (file:line)

  1. Primary — init/run failure exits rc=0 on human -q path

    • cli.py non-quiet single-query path called cli.chat(...) then returned without checking the result.
    • On init failure (chat() returns None), CLI prints Failed to initialize agent… + Goodbye! and exits 0.
    • Live evidence: /Users/danieldezago/.hermes/kanban/logs/t_fe90ab52.log shows 50+ segments of exactly that pattern.
    • Dispatcher only notices after the next tick (dispatch_interval_seconds: 60 on the gateway/01-max-headroom profile) → surface duration clusters at exactly ~60s with protocol_violation: true.
  2. Secondary — SIGTERM handler exits 0

    • cli.py _signal_handler_q (kanban branch) called os._exit(0) on SIGTERM so forced kills looked identical to clean successful exits.
    • hermes_cli/kanban_db.py _classify_worker_exit only special-cased 0 / rate-limit / generic nonzero, so kill=0 became protocol_violation.

Fix

Change Behavior
cli.py human -q path sys.exit(1) when chat() returns None
cli.py kanban SIGTERM handler os._exit(128+signum) (143 for SIGTERM)
kanban_db._classify_worker_exit new forced_signal kind for shell 128+N
detect_crashed_workers forced kills → crash/retry, not protocol_violation

Legitimate yellow-field clean exits (true narrated stop, rc=0, no terminal tool) still classify as protocol_violation.

Minimal repro (pre-fix)

# Worker fails agent init (or otherwise chat()→None) and exits 0:
hermes -p paul-park --cli chat -q "work kanban task <id>"
echo exit:$?   # was 0 after Goodbye; now 1 after init fail

# SIGTERM during a real worker used to exit 0:
kill -TERM <worker-pid>
# wait → dispatcher sees protocol_violation. Now 143 → forced_signal crash.

Tests

  • test_forced_sigterm_exit_is_not_protocol_violation
  • test_clean_exit_zero_still_protocol_violation
  • test_classify_worker_exit_forced_signal_vs_clean
  • test_sigterm_with_kanban_task_env_terminates_quickly (now asserts rc=143)
  • test_human_single_query_exits_1_when_chat_returns_none

Local: 8/8 targeted tests green.

Deployment impact

  • Runtime: editable install + gateway restart required so live dispatcher/worker pick up code.
  • No DB migration. No prod deploy from this PR alone.
  • Does not attempt merge; TRC + Rhea/Ellis merge lane.

Rollback

Revert this commit. Pre-fix behavior (rc=0 on init fail / SIGTERM) returns; boards can resume internal retry budgets but the crash loop habit returns.

Test plan

  • Unit/integration: forced_signal vs clean_exit distinction
  • SIGTERM synthetic worker exits 143 quickly
  • Single-query main exits 1 on chat()-None
  • CI green on exact head
  • TRC (tessa-cole) PASS on exact head
  • After gateway restart: spot-check no new protocol_violation clusters at exactly 60s from init-fail

Kanban: t_5229e0ea (parent t_fe90ab52)

@SSC-ENG

SSC-ENG commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator Author

Linear: HEL-3145 / tech-scope HEL-3146 (CPTC predicted CP=3). Kanban t_5229e0ea. Exact head d5594c000052a9783b20db7762e40c02d197227d. TRC gate + Rhea merge lane — do not self-merge.

…col_violation

Root cause of the t_fe90ab52 50× ~60s crash loop:
1. Non-quiet single-query (`chat -q`) init/run failures printed Goodbye
   and exited rc=0. Dispatcher reaped that as clean-exit protocol_violation
   after one dispatch_interval_seconds tick (~60s on default gateway).
2. Kanban SIGTERM handler called os._exit(0), collapsing forced kills.

Fix (rebased onto current fork/main; preserves terminal-loop exit helper):
- Human -q path: sys.exit(1) when chat() returns None
- Kanban SIGTERM: os._exit(128+signum) → 143
- Reaper: forced_signal kind is not protocol_violation

Kanban: t_5229e0ea / Linear HEL-3145
@SSC-ENG
SSC-ENG force-pushed the fix/kanban-worker-rc0-protocol-violation-60s branch from 4b691bc to bca834c Compare July 31, 2026 02:56
…rced_signal

_classify_worker_exit must still return terminal_loop_error for exit 76
(#17) before shell 128+N forced_signal. Restores CI slice failures from
tests/hermes_cli/test_kanban_terminal_loop_error.py without losing the
SIGTERM/rc=0 protocol_violation distinction.
@SSC-ENG

SSC-ENG commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator Author

CEA handoff for TRC

Exact head: 742cce3cad1fcc47b92d783694900308a6356791
mergeable: MERGEABLE
CI run 30600563940: all Python test slices + lints green; 0 failures.
Desktop E2E still may be in flight (Python-only change).

Root cause + fix summarized in PR body + HEL-3145.

GATEWAY-VERDICT: CEA=READY_FOR_TRC head=742cce3cad1fcc47b92d783694900308a6356791

Please post: GATEWAY-VERDICT: TRC=PASS head=742cce3cad1fcc47b92d783694900308a6356791 when review passes.

@SSC-ENG
SSC-ENG merged commit 996ac78 into main Jul 31, 2026
37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant