fix(kanban): circuit breaker halts saga after repeated identical failure signature (BUILD-261)

[nlachica]

Incident

On 2026-07-09 a releaser kanban card (t_90949a61) merged 8 PRs across two batches — 4 (aldnoah NousResearch#465-468) then 4 more (NousResearch#469-472) — whose post-merge "Master Release" workflow failed with the byte-identical error signature every single time:

##[error]smoke check failed: checkout (/checkout) returned 500 for access_gate=public

Each remediation was reviewed, verified, and merged, and the identical symptom recurred immediately after — nothing detected the non-convergence. The existing consecutive_failures circuit breaker never tripped because every attempt "succeeded" from the worker's point of view (a merge is a success; the counter resets to 0 on completion). check_respawn_guard fired 761x in 24h, but it only rate-limits how often a respawn is attempted — it never inspects whether repeated attempts are actually converging.

Fix

A second, content-aware circuit breaker in hermes_cli/kanban_db.py:

• normalize_failure_signature(text) — reduces a (possibly multi-line) failure log to a stable signature: the first ##[error]... line (or the final non-blank line if there's no such marker), with timestamps/run-ids/SHAs stripped and whitespace collapsed. Two CI runs of the identical underlying failure normalize to the identical signature even though timestamps/run-ids differ.
• block_task()'s worker-self-report path (kind=needs_input|capability|transient — a worker calling kanban_block with a reason it can't resolve) now also records a failure_signature task_event from the block reason.
  • Deliberately not hooked into the crash/timeout/spawn-failure funnel (_record_task_failure): that path already has its own independently-configurable failure_limit breaker (DEFAULT_FAILURE_LIMIT=2), and layering a second breaker with a different default threshold onto the identical event stream would silently override an operator's more lenient failure_limit for any task whose infra errors happen to repeat verbatim byte-for-byte (very common — e.g. "no PATH", workspace-resolution errors). Caught this exact regression while implementing (see test_record_task_failure_does_not_emit_failure_signature_event) — it broke 4 pre-existing tests in test_kanban_core_functionality.py before the fix was scoped correctly.
• check_failure_signature_breaker(conn, task_id, threshold=None) — compares the last N (default 2, configurable via kanban.failure_signature_threshold config or HERMES_KANBAN_FAILURE_SIGNATURE_THRESHOLD env, mirroring the failure_limit pattern) recorded signatures for the task and its linked remediation children (task_links, covering the "saga" shape where a parent card fans out a fresh child task per attempt instead of looping one task_id). If they're all identical, the signal is "not converging."
• Dispatch hook: _dispatch_once_locked's ready-task loop checks the breaker before check_respawn_guard (whose defer is only a one-tick skip). If tripped, the task is blocked (kind=needs_input) instead of respawned, with a comment describing the trip (both signatures + run refs) via _trip_failure_signature_breaker.
• Alerting: blocking reuses block_task's existing blocked task_event — the exact event kind the gateway's _kanban_notifier_watcher already polls and delivers to subscribers on whatever platform they're on (Telegram included). No new notify channel was added, per the spec.
• Distinct signatures (e.g. a different downstream check failing) never trip it, and a threshold of 1 is deliberately rejected (falls back to the default) since a single occurrence has no repetition to detect.

Tests (TDD — written first)

All in tests/hermes_cli/test_kanban_db.py:

• TestNormalizeFailureSignature — 9 unit tests including the literal incident sample, timestamp/run-id/SHA stripping, whitespace collapsing, final-line fallback, empty/None input.
• Trip-at-2-identical, no-trip-on-distinct, no-trip-with-only-1-failure, only-considers-most-recent-window, explicit/env-configurable threshold (rejects <2).
• Remediation-children inclusion (saga shape) — trips on identical child signatures, no-trip on distinct child signatures.
• Full dispatch_once integration: circuit breaker blocks + comments + prevents respawn on identical signatures; spawns normally on distinct signatures; honors signature_repeat_threshold kwarg.
• block_task integration: records signature for needs_input, does NOT record for dependency blocks (healthy backpressure, not a failure) or empty reason text.
• Regression guard: _record_task_failure does NOT emit failure_signature (documents/pins the scoping decision above).

Test results

uv run pytest tests/hermes_cli/test_kanban_db.py -q
→ 248 passed, 1 pre-existing failure (test_resolve_hermes_argv_module_actually_runs —
  environment-only: missing `dotenv` for a subprocess spawn using a hardcoded system
  python; fails identically on a clean checkout, confirmed via git stash)

uv run pytest tests/hermes_cli/ -k kanban -q
→ 775 passed, 19 failed (all 19 confirmed pre-existing/environment-specific —
  identical failure set on a clean checkout before this change), 1 skipped

Also ran clean: tests/gateway/test_kanban_watchers_mixin.py, test_kanban_notifier.py,
test_kanban_notifier_watcher_dispatch_gate.py, test_kanban_auto_decompose_live.py,
tests/tools/test_kanban_tools.py, test_kanban_redaction.py,
tests/plugins/test_kanban_dashboard_plugin.py, test_kanban_worker_runs.py,
test_kanban_attachments.py, tests/hermes_cli/test_kanban_cli.py,
test_kanban_cli_dispatch_passthrough.py, test_kanban_block_kinds.py,
test_kanban_blocked_sticky.py — all pass (one pre-existing notifier truncation
failure confirmed unrelated).

ruff check: all clean on touched files.

Files touched

• hermes_cli/kanban_db.py — normalize_failure_signature, check_failure_signature_breaker, _trip_failure_signature_breaker, _record_failure_signature, _resolve_failure_signature_repeat_threshold, _recent_failure_signatures, block_task hook, dispatch_once/_dispatch_once_locked new signature_repeat_threshold kwarg + ready-loop hook, DispatchResult.circuit_breaker_tripped.
• gateway/kanban_watchers.py — reads kanban.failure_signature_threshold config, threads it into dispatch_once.
• hermes_cli/kanban.py — same config threading for the hermes kanban dispatch CLI path.
• tests/hermes_cli/test_kanban_db.py — new test coverage described above.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FoCncveCNmgqD39G2geeF2