fix(cli): a rate-limited kanban worker was recorded as a protocol violation (exit 75 only existed on the -Q path)

[rodrigogs]

Three cards on a live board sat blocked with worker exited cleanly (rc=0) without calling kanban_complete or kanban_block — protocol violation. Nine runs between them, every worker log ending the same way:

❌ API failed after 3 retries — HTTP 429: The service may be temporarily overloaded, please try again later

The worker did nothing wrong. The provider walled it off, and the exit status said "clean exit" — so the dispatcher, which holds a pid and not a transcript, read the only channel it has and concluded the worker had skipped the protocol. Each one counted a failure until the card was blocked.

The contract already existed, on one of two paths

KANBAN_RATE_LIMIT_EXIT_CODE (75, EX_TEMPFAIL) means "quota wall, not a task error"; the reap classifier maps it to a rate_limited exit and releases the task back to ready without incrementing the failure counter, so a long quota window cannot trip the circuit breaker. run_conversation already returns failure_reason (agent/conversation_loop.py) for exactly this purpose.

That consumer lives in the -Q branch of the one-shot path (_run_quiet_single_query in cli.py). Workers are spawned with -Q only in goal mode (if task.goal_mode in _worker_argv, used by _default_spawn), so an ordinary card — goal_mode=0, the common case — runs the human-facing branch of _run_single_query_mode instead: cli.chat(...), _print_exit_summary(), return. No exit-code logic at all, so the process exits 0. All three blocked cards had goal_mode=0.

What changes

• The decision moves into _kanban_quota_wall_exit_code(result) and both one-shot runners (_run_quiet_single_query and _run_single_query_mode) ask it. It returns the sentinel only for a kanban worker whose turn failed on a quota-wall reason, and None otherwise — leaving each caller's own 0/1 contract untouched, so non-kanban one-shot runs behave exactly as before.
• chat() (in hermes_cli/cli_chat_turn_mixin.py) parks the turn's result on the CLI as _last_turn_result. It returns the response text, and the human path needs failure_reason; the assignment sits where the agent thread's dict is already in hand, beside the existing conversation_history update.
• upstream_rate_limit is deliberately not a quota-wall reason. The tuple is the one the -Q path has always used, and a per-model 429 from an aggregator can be durable for that model — requeueing on it would spin. Widening it is a separate decision with its own risk.

Relationship to #85083

Not a mechanism-level duplicate (no shared files; merges cleanly in either order), though both close the same user-visible bug. #85083 makes _default_spawn always pass -Q, so dispatcher-spawned workers land on the branch that already exits 75. This PR fixes the exit semantics where they are decided, so any kanban worker running under HERMES_KANBAN_TASK without -Q (manual run, other tooling, a future spawn path) also exits 75 instead of 0. Keeping both is harmless.

Verification

8 new tests in tests/cli/test_kanban_worker_quota_wall_exit.py (all pass on the rebased branch; neighbours tests/cli/test_chat_q_exit_clear.py, tests/cli/test_single_query_session_finalize.py, tests/hermes_cli/test_kanban_db.py, tests/hermes_cli/test_kanban_worker_spawn_toolsets.py green; ruff clean on the changed files).

Mutation-verified: 5 mutants, 0 survivors — dropping the human-path call; dropping the _last_turn_result assignment (which would leave the fix in place and inert); dropping the worker-only env guard; dropping the reason check; restoring the inline tuple in the -Q branch. Each is killed by a different assertion.

The negative assertion is the one that matters most: a server_error, timeout, auth_permanent or upstream_rate_limit failure must not be laundered into a throttle, or the breaker stops seeing failures the task itself caused.

Two tests read the source rather than run it. Since the cli.py split, the two one-shot exits live in _run_quiet_single_query and _run_single_query_mode, which still build a whole CLI before reaching them, so there is no honest way to exercise them in-process — and what is worth pinning is structural anyway: that the reasons are named in exactly one place, and that both runners ask the helper (exactly one call each, two module-wide). The producer side of this contract had no test at all before, which is how the asymmetry survived.

[alt-glitch]

This was generated by AI during triage.

Duplicate of #85083. Both repair the non-goal Kanban worker rate-limit path that exits 0 and is then classified as a protocol violation; #85083 is the earlier, smaller spawn-path fix using the existing -Q/EX_TEMPFAIL contract.

[rodrigogs]

For the record — not a duplicate of #85083 at the mechanism level (no shared files; merges cleanly in either order), though both close the same user-visible bug:

• fix(kanban): route worker rate limits through tempfail #85083 (spawn side): _default_spawn now always appends -Q, so every dispatcher-spawned worker lands on the quiet CLI branch that already maps quota walls to KANBAN_RATE_LIMIT_EXIT_CODE (75).
• This PR (CLI side): makes the non--Q branch itself exit 75 on a quota wall (plus captures _last_turn_result so the exit decision has the failure_reason in hand), and adds a dedicated regression test (test_kanban_worker_quota_wall_exit.py).

Difference in coverage: after #85083, workers spawned by the dispatcher are safe by construction, but any kanban worker invoked outside _default_spawn (manual run, other tooling under HERMES_KANBAN_TASK) still exits 0 and reproduces the protocol violation. This PR fixes the exit semantics where they are decided, so the trap cannot reappear on a future spawn path. Keeping both is harmless (zero conflict); if only one should land, this one is the class fix.

[teknium1]

Landed on main at 1b2a75f via #112500, which salvaged #110917 (@dmelkk-secondbrain) — the same fix as this PR: chat -q kanban workers now share the -Q exit contract (0 success, 1 failure, 75 = EX_TEMPFAIL on a rate-limit/billing wall → rate_limited, requeued without a failure tick), gated on HERMES_KANBAN_TASK. Kanban workers on a quota wall are no longer booked as protocol violations.

Credit where due: the earliest PR fixing this exact fall-through was #97623 by @rodrigogs (Aug 29), and this cluster grew to ten PRs before one landed — my sweep should have picked up the earliest, and I have noted the precedence on the merged PR. Closing this as implemented on main; thank you for the fix.

Yours was first. Sorry the salvage picked a later sibling; the landed diff is the same mechanism as yours (shared exit mapping + _last_turn_result recorded at settle).