Repository navigation
fix(cli): infer quota wall for kanban exit code when failure_reason is missing (fixes #101800) - #102318
fix(cli): infer quota wall for kanban exit code when failure_reason is missing (fixes #101800)#102318aniruddhaadak80 wants to merge 2 commits into
Conversation
Related: this is the 4th open PR in the kanban rate-limit exit-75 cluster for #101800 — #98969 (preserves the structured |
…it-exit-sentinel-101800 # Conflicts: # cli.py
|
Landed on main at 1b2a75f via #112500, which salvaged #110917 (@dmelkk-secondbrain) — the same fix as this PR: Credit where due: the earliest PR fixing this exact fall-through was #97623 by @rodrigogs (Aug 29), and this cluster grew to ten PRs before one landed — my sweep should have picked up the earliest, and I have noted the precedence on the merged PR. Closing this as implemented on main; thank you for the fix. The |
What does this PR do?
Kanban workers that die purely on provider quota exhaustion (429 / billing) must exit with the
EX_TEMPFAILsentinel (KANBAN_RATE_LIMIT_EXIT_CODE = 75) so the dispatcher requeues toreadyasrate_limitedwithout incrementing the failure counter. Instead, a 13-minute 429 window produced 42crashedruns and 0rate_limited, and the misleading "protocol violation" streaks pushed the board toward the circuit breaker.The exit sentinel in
cli.py:22376only fires whenresult["failure_reason"]is exactly"rate_limit"/"billing". When the failing turn never stamps that key (quota exhaustion surfacing through a path that carries only proseerror), the worker falls through to generic exit 1, which the reaper reads as "exited cleanly without kanban_complete/kanban_block" and redispatches into the same quota window.This PR adds a narrow consumer-side fallback, kanban-workers-only: if the run failed under
HERMES_KANBAN_TASKwith no usablefailure_reason, scan the result'serrortext for quota signals (rate limit,429,quota,usage limit,billing,credits exhausted,insufficient, …) and treat a hit asrate_limitfor exit-code purposes. An explicitfailure_reasonalways wins; non-kanban runs keep the plain 0/1 contract. Worst case a genuinely-broken task mentioning quota exits 75 (requeued, retried) instead of 1 — it delays failure accounting but never loses work, versus today's crashloop that wedges the board.Related Issue
Fixes #101800
Type of Change
Changes Made
cli.py:22376— readfailure_reasoninto_failure_reason; when missing/unusable underHERMES_KANBAN_TASK, infer fromresult["error"]quota keywords before the existingin ("rate_limit", "billing")gate. Comment references Rate-limit exit sentinel is unreachable: quota exhaustion is misclassified as a clean-exit protocol violation and crashloops the board #101800.How to Test
{"failed": True, "error": "Rate limited after 3 retries (429)"}with nofailure_reasonunderHERMES_KANBAN_TASK=1→ before: exit 1 (crashed); after: exit 75 (rate_limited, task back toready, failure counter untouched).failure_reason: "rate_limit"/"billing"still exits 75; explicit other reasons and non-quota errors still exit 1; non-kanban runs unaffected.Python tests / Run testsandPython lintspass.Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests pass (CI:Python tests / Run testspass)Documentation & Housekeeping
docs/, docstrings) — or N/Acli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/AFor New Skills
Screenshots / Logs
Before: 42
crashed/ 0rate_limitedacross a 429 window, protocol-violation streaks accrue. After: quota-text failures exit 75 →rate_limitedrequeues with no failure-counter increment.