Skip to content

fix(cli): infer quota wall for kanban exit code when failure_reason is missing (fixes #101800) - #102318

Closed
aniruddhaadak80 wants to merge 2 commits into
NousResearch:mainfrom
aniruddhaadak80:fix/kanban-rate-limit-exit-sentinel-101800
Closed

aniruddhaadak80 wants to merge 2 commits into
NousResearch:mainfrom
aniruddhaadak80:fix/kanban-rate-limit-exit-sentinel-101800

Conversation

@aniruddhaadak80

Copy link
Copy Markdown

What does this PR do?

Kanban workers that die purely on provider quota exhaustion (429 / billing) must exit with the EX_TEMPFAIL sentinel (KANBAN_RATE_LIMIT_EXIT_CODE = 75) so the dispatcher requeues to ready as rate_limited without incrementing the failure counter. Instead, a 13-minute 429 window produced 42 crashed runs and 0 rate_limited, and the misleading "protocol violation" streaks pushed the board toward the circuit breaker.

The exit sentinel in cli.py:22376 only fires when result["failure_reason"] is exactly "rate_limit"/"billing". When the failing turn never stamps that key (quota exhaustion surfacing through a path that carries only prose error), the worker falls through to generic exit 1, which the reaper reads as "exited cleanly without kanban_complete/kanban_block" and redispatches into the same quota window.

This PR adds a narrow consumer-side fallback, kanban-workers-only: if the run failed under HERMES_KANBAN_TASK with no usable failure_reason, scan the result's error text for quota signals (rate limit, 429, quota, usage limit, billing, credits exhausted, insufficient, …) and treat a hit as rate_limit for exit-code purposes. An explicit failure_reason always wins; non-kanban runs keep the plain 0/1 contract. Worst case a genuinely-broken task mentioning quota exits 75 (requeued, retried) instead of 1 — it delays failure accounting but never loses work, versus today's crashloop that wedges the board.

Related Issue

Fixes #101800

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

How to Test

  1. Repro: fake a kanban worker result {"failed": True, "error": "Rate limited after 3 retries (429)"} with no failure_reason under HERMES_KANBAN_TASK=1 → before: exit 1 (crashed); after: exit 75 (rate_limited, task back to ready, failure counter untouched).
  2. Explicit failure_reason: "rate_limit"/"billing" still exits 75; explicit other reasons and non-quota errors still exit 1; non-kanban runs unaffected.
  3. CI: Python tests / Run tests and Python lints pass.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass (CI: Python tests / Run tests pass)
  • I've added tests for my changes (covered by existing kanban/cli exit-path tests; verified via manual repro above)
  • I've tested on my platform: Windows 11, Python 3.14

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A (pure string logic)
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A

For New Skills

  • N/A

Screenshots / Logs

Before: 42 crashed / 0 rate_limited across a 429 window, protocol-violation streaks accrue. After: quota-text failures exit 75 → rate_limited requeues with no failure-counter increment.

@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management area/usage-cost Token accounting, usage reporting, billing, cost tracking labels Sep 3, 2026
@alt-glitch

Copy link
Copy Markdown

This was generated by AI during triage.

Related: this is the 4th open PR in the kanban rate-limit exit-75 cluster for #101800 — #98969 (preserves the structured rate_limit/billing result through chat -q, earliest), #85083 (spawn-path -Q/EX_TEMPFAIL routing), #100771 and #97623 (already marked duplicates). This one takes a different approach (keyword-scan of the prose error text when failure_reason is missing). Maintainer should pick one mechanism; the keyword heuristic is a fallback that only matters if #98969's structured-reason preservation is not merged.

…it-exit-sentinel-101800

# Conflicts:
#	cli.py
@teknium1

Copy link
Copy Markdown
Collaborator

Landed on main at 1b2a75f via #112500, which salvaged #110917 (@dmelkk-secondbrain) — the same fix as this PR: chat -q kanban workers now share the -Q exit contract (0 success, 1 failure, 75 = EX_TEMPFAIL on a rate-limit/billing wall → rate_limited, requeued without a failure tick), gated on HERMES_KANBAN_TASK. Kanban workers on a quota wall are no longer booked as protocol violations.

Credit where due: the earliest PR fixing this exact fall-through was #97623 by @rodrigogs (Aug 29), and this cluster grew to ten PRs before one landed — my sweep should have picked up the earliest, and I have noted the precedence on the merged PR. Closing this as implemented on main; thank you for the fix.

The failure_reason-inference half is unnecessary on current main: max_retries_exhausted_result writes failure_reason since #82154.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/usage-cost Token accounting, usage reporting, billing, cost tracking comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rate-limit exit sentinel is unreachable: quota exhaustion is misclassified as a clean-exit protocol violation and crashloops the board

3 participants