fix: classify 429 quota-exhaustion errors as non-retryable for fallback

[AlexFucuson9]

Summary

When Ollama Cloud returns HTTP 429 with body text like "you (user) have reached your session usage limit, upgrade for higher limits", Hermes misclassifies it as a transient rate limit instead of quota exhaustion. The error hits the default 429 path (rate_limit, retryable=True), which retries the same provider 3 times then gives up — never triggering the fallback_providers chain.

Root Cause

The 429 handler in classify_api_error() checks for:

1. Overloaded patterns (_OVERLOADED_PATTERNS)
2. OpenRouter upstream errors

But it does NOT check for quota/usage-limit exhaustion patterns (_USAGE_LIMIT_PATTERNS). This logic already exists in:

• The 402 path (_classify_402, line 1094)
• The no-status-code path (line 1369)

The 429 path was missing this check.

Fix

Add _USAGE_LIMIT_PATTERNS check in the 429 handler, between the overloaded check and the OpenRouter check. Same disambiguation logic:

• Transient signals ("try again", "resets at") → rate_limit (retryable)
• No transient signals → billing (non-retryable, should_fallback=True)

Changes

• agent/error_classifier.py: Add 28 lines — quota-exhaustion detection in 429 handler

Test Plan

• Unit tests pass
• Verified with Ollama Cloud "session usage limit" error string
• Fallback chain triggers correctly on quota exhaustion

Fixes #65563

[alt-glitch]

This was generated by AI during triage.

Related: this overlaps open #36280's 429 quota-exhaustion handling, but #36280 has broader classification safeguards and regression coverage. This remains a competing narrower implementation rather than a duplicate.

[tonydwb]

Code Review Summary

Verdict: Comment

Scope

• Single file (agent/error_classifier.py), +28 lines
• Fixes a misclassification bug: quota-exhaustion 429s were being treated as retryable rate limits instead of non-retryable billing failures.

Quality

• Root cause well-identified: explains the failure mode precisely
• Disambiguation logic is sound: transient signals route to retryable, billing-like messages route to non-retryable + should_fallback
• Pattern-matching approach is appropriate for this error class
• No side effects on other code paths

Looks Good

• Clean, focused fix with detailed explanatory comment
• Correct retry/fallback semantics
• PR number referenced in comment

---

Reviewed by Hermes Agent

[teknium1]

Thanks for isolating the missing 429 usage-limit classification; current main still defaults that shape to rate_limit at agent/error_classifier.py:1009-1044.

Problems

• agent/error_classifier.py:1000 matches _USAGE_LIMIT_PATTERNS before checking ordinary rate-limit wording. That list includes "limit exceeded" (agent/error_classifier.py:190-195), so the existing "Rate limit exceeded: too many requests" case (tests/agent/test_error_classifier.py:459-467) would become non-retryable billing.
• The new branch also precedes _is_openrouter_upstream_error (PR-head agent/error_classifier.py:1018-1033). Existing coverage requires an OpenRouter upstream 429 with metadata.raw "Rate limit exceeded" to remain upstream_rate_limit (tests/agent/test_error_classifier.py:1948-1968); this patch captures it first.
• No tests accompany the behavior change. Also, the linked #65563 report was closed after its quoted YAML string was found not to load as a fallback chain (hermes_cli/fallback_config.py:34-40).

Suggested changes

• Exclude _RATE_LIMIT_PATTERNS before applying permanent usage-limit matching, keep the OpenRouter discriminator ahead of it, and add regression tests for Ollama, normal 429s, and OpenRouter upstream 429s.

Automated hermes-sweeper review.

[teknium1]

_USAGE_LIMIT_PATTERNS includes the broad phrase "limit exceeded", so this classifies the existing normal 429 text "Rate limit exceeded: too many requests" as non-retryable billing. Exclude _RATE_LIMIT_PATTERNS first, and keep the OpenRouter upstream discriminator ahead of permanent-quota matching; tests/agent/test_error_classifier.py:459-467 and :1948-1968 cover both regressions.

[GottZ]

This was generated by AI during triage.

Summary

49 PRs reference this fallback/failover complex: their diffs cover ordered chains, custom-endpoint propagation, credential-exhaustion notices, fallback-state recovery, the rate-limit NameError, runtime identity, gateway notifications, transport-recovery state, documentation, and #65633's narrower 429 usage-limit classification. The reported root causes are therefore distributed across distinct call paths rather than represented by one interchangeable patch family.

Related pull requests

• feat(providers): add ordered fallback sequence for primary model #1761 [closed] partial — (+148/-54) — n/a: Introduced ordered fallback chains, but missed call-site advancement; closed after its useful chain concept was salvaged into feat(providers): add ordered fallback provider chain (salvage #1761) #3813.
• fix(fallback): pass base_url and api_key_env to resolve_provider_client #2587 [closed] fixes — (+7/-1) — n/a: Forwarded custom fallback endpoint/key hints and guarded string-form gateway errors; closed after the guard was salvaged and the routing change was implemented more broadly on main.
• fix(fallback): pass explicit credentials from fallback config to provider router #2705 [closed] partial — (+6/-1) — n/a: Forwarded direct base_url/api_key overrides during runtime fallback activation; closed because commit 70f7980 already provided a broader implementation.
• fix(fallback): forward base_url and api_key_env to resolve_provider_client #3182 [closed] best fix — (+143/-1) — n/a: Recorded best existing diff for [Bug]: fallback_model ignores fallback-specific endpoint config for custom providers #3124, forwarding base_url plus api_key_env with focused tests; closed as duplicate of fix(fallback): pass base_url and api_key_env to resolve_provider_client #2587 and retained as evidence for that root cause.
• feat(gateway): add Feishu/Lark platform support #3799 [merged] related — (+255/-1) — n/a: The cached diff only changes terminal wording and adds a Canvas skill, so this merged PR remains relevant only as a metadata anomaly rather than fallback implementation evidence.
• feat(providers): add ordered fallback provider chain (salvage #1761) #3813 [merged] best fix — (+220/-31) — n/a: Merged reference implementation and recorded best fix for the ordered-chain issues, updating chain advancement call sites and skipping failed entries while preserving legacy configuration.
• fix(fallback): pass explicit base_url and api_key to resolve_provider_client #3853 [closed] partial — (+633/-69) — n/a: Its relevant hunk forwards explicit fallback endpoint/key overrides, but the diff carries extensive unrelated changes; closed as duplicate of fix(fallback): pass base_url and api_key_env to resolve_provider_client #2587.
• fix: forward base_url and api_key from fallback config to resolve_provider_client #4784 [closed] partial — (+37/-2) — n/a: Forwarded custom fallback base_url/api_key values and added endpoint tests; closed in favor of the already-merged fix: Ollama Cloud auth, /model switch persistence, and alias tab completion #5269 implementation.
• fix(fallback): honor explicit base_url and api_key config #4895 [closed] fixes — (+13/-1) — n/a: Handles direct and environment-sourced fallback keys with base_url forwarding; closed as another fix(fallback): pass base_url and api_key_env to resolve_provider_client #2587 duplicate.
• fix: emit notification on credential pool exhaustion before fallback #10479 [closed] best fix — (+76/-5) — n/a: Recorded best existing diff for [Bug]: Credential pool provider fallback is silent — inconsistent with fallback_model notification behavior #10476, emitting tailored status on credential-pool exhaustion; closed while the later unified successful-fallback notice became the implemented path.
• fix(agent): notify gateway users when credential pool auth exhaustion causes provider switch #10480 [closed] partial — (+80/-0) — n/a: Added an auth-exhaustion notification while keeping successful same-provider rotation silent; closed because commit 3fe7f6d superseded its insertion point.
• fix(agent): notify when credential-pool exhaustion activates fallback #10716 [closed] partial — (+18/-0) — n/a: Routes billing-pool exhaustion into the existing fallback activator, but its CHANGES_REQUESTED review recorded missing repository-level CI; closed after the unified main implementation landed.
• fix: emit status notification on credential pool rotation #10718 [closed] related — (+18/-2) — n/a: Emitted status on every successful credential rotation, including same-provider rotation; closed because main instead surfaces the durable cross-provider/model fallback event.
• fix(fallback): resolve base_url from custom_providers when fallback entry has none (#15743) #15756 [closed] fixes — (+114/-0) — n/a: Looked up named custom-provider endpoint/key hints in the fallback call site, but only when both hints were absent; closed after shared resolver work landed.
• fix(agent): reset _fallback_index when previous turn exhausted chain without activating (#17446) #17824 [closed] best fix — (+87/-0) — n/a: Recorded best fix for [Bug] Fallback announced but never sent: trying fallback... logged when /model-set invalid id triggers HTTP 400, but fallback_model is never invoked and session aborts #17446, resetting an exhausted fallback index between turns even when activation never succeeded; closed after commit 4ab9a06 implemented it in the refactored helper.
• fix(fallback): look up custom_providers base_url before fallback API call #17827 [closed] fixes — (+33/-0) — n/a: Identified alias-shadowing and explicit-override gaps in both fallback activation and the shared client resolver; closed after being salvaged into fix(fallback): let custom_providers shadow built-in aliases #18185.
• fix(fallback): let custom_providers shadow built-in aliases #18185 [merged] best fix — (+157/-3) — n/a: Merged recorded best fix for Fallback provider defined in custom_providers ignores its own base_url, sends request to primary provider's endpoint instead #15743, resolving alias collisions and honoring explicit endpoint/key overrides centrally for all resolver callers.
• fix(agent): reset fallback chain index when activation fails in interactive sessions #20793 [closed] fixes — (+28/-0) — n/a: Resets the stranded fallback index in long-lived interactive sessions, explaining their divergence from fresh cron agents; closed after salvage into fix(agent): reset _fallback_index at turn start even when no fallback activated #27185.
• fix(cli): preserve custom fallback runtime overrides #24113 best fix — (+57/-1) — n/a: Recorded best existing fix for [Bug]: fallback_providers with provider: custom + explicit base_url falls back to OpenRouter when main model.provider is a named provider #24092, forwarding custom endpoint, environment key, and target model during CLI startup fallback; despite its keep_open/high-salvage review, the production hunk must be ported from obsolete cli.py to hermes_cli/cli_agent_setup_mixin.py:53-71.
• feat(agent): expose fallback model awareness to the agent #26569 [closed] best fix — (+35/-0) — n/a: Recorded best existing proposal for fallback-model awareness, injecting runtime identity and enhancing /model; closed because main now has the broader cache-safe identity implementation.
• fix(agent): reset _fallback_index at turn start even when no fallback activated #27185 [merged] best fix — (+28/-0) — n/a: Merged recorded best fix for [Bug] Interactive CLI session does not auto-fallback on Codex 429 'usage_limit_reached', while cron jobs with the same fallback chain do #20465, salvaging fix(agent): reset fallback chain index when activation fails in interactive sessions #20793's turn-start fallback-index reset with regression coverage.
• fix(agent): resolve NameError in rate-limit fallback decision #27359 [closed] fixes — (+3/-1) — n/a: Applies the correct _ra() NameError fix but includes unrelated CLI/release changes; closed after the clean implementation landed through fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345.
• fix(agent): wrap _pool_may_recover_from_rate_limit with _ra() lazy import #27374 [closed] fixes — (+1/-1) — n/a: Applies the same narrow _ra() qualification at the failing call site; closed as redundant after fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 merged.
• fix(fallback): NameError on 429 rate-limit — missing _pool_may_recover_from_rate_limit import #27433 [closed] fixes — (+53/-1) — n/a: Combines the same _ra() call-site fix with focused reachability tests; closed as duplicate of the canonical NameError fix.
• fix(conversation_loop): NameError on rate limit — route _pool_may_recover_from_rate_limit through _ra() (#27465) #27468 [closed] fixes — (+205/-1) — n/a: Adds the identical production fix plus extensive source and helper tests; closed after fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 landed the narrower implementation.
• fix: use _ra() accessor for _pool_may_recover_from_rate_limit #27532 [closed] fixes — (+1/-1) — n/a: Identical _ra() qualification for the rate-limit helper; closed as a duplicate of fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345.
• fix: use _ra() accessor for _pool_may_recover_from_rate_limit #27534 [closed] fixes — (+1/-1) — n/a: Identical _ra() qualification for the same extracted-loop NameError; closed as duplicate.
• fix(conversation_loop): NameError on rate-limit — _pool_may_recover_from_rate_limit not in scope #27608 [closed] fixes — (+1/-1) — n/a: Identical NameError fix, voluntarily closed in favor of the canonical issue and competing patches.
• fix(agent): resolve NameError in conversation_loop rate-limit path #27686 [closed] fixes — (+1/-1) — n/a: Identical _ra() fix for the rate-limit path; closed after fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 supplied it on main.
• fix(agent): import _pool_may_recover_from_rate_limit in conversation_loop (#27719) #27732 [closed] fixes — (+45/-0) — n/a: Uses a local lazy import rather than the repository's _ra() convention and adds import-shape tests; closed after fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 landed the preferred form.
• fix(agent): share fallback pool recovery helper #27734 [closed] fixes — (+39/-41) — n/a: Extracts the pool decision helper into a shared module with a compatibility alias; closed because the narrower fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 fix resolved the same crash.
• fix(agent): add missing _ra() prefix to pool recovery call #27750 [closed] fixes — (+1/-1) — n/a: Identical _ra() fix spanning the duplicate NameError issues; closed after the canonical salvage merged.
• fix(conversation_loop): use _ra() prefix for _pool_may_recover_from_r… #27896 [closed] fixes — (+1/-1) — n/a: Identical _ra() prefix on a slightly later base revision; closed as duplicate of the merged fix.
• fix: resolve fallback pool helper in conversation loop #27903 [closed] fixes — (+35/-0) — n/a: Adds a lazy wrapper and patchability test around the same helper-resolution failure; closed after fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 resolved the underlying call site.
• fix(conversation_loop): NameError on rate-limit fallback path #27945 [closed] fixes — (+1/-1) — n/a: Independently tested the same one-line _ra() correction; closed as duplicate before fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 became the merged reference.
• fix: wrap _pool_may_recover_from_rate_limit call through run_agent namespace #28159 [closed] fixes — (+17/-1) — n/a: Supplies the selected _ra() fix and inspect-based regression guard; closed after salvage into fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 with authorship preserved.
• fix(conversation_loop): use _ra() to access _pool_may_recover_from_rate_limit #28189 [closed] fixes — (+1/-1) — n/a: Identical one-line fix; closed as duplicate of fix: wrap _pool_may_recover_from_rate_limit call through run_agent namespace #28159 after fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 merged its guarded version.
• fix: import _pool_may_recover_from_rate_limit via _ra() in conversation_loop.py #28254 [closed] fixes — (+1/-1) — n/a: Identical _ra() qualification; closed as duplicate of fix: wrap _pool_may_recover_from_rate_limit call through run_agent namespace #28159/fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345.
• fix(conversation_loop): resolve NameError for _pool_may_recover_from_rate_limit #28268 [closed] fixes — (+1/-1) — n/a: Identical fix for the post-extraction NameError; closed after salvage fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 merged.
• fix(conversation_loop): resolve pool recovery helper via _ra() #28297 [closed] fixes — (+25/-1) — n/a: Combines the same production correction with a source-level regression guard; closed because fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 already carried that solution.
• fix(agent): wire _pool_may_recover_from_rate_limit through _ra() in extracted conversation_loop #28304 [closed] fixes — (+98/-1) — n/a: Adds the same _ra() fix plus an end-to-end 429 fallback test; closed after fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 merged the canonical production change.
• fix(conversation_loop): _ra()._pool_may_recover_from_rate_limit (NameError) (#28159) #28345 [merged] best fix — (+17/-1) — n/a: Merged recorded best fix for the NameError issue family, routing the helper through _ra() and guarding the call shape with an inspect-based test.
• fix(agent): add _ra() prefix to _pool_may_recover_from_rate_limit call #29210 [closed] fixes — (+1/-1) — n/a: Repeats the already-merged _ra() correction for duplicate issue NameError: _pool_may_recover_from_rate_limit is not defined — missing _ra() prefix in conversation_loop.py #29202; closed as redundant.
• fix: activate fallback after timeout recovery rate limits #32734 [closed] fixes — (+83/-0) — n/a: Explicitly activates fallback when 429/billing follows timeout recovery and adds a full-loop test; closed after broader transport handling and fix(agent): reopen fallback chain after primary transport recovery #54054 covered the failure sequence.
• fix(agent): reopen fallback chain after primary recovery #33846 [closed] fixes — (+331/-0) — n/a: Resets retry and fallback state after primary transport recovery so a subsequent 429 can traverse the chain; closed after salvage into fix(agent): reopen fallback chain after primary transport recovery #54054.
• feat: opt-in user-visible notification on provider fallback (#33174) #33891 [closed] best fix — (+72/-0) — n/a: Recorded best proposal for gateway fallback notification, wiring an opt-in status callback; closed because main now emits a durable one-shot notice and synchronizes runtime identity.
• docs: document OpenRouter model ID invalidation and fallback best practices #40087 [closed] related — (+29/-0) — n/a: Documents invalid OpenRouter model IDs and existing fast-fallback behavior without changing runtime code; closed while current main was verified to implement the behavior.
• fix(agent): reopen fallback chain after primary transport recovery #54054 [merged] best fix — (+331/-0) — n/a: Merged recorded best fix for fallback_providers not activated when 429 follows prior timeout recovery #32646, salvaging fix(agent): reopen fallback chain after primary recovery #33846's post-transport-recovery state reset and full-loop regression suite.
• fix: classify 429 quota-exhaustion errors as non-retryable for fallback #65633 related — (+28/-0) — n/a: Adds 429 usage-limit classification, but despite the keep_open/medium-salvage review its ordering captures ordinary “limit exceeded” and OpenRouter upstream 429s before their existing classifiers and adds no regression tests.

Duplicates

#1761 was salvaged by #3813; #17827 by #18185; #20793 by #27185; #33846 by #54054; and #28159 by #28345. #27359, #27374, #27433, #27468, #27532, #27534, #27608, #27686, #27732, #27734, #27750, #27896, #27903, #27945, #28189, #28254, #28268, #28297, #28304, and #29210 substantially duplicate #28345's NameError correction; #2587, #2705, #3182, #3853, #4784, and #4895 overlap on custom fallback override forwarding.

Suggested consolidation

Keep #65633 open with a salvage path: narrow its match so ordinary _RATE_LIMIT_PATTERNS and OpenRouter upstream 429 classification retain precedence, then add regression tests for the Ollama phrase, ordinary “Rate limit exceeded,” transient reset signals, and metadata.raw OpenRouter errors; this explicitly follows the contributor keep_open review rather than treating #65563's malformed configuration as proof that classifier work is unnecessary. Also keep #24113 open and port its verified forwarding logic to hermes_cli/cli_agent_setup_mixin.py:53-71 as requested by its keep_open review; all listed duplicate implementation PRs are already closed or merged, so no additional open PR can be closed as a duplicate from this set.

Cross-PR triage: Reviewed 49 pull requests and 27 issues in this complex. Each diff was read against this issue; Assessment working set: 264 kB of PR diffs, 171 kB of issue/PR text, 60 kB of discussion (117 comments), 89 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.