Skip to content

fix(classifier): a 429 about money is billing, not a rate limit - #90503

Open
rodrigogs wants to merge 1 commit into
NousResearch:mainfrom
rodrigogs:fix/429-billing-is-not-rate-limit
Open

fix(classifier): a 429 about money is billing, not a rate limit#90503
rodrigogs wants to merge 1 commit into
NousResearch:mainfrom
rodrigogs:fix/429-billing-is-not-rate-limit

Conversation

@rodrigogs

Copy link
Copy Markdown

Problem

Providers reuse HTTP 429 for a depleted balance. z.ai answers one with 1113 Insufficient balance, Please recharge, and an exhausted weekly allowance with a message naming a reset days away — both status 429, neither about rate.

The 429 branch fell straight through to FailoverReason.rate_limit, which earns the 429 cooldown:

value
EXHAUSTED_TTL_429_SECONDS 3600
EXHAUSTED_TTL_SOLE_CREDENTIAL_SECONDS 60
actual wait for a depleted balance until a top-up
actual wait for an exhausted allowance until the window rolls over — days

credential_pool.py caps the bench at 60s when the pool holds a single credential and the failure is not billing. On the install that surfaced this, every per-profile pool holds exactly one entry, so a provider that could not serve a request for days was benched for a minute and then hammered again, every minute, while the fallback chain rotated onto it in turn.

_BILLING_PATTERNS already carries "insufficient balance" and "credits have been exhausted". It is consulted at four places in this file — none of them the 429 branch.

Fix

Check _BILLING_PATTERNS inside the 429 branch and classify as billing (not retryable, rotate, fall back), which is the same shape the 402 and 404 paths already use for the same condition.

Placed after the overload and OpenRouter-upstream checks, deliberately: a busy endpoint and an aggregator throttling its own upstream are both 429s that say nothing about this account's balance, and both already have the correct recovery above. Billing must not steal either.

Verification

4 tests, next to the existing test_429_with_overloaded_body_is_overloaded_not_rate_limit sibling:

  • test_429_with_insufficient_balance_is_billing_not_rate_limit — the z.ai 1113 shape
  • test_429_with_an_exhausted_plan_allowance_is_billing
  • test_429_without_a_money_body_is_still_rate_limit — guard against over-classifying; an ordinary throttle stays rate_limit and stays retryable, so the caller still backs off instead of benching
  • test_429_overload_still_outranks_a_money_reading — ordering guard, with both phrases in one body

Mutation-checked: reverting agent/error_classifier.py and keeping the tests fails exactly the two billing tests and leaves the two guards green, so the guards are known to pin pre-existing behaviour rather than the new branch.

pytest tests/agent/test_error_classifier.py101 passed. ruff clean.

Posting the local run because this repo does not run checks on PRs from forks.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/billing Account usage, credit usage, billing (cross-cutting) area/usage-cost Token accounting, usage reporting, billing, cost tracking provider/zai ZAI provider labels Aug 20, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related: open #36280 covers the same 429 billing/quota classification family with broader permanent-usage-limit handling. This PR is the focused existing-_BILLING_PATTERNS approach; maintainers should choose whether to consolidate the overlapping paths.

Providers reuse HTTP 429 for a depleted balance. z.ai answers one with
"1113 Insufficient balance, Please recharge", and a weekly-allowance overrun with
a message naming a reset days out — both status 429, neither about rate.

The 429 branch fell straight through to FailoverReason.rate_limit, which earns the
429 cooldown: one hour, or 60 seconds when it is the pool's only credential
(EXHAUSTED_TTL_SOLE_CREDENTIAL_SECONDS). Every per-profile pool on the install
that surfaced this holds exactly one entry, so a provider blocked until a top-up
or a window rollover got benched for a minute and then hammered again. The wait is
days; the bench was 60s.

_BILLING_PATTERNS already carries "insufficient balance" and "credits have been
exhausted" — it was simply never consulted for 429. It is checked at four places
in this file, all of them other status codes.

Placed after the overload and OpenRouter-upstream branches, deliberately. A busy
endpoint and an aggregator throttling its own upstream are both 429s that say
nothing about this account's balance, and both already have the right recovery
above; billing must not steal either.

4 tests, next to the existing 429-overload sibling. Two pin the fix, two are
guards: an ordinary throttle stays rate_limit and stays retryable, and overload
still outranks a money reading when both phrases appear. Verified by mutation —
reverting agent/error_classifier.py fails exactly the two billing tests and leaves
the two guards green, so they are known to pin pre-existing behaviour.

101 passed in tests/agent/test_error_classifier.py.
@rodrigogs
rodrigogs force-pushed the fix/429-billing-is-not-rate-limit branch from 33212b2 to 953b83a Compare August 20, 2026 20:13
@rodrigogs

Copy link
Copy Markdown
Author

Useful pointer — I looked at #36280 to answer the consolidation question concretely
rather than leave it open.

They overlap in one file and diverge in scope. Both touch
agent/error_classifier.py and tests/agent/test_error_classifier.py:

  • fix(agent): trigger fallback on quota-exhaustion 429s (e.g. Kimi monthly limit) #36280 (2026-06-01, +83/-4, 3 files) fixes two gaps for permanent
    quota-exhaustion 429s: the unconditional rate_limit classification, and
    recover_with_credential_pool in agent/agent_runtime_helpers.py, whose
    hardcoded strings missed Kimi's "monthly usage limit" phrasing so the
    fallback_model chain never engaged.
  • This PR (+67/-0, 2 files) only does the classification half, and does it by
    consulting the existing _BILLING_PATTERNS set inside the 429 branch — after the
    overload and OpenRouter-upstream checks — rather than adding new literals.

So the classifier halves are two approaches to the same defect, and #36280's
agent_runtime_helpers.py half is not covered here at all.

Two coherent ways to land them, whichever the maintainers prefer:

  1. Take this PR for the classification (it reuses a maintained pattern set instead
    of growing a second list of hardcoded phrases), and reduce fix(agent): trigger fallback on quota-exhaustion 429s (e.g. Kimi monthly limit) #36280 to its
    recover_with_credential_pool half. The two then compose with no shared lines.
  2. Take fix(agent): trigger fallback on quota-exhaustion 429s (e.g. Kimi monthly limit) #36280 whole, and this PR reduces to whatever _BILLING_PATTERNS matches
    that its literals do not.

I am happy to do the reduction either way — say which one you want and I will
rebase this branch to fit rather than leave you two overlapping diffs to
reconcile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/billing Account usage, credit usage, billing (cross-cutting) area/usage-cost Token accounting, usage reporting, billing, cost tracking comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists provider/zai ZAI provider type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants