fix(credential-pool): reduce TTL for HTTP 402 (Payment Required) from 1h to 2min

[webtecnica]

Problem

When any provider returns HTTP 402 (Payment Required — credits exhausted), the credential pool marks the key as STATUS_EXHAUSTED with the default 1-hour TTL. After the user recharges credits, they must wait the full hour for the credential to become available again — even across gateway restarts, since the status is persisted in auth.json.

This affects all API-key providers that use pay-as-you-go billing:

• deepseek, openai-api, openrouter, anthropic (API key mode)
• alibaba, zai, kimi-for-coding, stepfun, minimax, minimax-cn
• nvidia, fireworks, upstage, gmi, xiaomi, tencent-tokenhub
• arcee, ollama-cloud, novita, azure-foundry, kilo, github-copilot
• opencode, opencode-go, vercel, huggingface

And OAuth providers: anthropic (OAuth), openai-codex, nous, xai-oauth, minimax-oauth, qwen-oauth.

Root Cause

_exhausted_ttl() only had special cases for HTTP 401 (5 min) and 429 (1 hour). HTTP 402 fell through to the default 1-hour TTL, which is appropriate for rate limits but too aggressive for billing/payment errors.

Fix

• New constant: EXHAUSTED_TTL_402_SECONDS = 120 (2 minutes)
• _exhausted_ttl(): now checks for error_code == 402 before the 429/generic fallthrough

HTTP 402 is fundamentally different from 429: when the user recharges, the key works again immediately. A 2-minute window is enough to prevent rapid retry storms while being short enough that the credential auto-recovers shortly after recharging. If credits are still out after 2 min, the next request fails again and re-marks the credential with a fresh TTL.

Related

Companion PR in Hermes WebUI: nesquena/hermes-webui#6626

[webtecnica]

Good fix! A 1-hour TTL for HTTP 402 is far too punishing — the user recharges credits and the provider stays "exhausted" for an hour with no explanation. 2 minutes gives the balance time to propagate without making the user think the provider is broken.

[teknium1]

Thanks for the focused credential-pool fix. The reported behavior is present on current main: agent/credential_pool.py:289-295 sends 402 through the one-hour default, and agent/credential_pool.py:376-383 uses that result when no provider reset time is available.

Problems

• The PR adds no regression test for the new 402 cooldown. tests/agent/test_credential_pool.py:42-80 covers an explicit 429 reset timestamp and :82-145 covers 402 sibling-key exhaustion, but neither verifies 402 TTL expiry.

Suggested changes

• Add a focused credential-pool test that creates an exhausted 402 entry without last_error_reset_at and verifies it remains unavailable before, then becomes selectable after, EXHAUSTED_TTL_402_SECONDS. This exercises the production path in agent/credential_pool.py:376-383 and protects the intended distinction from 429.

Automated hermes-sweeper review.

[teknium1]

Please add a regression test for this status-specific TTL. Existing pool tests cover 402 rotation and 429 reset timestamps, but not a 402 entry becoming available after this new default cooldown.

[webtecnica]

Thanks for the review, @teknium1! I've addressed the request and added the regression test for the status-specific 402 TTL.

New test: test_exhausted_402_without_reset_at_recovers_after_short_ttl in tests/agent/test_credential_pool.py

It covers exactly the gap you pointed out — a 402-exhausted entry without a provider-supplied last_error_reset_at recovering via the new default cooldown:

1. Before EXHAUSTED_TTL_402_SECONDS elapses — the entry stays unavailable (has_available() is False, select() is None).
2. After EXHAUSTED_TTL_402_SECONDS elapses — the entry becomes selectable again, and the cooldown is genuinely cleared (persisted back to STATUS_OK on disk, last_error_code reset), not just skipped.
3. Distinguishes 402 from 429 — a 429-exhausted entry of the same age is still benched (1 h TTL), proving the short 2-minute recovery is 402-only.

The test exercises the production fallback path in _exhausted_until (last_status_at + _exhausted_ttl(last_error_code) when no reset timestamp is present), which is the path this PR's change affects. I also verified it's a genuine regression test: with the _exhausted_ttl 402 branch removed (constant kept), the test fails at assert pool.has_available() is True.

Branch was also rebased onto the latest upstream main so the PR is mergeable — the diff is now just the 3-line fix plus the new test. Full file run: 59 passed (tests/agent/test_credential_pool.py, -p no:xdist).

Commits:

• 22dbbafbc fix(credential-pool): reduce TTL for HTTP 402 (Payment Required) from 1h to 2min
• 41ea1fb2a test(credential-pool): verify 402 TTL expiry without last_error_reset_at (fix(credential-pool): reduce TTL for HTTP 402 (Payment Required) from 1h to 2min #74946)

[webtecnica]

Rebased the branch onto the latest upstream main — the PR is mergeable again.

What changed since the last review:

1. Rebase onto current main — upstream refactored _exhausted_ttl() (sole-credential handling + is_billing classification) since this PR was last rebased. Resolved the conflict by keeping upstream's new logic intact and layering the 402 short-TTL on top: the early if error_code == 402 return runs before the billing/sole-credential path (which would otherwise bench 402 for a full hour). The is_billing check no longer needs the error_code == 402 term since 402 never reaches it.

2. Regression test updated for the new upstream semantics — the original 429-comparison case assumed a single-entry pool, but upstream now shortens transient throttles for a sole credential (that's exactly the 429/402 distinction the sweeper asked to protect). The test now uses a two-entry pool for the 429 leg so it keeps its full 1-hour bench, proving the short TTL is 402-only. 60 passed in tests/agent/test_credential_pool.py, ruff clean.

Why the short 402 TTL matters (production evidence): this has been running in our production deployment since late July. Prepaid/PAYG providers (DeepSeek etc.) return 402 when the balance runs out mid-workload; with the 1-hour default the key stays STATUS_EXHAUSTED (persisted in auth.json) for the remaining hour even after the account is recharged. With the 2-minute TTL the pool self-heals: recharge → key back in rotation within ~2 min, and if still unfunded the next request just re-marks it with a fresh TTL.

Related: this behavior is also documented on the configurable-TTL feature request — #33049 (comment) — which proposes moving these cooldowns into config.yaml (402: 120 etc.). This PR is the interim hardcoded default; a configurable TTL would supersede it cleanly.

@teknium1 — could you re-review? Both points from the sweeper review are addressed (regression test added + mergeable on latest main).