credential_pool: scope model-attributable 429 cooldowns to (credential, model)

[darkyy92]

Fixes #61451.

Problem

A 429 from a model-specific quota bucket (e.g. the separate Fable/Mythos-class quota on Anthropic subscriptions) marks the whole credential STATUS_EXHAUSTED and adopts that bucket's reset_at — which can be days out. The credential is then parked for every model, even ones whose shared unified budget is still free. Once every pool entry trips the same way, token resolution fails pool-wide and all Anthropic models error out.

Approach (conservative variant from the issue discussion)

No PooledCredential schema change; default behavior is unchanged.

• Attribution: a 429 is treated as model-scoped only when the response's anthropic-ratelimit-unified-{5h,7d}-utilization headers show every reported unified budget clearly below its ceiling (< 0.98). If a unified budget is at its ceiling, or the headers are missing/unparseable, the existing credential-wide cooldown applies. This mirrors the header-based genuine-vs-upstream distinction already used in nous_rate_guard.
• Cooldown: mark_exhausted_and_rotate(model=…, model_scoped=True) records an in-memory (credential_id, model) → reset_at cooldown instead of calling _mark_exhausted. _available_entries/_select_unlocked accept an optional model and skip entries only for that model, so rotation still moves the rate-limited model to the next credential while the entry stays available for everything else. In-memory is deliberate: losing the map on restart costs at most one extra 429 before it's re-learned.
• Wiring: recover_with_credential_pool (main loop; utilizations are captured into error_context by extract_api_error_context) and _recover_provider_pool (auxiliary client, from the exception's response headers). Billing (402) and auth (401/403) paths are untouched.

Tests

tests/agent/test_credential_pool_model_scoped_429.py: model-scoped cooldown blocks only that model and rotates correctly, not persisted to disk, expires, returns None when all entries are cooled for the model, credential-wide 429 unchanged, header/context heuristic parsing (fractions, percent-style, missing, garbage), and extract_api_error_context header capture. Existing routing-test stubs updated for the new kwargs. Pool-related suites pass locally; two pre-existing failures on my machine (test_persist_preserves_concurrent_disk_only_entry, test_remove_index_does_not_resurrect_via_disk_merge) fail identically on clean main.

https://claude.ai/code/session_013akwLFDAY6WqFwZUN7Zw5e

[alt-glitch]

This was generated by AI during triage.

Competing fix for #61451 with open PR #61481. This PR (5 files, focused) uses an in-memory (credential_id, model) -> reset_at cooldown map + header-based attribution (treat the 429 as model-scoped only when all anthropic-ratelimit-unified-* utilizations are < 0.98), no PooledCredential schema change. #61481 instead keys exhaustion state per-(credential, model) on the schema and arrives on a larger contaminated stacked branch (20 files). Same goal, different mechanism -> related, not duplicate. Flagging for a maintainer to pick the canonical approach.

[teknium1]

Thanks for the focused conservative implementation. The premise is confirmed on current main: agent/agent_runtime_helpers.py:846 rotates a 429 without model context, while agent/credential_pool.py:1365-1485 selects only against credential-wide exhaustion.

Suggested changes

• Consider direct recovery-path tests for recover_with_credential_pool() and _recover_provider_pool() that exercise the Anthropic response headers. The new pool tests cover the cooldown primitive well; these would additionally lock down the main-loop and auxiliary attribution wiring.

Automated hermes-sweeper review.

[teknium1]

The conservative Anthropic-only attribution direction is substantially safer than blanket model scoping, but this is not merge-ready yet. The in-memory cooldown is attached to one CredentialPool instance; auxiliary recovery evicts clients and later resolution loads a fresh pool, losing the cooldown immediately. Public select() / has_available() seams also remain partly unscoped, and model keys need canonical normalization.

Please keep ambiguous/missing-header 429s and every 401/402/403 credential-wide, then make the attributed cooldown survive pool reloads and thread the model through every selection/recovery seam. With those changes plus a real auxiliary recovery regression, this remains salvageable.

[darkyy92]

Addressed the latest review in d4383d936 and merged current main into the branch first.

Changes:

• moved attributed model cooldowns to a profile-isolated, lock-protected process registry, so fresh CredentialPool instances and auxiliary client eviction/rebuilds retain the cooldown without changing PooledCredential / auth.json
• canonicalized cooldown model keys through the provider model normalizer
• threaded model through public select() / has_available() (plus peek()), primary fallback/restore recovery, and sync/async auxiliary recovery + client rebuild
• kept missing/ambiguous-header 429s and all 401/402/403 paths credential-wide
• preserved provider reset timing from x-ratelimit-reset / retry-after in auxiliary recovery
• added direct main recovery and real auxiliary eviction/reload regressions

Verification:

• 186/186 credential-pool, main recovery, restore, and fallback tests passed
• 300/300 auxiliary-client tests passed
• focused post-review rerun: 320/320 passed
• Ruff on all changed Python files and git diff --check passed

I intentionally did not persist model cooldowns across process restarts: the conservative design avoids an auth-store schema/migration change; a restart may cause one re-probe, while the requested pool reload/client eviction lifecycle is preserved.

[teknium1]

Closing after re-checking the premise against Anthropic's documented rate-limit model and the available subscription-header evidence.

The implementation is focused and deliberately conservative, but its key inference is not established: low anthropic-ratelimit-unified-{5h,7d}-utilization does not prove that the 429 belongs to a model-specific bucket. Those OAuth subscription headers are undocumented, include separate status and representative-claim semantics, and a 429 can also come from acceleration, capacity, or another limiter. Issue #61451 does not include the actual failing 429 headers identifying a Fable/Mythos-specific limiter.

Changing credential selection on that heuristic could reuse a credential whose applicable shared limit is genuinely exhausted. The in-memory cooldown also disappears when auxiliary recovery reloads the pool, so the patch would not consistently enforce its own decision.

Thank you for the careful fail-closed defaults, header parsing, and credential-pool tests. If Anthropic supplies a stable explicit exhausted-limiter identity in captured 429 responses, a narrowly signal-gated fix can be reconsidered with that evidence.