fix(aux): discard stale credentials after auth refresh

[marveon]

Summary

• call_llm() / async_call_llm() correctly detect a 401 on an auxiliary request (context compression, title generation, ...), call _refresh_provider_credentials("anthropic") to rotate the Claude Code OAuth token on disk, and evict the cached client — but then retry using the exact resolved_api_key string that had just been rejected.
• _get_cached_client() treats an explicit api_key as authoritative and never re-derives it, so the "retry" silently rebuilds a client with the same dead token and 401s again.
• Observed in production: an auxiliary client kept hitting OAuth access token has been revoked every ~5-9 minutes for 4+ hours while the main conversation loop (which resolves credentials independently via credential_pool) kept working fine. Because context compression could never succeed, the session transcript grew unchecked to ~336k tokens, and the turn eventually died once the account also hit a real billing/usage cap and fell back to a local model with a much smaller context window.
• Fix: re-resolve the token via resolve_anthropic_token() before retrying, in both the sync and async auth-refresh-retry paths.

Test plan

• Added test_call_llm_anthropic_401_retry_does_not_reuse_stale_api_key, verified it fails against the pre-fix code (stashed the fix and re-ran) and passes with the fix.
• pytest tests/agent/test_auxiliary_client.py — 363 passed.
• pytest tests/agent/test_auxiliary_client_xai_oauth_recovery.py — all passed (unaffected provider paths).

[alt-glitch]

This was generated by AI during triage.

Duplicate of #62008: its current generic retry path already clears the stale request-scoped API key after a successful auxiliary auth refresh, including the Anthropic case covered here.

[teknium1]

Thanks for the focused reproduction. The stale-key premise is confirmed on current main: the sync retry forwards resolved_api_key at agent/auxiliary_client.py:8379, and the async retry does the same at agent/auxiliary_client.py:9020; _get_cached_client() preserves a non-empty explicit key (agent/auxiliary_client.py:6707-6721).

Problems

• The patch special-cases Anthropic, but both paths use the same generic retry helpers for every refreshable provider. Other providers can still rebuild with the rejected request-scoped key. Open PR fix(auxiliary): discard stale credentials after auth refresh #62008 contains the broader stale-input-clearing direction, but it is not yet on main.
• The new test covers only the sync Anthropic path; the async Anthropic hunk has no corresponding regression assertion.

Suggested changes

• Consolidate this into a provider-neutral stale-input reset after successful auth refresh, before the retry helper is called.
• Add an async Anthropic 401 test asserting that the retry uses the refreshed credential.

Automated hermes-sweeper review.

[GottZ]

This was generated by AI during triage.

Summary

Twenty-two PRs address or reference this complex, but their diffs cover distinct causes behind similar Anthropic 400/429 symptoms: billing classification, endpoint and credential selection, OAuth tool/system-prompt fingerprints, billing attribution, subprocess alternatives, and stale credential reuse. For target #72263, the relevant cause is narrower: auxiliary retries retain a rejected request-scoped credential after a successful auth refresh.

Related pull requests

• [codex] Clarify Anthropic extra-usage exhaustion errors #6498 [closed] best fix — (+130/-9) — best fix for Anthropic Claude subscription auth returns 'You're out of extra usage' in Hermes even after restart/re-login #6475 and [Bug]: claude /login become API Error: HTTP 400 #65078: classifies the extra-usage 400 and adds actionable fallback guidance; it remains relevant as the closed reference implemented on main through fix(classifier): treat Anthropic "out of extra usage" 400 as billing #56128.
• fix: Anthropic OAuth sends empty X-Api-Key, misclassifies billing error #6854 [closed] best fix — (+28/-13) — best fix for the recorded OAuth (Claude Pro/Max): exposing memory or session_search tool schema deterministically triggers HTTP 400 "You're out of extra usage" #65365 facet: prevents an empty X-Api-Key alongside Bearer auth and adjusts classifier-sensitive prompt terms, but remains a closed mixed-scope reference superseded by focused classification and wire-alias changes.
• fix(anthropic): sanitize oauth system prompt for Claude Max proxy #10576 [closed] best fix — (+43/-6) — best fix for [Bug] Anthropic OAuth/Claude Max proxy path can misclassify full Hermes system prompt as extra-usage exhausted #10575, Anthropic Claude Code OAuth still appears to consume extra usage credits after June 16 policy rollback email #47260, OAuth (Claude Pro/Max): exposing memory or session_search tool schema deterministically triggers HTTP 400 "You're out of extra usage" #65365, and Anthropic OAuth (Claude Pro/Max subscription) hits "You're out of extra usage" on every real request — routes through pay-per-token endpoint instead of subscription quota #65564: sanitizes the bisected OAuth system-prompt terms, but remains closed because the blocking contributor decision rejected an open-ended classifier-evasion scrubber.
• fix(anthropic): use Claude Code-compatible MCP tool prefix for OAuth #17681 [closed] best fix — (+52/-16) — best fix for OAuth Anthropic + Pro/Max plan: mcp_ tool-name prefix triggers "out of extra usage" 400 on every tool-bearing request #28849 and Max OAuth requests rejected as third-party (HTTP 400 "extra usage") due to single-underscore mcp_ tool-name prefix #46675: changes OAuth tool names from mcp_ to mcp__ with response round-tripping; it remains relevant because b70a4e7533dce506a41f847b3fc082d6b6fe4a9b implemented the broader form on main.
• fix(auth): stop treating CLAUDE_CODE_OAUTH_TOKEN as Anthropic API key #19260 [closed] related — (+3/-3) — rejected alternative: removes intentional Claude Code OAuth credential discovery instead of repairing the failing request path, so the closed diff remains relevant as a documented non-solution.
• fix(anthropic): classify OAuth tool-use overage failures #21019 [closed] best fix — (+96/-1) — best fix for [Bug]: On a Claude Max 20x subscription with a valid OAuth access token from ~/.claude/.credentials.json, every Hermes request to native Anthropic (provider: anthropic, https://api.anthropic.com/v1/messages) is rejected with HTTP 400 #15080, Error code: 400 | "You're out of extra usage. Add more at claude.ai/settings/usage and keep going." #20732, [Bug]: Hermes does not work through Claude CLI #29125, [Bug]: Hermes w/ Anthropic models ratelimit/extra usage #31668, [Setup]: Problem at 0Auth #37363, and [Bug]: claude /login become API Error: HTTP 400 #65078: classifies tool-bearing OAuth overage failures and enables fallback guidance; equivalent supported behavior shipped through ea9e8d6e8c80fbdd0b0b4864b833d2bfd3b429bb and e00800fc89e761382326c3a5eef8f4cccfbbc205.
• fix(anthropic): work around OAuth third-party billing-lane classifier (#15080) #24250 best fix — (+1002/-26) — best existing broad experiment for [Bug]: On a Claude Max 20x subscription with a valid OAuth access token from ~/.claude/.credentials.json, every Hermes request to native Anthropic (provider: anthropic, https://api.anthropic.com/v1/messages) is rejected with HTTP 400 #15080: despite its keep_open review, the diff replaces the system prompt during a live turn and marks confirmed billing errors retryable, so author action should split out only a prompt-preserving, independently evidenced compatibility change.
• feat(agent): add ClaudeLocalClient subprocess shim for Claude Max OAuth accounts #26960 [closed] related — (+230/-0) — superseded subprocess prototype: bypasses the native OAuth classifier through claude, but lacks completed provider integration and end-to-end tool verification and was closed in favor of feat(providers): add Claude CLI subprocess transport #6427 and feat(providers): claude-code-acp — run Hermes on a Claude subscription, plus ACP timeout fixes #12229.
• fix(anthropic): remove mcp_ tool prefix on OAuth path that triggers overage gate #28872 [closed] best fix — (+115/-18) — best proposal-stage fix for OAuth Anthropic + Pro/Max plan: mcp_ tool-name prefix triggers "out of extra usage" 400 on every tool-bearing request #28849: despite its earlier keep_open review, wholesale prefix removal conflicts with the maintained tool-access contract, and the closed PR was superseded by fix(anthropic): no single-underscore mcp_ tool names on the OAuth wire (plan-limit billing) #47723's mcp__ normalization.
• fix(provider): route OAuth Pro/Max credentials to /v1/messages (#32243) #32318 [closed] best fix — (+267/-5) — best fix for [Bug]: OAuth Pro/Max credential returns "out of extra usage" 400 immediately after fresh login, despite empty/unused account quota #32243 and [Setup]: Hermes Model - Anthropic Claude Pro - OAuth wrong routing #33192: pins api.anthropic.com to native anthropic_messages; it remains a merged-reference implementation because fix(provider): route api.anthropic.com to anthropic_messages (#32243) #56214 incorporated its commits and widened the sibling fallback path.
• fix: classify Anthropic subscription_exhausted (HTTP 400 "out of extra usage") with actionable error message #40020 [closed] best fix — (+278/-9) — best mixed proposal for [Bug]: Hermes does not work through Claude CLI #29125, [Bug]: Hermes w/ Anthropic models ratelimit/extra usage #31668, and Claude Code OAuth (Max/Pro plan) still hits pay-per-token API endpoint — drains 'extra usage' credits instead of using subscription quota #40014: combines valid billing classification with an incomplete Claude CLI subprocess provider, remaining relevant after closure as the work that reviewers required to be split.
• fix: classify Anthropic subscription_exhausted (HTTP 400 "out of extra usage") correctly #40073 [closed] best fix — (+27/-2) — best focused recovery fix for [Bug]: Hermes does not work through Claude CLI #29125, [Bug]: Hermes w/ Anthropic models ratelimit/extra usage #31668, [Setup]: Problem at 0Auth #37363, Claude Code OAuth (Max/Pro plan) still hits pay-per-token API endpoint — drains 'extra usage' credits instead of using subscription quota #40014, and [Bug]: claude /login become API Error: HTTP 400 #65078: maps the 400 to non-retryable billing recovery and credential rotation, with equivalent behavior implemented on main through fix(classifier): treat Anthropic "out of extra usage" 400 as billing #56128.
• fix(anthropic): use double-underscore mcp__ prefix for OAuth tool names #46687 [closed] best fix — (+49/-30) — best fix for OAuth Anthropic + Pro/Max plan: mcp_ tool-name prefix triggers "out of extra usage" 400 on every tool-bearing request #28849 and Max OAuth requests rejected as third-party (HTTP 400 "extra usage") due to single-underscore mcp_ tool-name prefix #46675: symmetrically changes request and response names to mcp__; it remains relevant because fix(anthropic): no single-underscore mcp_ tool names on the OAuth wire (plan-limit billing) #47723 cherry-picked it and added legacy MCP-server normalization.
• fix(agent): send Claude Code billing-attribution system block on OAuth requests #48177 [closed] best fix — (+88/-4) — best fix for [Bug]: OAuth Pro/Max/Team requests rejected with HTTP 400 "third-party / extra usage" (missing x-anthropic-billing-header system block) #48176: adds the A/B-backed x-anthropic-billing-header mechanism, but remains closed because the blocking contributor decision rejected presenting Hermes as the first-party SDK surface due to account-suspension risk.
• fix(anthropic): add billing header for OAuth plan-tier routing (#48176) #48202 [closed] fixes — (+20/-7) — duplicate of fix(agent): send Claude Code billing-attribution system block on OAuth requests #48177, closed: adds the same cc_entrypoint=sdk-cli billing block with less coverage, and its non-contributor approval does not override the contributor duplicate finding and policy block.
• fix(anthropic): add OAuth plan billing marker #69844 best fix — (+62/-4) — best fix for Claude Code OAuth (Max/Pro plan) still hits pay-per-token API endpoint — drains 'extra usage' credits instead of using subscription quota #40014, [Bug]: OAuth Pro/Max/Team requests rejected with HTTP 400 "third-party / extra usage" (missing x-anthropic-billing-header system block) #48176, and [Bug]: claude /login become API Error: HTTP 400 #65078: the keep_open review and current-main exact-path A/B support the focused billing-marker diff, but the earlier blocking policy objection to cc_entrypoint=sdk-cli must be resolved before progression.
• fix(agent): strip x-stainless headers from OAuth Anthropic requests to prevent 429 #70054 [closed] best fix — (+120/-0) — best proposal for Anthropic subscription OAuth token gets generic 429 through Hermes, while the genuine Claude Code binary works with the same token #70039's header-fingerprint hypothesis: strips x-stainless-* headers, but remains closed because its tests prove header removal rather than successful live 429 recovery and the close verdict found no established resolution.
• fix(anthropic): keep OAuth requests on included subscription billing #72173 best fix — (+322/-0) — best fix for Anthropic Claude Code OAuth still appears to consume extra usage credits after June 16 policy rollback email #47260, Claude Code / subscription OAuth requests are billed as overage (HTTP 429 "monthly spend limit") — root cause: the agent system prompt trips Anthropic's subscription billing classifier #53212, Anthropic OAuth (Claude Pro/Max subscription) hits "You're out of extra usage" on every real request — routes through pay-per-token endpoint instead of subscription quota #65564, and Anthropic OAuth: system-prompt shape routes subscription requests into the metered "extra usage" lane (root cause + bisection for #65564) #72171: directly targets the bisected system-prompt trigger, but its keep_open review requires prompt-preserving relocation plus production cache-block and breakpoint-limit coverage instead of default truncation.
• fix(aux): discard stale credentials after auth refresh #72263 related — (+255/-3) — adjacent stale-credential fix: clears rejected request-scoped keys before sync and async auxiliary retries, but despite the keep_open review on fix(aux): discard stale credentials after auth refresh #72263, the diff addresses the same generic stale-input class already attributed to broader fix(auxiliary): discard stale credentials after auth refresh #62008 rather than an Anthropic-only defect.
• fix(anthropic): never refresh an API-key session into Claude Code OAuth credentials (#75641) #75697 best fix — (+24/-3) — best fix for [Bug]: Anthropic API-key request returns 429, then retry produces OAuth extra-usage 400 #75641: preserves an explicitly selected Console API key across retries and prevents the confirmed API-key-to-OAuth identity swap, matching its high-salvageability keep_open review.
• fix(anthropic): alias session search on OAuth wire #76669 [closed] best fix — (+97/-12) — best initial fix for OAuth (Claude Pro/Max): exposing memory or session_search tool schema deterministically triggers HTTP 400 "You're out of extra usage" #65365, closed after salvage: aliases session_search on the OAuth wire and reverses it before dispatch, while fix(anthropic): alias the session_search and memory schemas on the OAuth wire #76807 carries the commit with attribution and closes the broader trigger and collision gaps.
• fix(anthropic): alias the session_search and memory schemas on the OAuth wire #76807 best fix — (+319/-7) — best expanded fix for OAuth (Claude Pro/Max): exposing memory or session_search tool schema deterministically triggers HTTP 400 "You're out of extra usage" #65365: aliases session_search and memory only on the OAuth wire, preserves API-key behavior, handles collisions and registry precedence, and round-trips responses.

Duplicates

#6498, #21019, #40020, and #40073 overlap on billing classification and guidance; #17681, #28872, and #46687 form the MCP-prefix chain. #48177, #48202, and #69844 share the billing-marker mechanism; #76669 was salvaged into #76807; and #72263 overlaps the broader stale-input reset in #62008.

Suggested consolidation

Close #72263 as duplicate of #62008, with the explicit chain #72263 → #62008: despite the keep_open review on #72263, its diff clears the rejected resolved_api_key and stale runtime at the generic auxiliary retry call sites, while the contributor discussion identifies #62008 as the provider-neutral implementation of that same class. Preserve or transfer #72263's sync and async Anthropic regression assertions to #62008; keep #75697 and #76807 open separately because they address distinct credential-identity and OAuth wire-schema causes.

Cross-PR triage: Reviewed 22 pull requests and 22 issues in this complex. Each diff was read against this issue; Assessment working set: 244 kB of PR diffs, 253 kB of issue/PR text, 82 kB of discussion (169 comments), 110 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.