fix(auxiliary): auth recovery for auto routes and stale fallback candidates - #59837
Merged
Conversation
Infer the concrete auxiliary auth provider from the selected client base URL so provider:auto routes can refresh Copilot/Codex/Anthropic/Nous credentials after auth errors, instead of skipping refresh because resolved_provider stayed 'auto'. Adds the copilot branch to _refresh_provider_credentials and evicts the stale auto-route cache before retrying. Fixes #20832. Salvaged from PR #20837, reapplied surgically onto current main (branch predated the _retry_same_provider_sync/async extraction).
…tead of aborting A fallback candidate can itself carry a stale credential (e.g. an expired ANTHROPIC_TOKEN picked up by _try_anthropic). Its 401 previously propagated out of the fallback call site and aborted the auxiliary task — for compression: a 60s cooldown + context marker while the session kept growing past the context cap. Live case: mattalachia debug dump (Jul 2026), Codex timeout → Anthropic 401 x5 → 296K 'Cannot compress further'. Now each fallback candidate call is wrapped: on auth error, refresh the candidate's provider credentials and retry once; if unrefreshable, mark the provider unhealthy and walk the discovery chain again so the next viable candidate serves. Sync + async paths. Non-auth errors still raise unchanged.
alt-glitch
added
type/bug
This was referenced Jul 7, 2026
This was referenced Jul 7, 2026
19 tasks
dstkwll
added a commit
to dstkwll/hermes-agent
that referenced
this pull request
Jul 24, 2026
…oken 401 Copilot degrades in two related ways that both abort a turn as non-retryable and only clear on a gateway restart (a cold process re-runs the token exchange): 1. HTTP 400 model_not_available_for_integrator / model_not_supported — a raw/degraded token routes to the restricted copilot-language-server integrator whose allowlist omits enterprise-only models (e.g. claude-opus-4.8). Because it is a 400 (not 401), the existing 401 refresh path never fired. Prevented (retry-with-backoff exchange + on-disk JWT persistence + header guard at the client chokepoint) and self-healed at runtime (single-shot forced re-exchange + client rebuild + retry before fallback). 2. HTTP 401 'IDE token expired: unauthorized: token expired' — the short-TTL *exchanged* IDE token expires mid-turn. The clean-401 path DID fire and call _try_refresh_copilot_client_credentials(), but that method only re-resolved the stable raw ghu_ token and rebuilt the client — it never evicted the cached exchanged JWT or forced a fresh exchange, so the retry put the SAME expired token back on the wire, 401'd again, and the single-shot guard aborted the turn. Fix: force a fresh IDE-token exchange (evict cached JWT via evict_cached_exchanged_token + re-mint via get_copilot_api_token) before the client rebuild, mirroring the merged auxiliary-path recovery (NousResearch#59837) and the 400 recovery in this same PR. Graceful fallback to the resolved token if the exchange endpoint is unreachable; picks up the enterprise base_url on re-exchange. Brings main-loop clean-401 recovery to parity with the merged auxiliary path (NousResearch#59837), using the newer on-disk-aware evict helper. Companion context: NousResearch#58743 (this PR, expanded), NousResearch#51313, NousResearch#63204 (which assumed the 401 path already recovered — it reached the method but the method was too weak). Tests: exchange retry/persist round-trip, restart-blip disk reuse, stale-cred 400 classifier, 400 recovery, and 3 new 401 cases (fresh exchanged token on the wire; network-blip fallback to resolved token). 58 copilot tests green on current main.
teknium1
pushed a commit
that referenced
this pull request
Aug 1, 2026
…oken 401 Copilot degrades in two related ways that both abort a turn as non-retryable and only clear on a gateway restart (a cold process re-runs the token exchange): 1. HTTP 400 model_not_available_for_integrator / model_not_supported — a raw/degraded token routes to the restricted copilot-language-server integrator whose allowlist omits enterprise-only models (e.g. claude-opus-4.8). Because it is a 400 (not 401), the existing 401 refresh path never fired. Prevented (retry-with-backoff exchange + on-disk JWT persistence + header guard at the client chokepoint) and self-healed at runtime (single-shot forced re-exchange + client rebuild + retry before fallback). 2. HTTP 401 'IDE token expired: unauthorized: token expired' — the short-TTL *exchanged* IDE token expires mid-turn. The clean-401 path DID fire and call _try_refresh_copilot_client_credentials(), but that method only re-resolved the stable raw ghu_ token and rebuilt the client — it never evicted the cached exchanged JWT or forced a fresh exchange, so the retry put the SAME expired token back on the wire, 401'd again, and the single-shot guard aborted the turn. Fix: force a fresh IDE-token exchange (evict cached JWT via evict_cached_exchanged_token + re-mint via get_copilot_api_token) before the client rebuild, mirroring the merged auxiliary-path recovery (#59837) and the 400 recovery in this same PR. Graceful fallback to the resolved token if the exchange endpoint is unreachable; picks up the enterprise base_url on re-exchange. Brings main-loop clean-401 recovery to parity with the merged auxiliary path (#59837), using the newer on-disk-aware evict helper. Companion context: #58743 (this PR, expanded), #51313, #63204 (which assumed the 401 path already recovered — it reached the method but the method was too weak). Tests: exchange retry/persist round-trip, restart-blip disk reuse, stale-cred 400 classifier, 400 recovery, and 3 new 401 cases (fresh exchanged token on the wire; network-blip fallback to resolved token). 58 copilot tests green on current main.
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…oken 401 Copilot degrades in two related ways that both abort a turn as non-retryable and only clear on a gateway restart (a cold process re-runs the token exchange): 1. HTTP 400 model_not_available_for_integrator / model_not_supported — a raw/degraded token routes to the restricted copilot-language-server integrator whose allowlist omits enterprise-only models (e.g. claude-opus-4.8). Because it is a 400 (not 401), the existing 401 refresh path never fired. Prevented (retry-with-backoff exchange + on-disk JWT persistence + header guard at the client chokepoint) and self-healed at runtime (single-shot forced re-exchange + client rebuild + retry before fallback). 2. HTTP 401 'IDE token expired: unauthorized: token expired' — the short-TTL *exchanged* IDE token expires mid-turn. The clean-401 path DID fire and call _try_refresh_copilot_client_credentials(), but that method only re-resolved the stable raw ghu_ token and rebuilt the client — it never evicted the cached exchanged JWT or forced a fresh exchange, so the retry put the SAME expired token back on the wire, 401'd again, and the single-shot guard aborted the turn. Fix: force a fresh IDE-token exchange (evict cached JWT via evict_cached_exchanged_token + re-mint via get_copilot_api_token) before the client rebuild, mirroring the merged auxiliary-path recovery (NousResearch#59837) and the 400 recovery in this same PR. Graceful fallback to the resolved token if the exchange endpoint is unreachable; picks up the enterprise base_url on re-exchange. Brings main-loop clean-401 recovery to parity with the merged auxiliary path (NousResearch#59837), using the newer on-disk-aware evict helper. Companion context: NousResearch#58743 (this PR, expanded), NousResearch#51313, NousResearch#63204 (which assumed the 401 path already recovered — it reached the method but the method was too weak). Tests: exchange retry/persist round-trip, restart-blip disk reuse, stale-cred 400 classifier, 400 recovery, and 3 new 401 cases (fresh exchanged token on the wire; network-blip fallback to resolved token). 58 copilot tests green on current main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Auxiliary tasks (compression, title generation, vision) now recover from expired OAuth credentials instead of 401-looping: auto-routed calls refresh the concrete backend's token, and a fallback candidate with a dead credential is refreshed-or-skipped rather than aborting the whole task. Fixes #20832, #23670; closes out the residual failure link from the mattalachia/Luisillo gpt-5.5 compaction debug dump.
Two commits:
1. Auto-route auth refresh (salvage of PR #20837 by @fanyangCS, authorship preserved — surgically reapplied, his branch predated the
_retry_same_provider_sync/asyncextraction).resolved_providerstays"auto"even after the client cache picks a concrete backend, so the auth-refresh retry gate never fired for auto routes. New_auth_refresh_provider_for_route()infers the backend from the client's base URL (Copilot/Codex/Anthropic/Nous), plus acopilotbranch in_refresh_provider_credentials(JWT-exchange cache eviction). This is the exact preflight-compression 401 from #23670.2. Stale fallback-candidate recovery (ours). The fallback call site had no auth handling: a candidate carrying an expired credential (e.g.
_try_anthropic()building a client from a deadANTHROPIC_TOKEN) 401'd and aborted the task with a 60s cooldown. In mattalachia's debug dump this fired 5 times in one session — Codex 120s timeout → Anthropic 401 → no summary ever → session overflowed to 296K → "Cannot compress further". Now:_call_fallback_candidate_sync/asyncrefresh the candidate's credentials and retry once; if unrefreshable, quarantine via_mark_provider_unhealthyand re-walk the chain so the next viable candidate serves. Non-auth errors propagate unchanged.Changes
agent/auxiliary_client.py:_auth_refresh_provider_for_route(), copilot refresh branch, auto-route eviction,_call_fallback_candidate_sync/asyncwired at both fallback call sites (sync + async)tests/agent/test_auxiliary_client.py: contributor's auto→Copilot tests (sync+async pattern), auto→Codex compression test (Preflight compression surfaces Codex OAuth 401 instead of falling back #23670),TestStaleFallbackCandidateSkip(refresh-retry / quarantine-skip / non-auth-raises)scripts/release.py: AUTHOR_MAP entryValidation
E2E replay of the mattalachia chain (codex timeout → expired-anthropic 401 → openrouter serves) via real
call_llmagainst temp HERMES_HOME.Infographic