fix(copilot): recover from stale/degraded token 400 AND expired IDE-token 401 (salvage of #58743) - #75864
Merged
Conversation
…oken 401 Copilot degrades in two related ways that both abort a turn as non-retryable and only clear on a gateway restart (a cold process re-runs the token exchange): 1. HTTP 400 model_not_available_for_integrator / model_not_supported — a raw/degraded token routes to the restricted copilot-language-server integrator whose allowlist omits enterprise-only models (e.g. claude-opus-4.8). Because it is a 400 (not 401), the existing 401 refresh path never fired. Prevented (retry-with-backoff exchange + on-disk JWT persistence + header guard at the client chokepoint) and self-healed at runtime (single-shot forced re-exchange + client rebuild + retry before fallback). 2. HTTP 401 'IDE token expired: unauthorized: token expired' — the short-TTL *exchanged* IDE token expires mid-turn. The clean-401 path DID fire and call _try_refresh_copilot_client_credentials(), but that method only re-resolved the stable raw ghu_ token and rebuilt the client — it never evicted the cached exchanged JWT or forced a fresh exchange, so the retry put the SAME expired token back on the wire, 401'd again, and the single-shot guard aborted the turn. Fix: force a fresh IDE-token exchange (evict cached JWT via evict_cached_exchanged_token + re-mint via get_copilot_api_token) before the client rebuild, mirroring the merged auxiliary-path recovery (#59837) and the 400 recovery in this same PR. Graceful fallback to the resolved token if the exchange endpoint is unreachable; picks up the enterprise base_url on re-exchange. Brings main-loop clean-401 recovery to parity with the merged auxiliary path (#59837), using the newer on-disk-aware evict helper. Companion context: #58743 (this PR, expanded), #51313, #63204 (which assumed the 401 path already recovered — it reached the method but the method was too weak). Tests: exchange retry/persist round-trip, restart-blip disk reuse, stale-cred 400 classifier, 400 recovery, and 3 new 401 cases (fresh exchanged token on the wire; network-blip fallback to resolved token). 58 copilot tests green on current main.
- Bound ALL reads of the on-disk JWT store through one _read_jwt_store() helper (load, eviction, save-merge) — the 1 MiB cap previously only covered the load path; eviction and save could still parse an oversized/corrupt store and rewrite it back out (sweeper finding). - Fix the class, not the site: the recovery gates checked the literal provider == "copilot" while /model and profile configs can leave the alias spelling in place (the reporter's own log shows provider=copilot AND provider=github-copilot in one session — the aliased turns would have silently skipped recovery). Single owner: AIAgent._is_copilot_provider() (slug aliases + Copilot base-URL fallback), used by both run_agent recovery methods and both conversation_loop gates. - Update the salvaged 401 test to current main's client-retirement contract (release deferred to GC — no synchronous .close()). - Add copilot_stale_cred_retry_attempted to the TurnRetryState field contract test; add bounded-store and alias-gate regression tests.
Matches the repo-wide convention (c89481d); unblocks ruff enforcement and the Windows footgun checker on the salvaged JWT persistence code.
Contributor
૮ >ﻌ< ა ci reviewran on bdf4027 ℹ️ InfoDesktop E2E visual evidence · View test artifacts · View job3 visual diffs. inline evidence upload failed. Failed to upload diff-1508682a2ae8-boot-ready-diff.png with gh image (exit code 1): Error uploading /home/runner/work/_temp/e2e-evidence/diff-1508682a2ae8-boot-ready-diff.png: step 0 (get upload token): uploadToken not found on repo page — do you have write access to NousResearch/hermes-agent? (or, if NousResearch enforces SAML SSO, authorize at https://github.com/orgs/NousResearch/sso) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(copilot): recover from stale/degraded token 400 AND expired IDE-token 401 (salvage of #58743)
Salvage of #58743 by @dstkwll (authorship preserved via cherry-pick), rebased onto current main with follow-up hardening.
Summary
GitHub Copilot degrades in two related ways that both abort a turn as a non-retryable error and only clear on a full restart. This PR prevents the degraded state and self-heals from both at runtime — fixing the widely-reported "most advertised models return HTTP 400 while gpt-4.1/gpt-4o work" failure.
Bug 1 — HTTP 400
model_not_available_for_integrator/model_not_supported. When the raw→API token exchange fails (startup racing network readiness is the common trigger),get_copilot_api_token()silently falls back to the rawghu_OAuth token and that degraded state is cached for the process lifetime. A raw token routes requests to the restrictedcopilot-language-serverintegrator whose model allowlist omits most models. Because it's a 400 — not a 401 — no refresh path ever fired.Bug 2 — HTTP 401
IDE token expired. The short-TTL exchanged token expires mid-turn; the 401 refresh path re-resolved the (stable) raw token and rebuilt the client but never evicted the cached exchanged JWT, so the retry replayed the same expired token.Changes
Contributor commit (@dstkwll):
hermes_cli/copilot_auth.py: exchange retry-with-backoff (3 attempts); persist last-good exchanged JWT to disk (.copilot_jwt.json, 0600, profile-aware, expired entries pruned);evict_cached_exchanged_token()drops both cache tiers.agent/conversation_loop.py+agent/turn_retry_state.py+run_agent.py: narrow 400 classifier (status 400AND integrator/model_not_supportedmarker) → single-shot copilot-scoped force-re-exchange + client rebuild + retry; 401 recovery now evicts + re-exchanges before rebuild.agent/agent_runtime_helpers.py: defense-in-depth Copilot header guard at thecreate_openai_client()chokepoint (fills missing headers only, never overrides).agent/credential_pool.py: WARNING when the seed degrades to the raw token; skip the generic env-seeding loop for copilot (its dedicated branch is authoritative).Follow-ups (ours):
_read_jwt_store()helper (load, eviction, save-merge) — the 1 MiB cap previously covered only the load path (sweeper finding).provider == "copilot"while/modeland profile configs can leavegithub-copilot/githubalias spellings in place (the reporter's own log shows both spellings in one session). Single ownerAIAgent._is_copilot_provider()(aliases + Copilot base-URL fallback) used by all four gate/guard sites.TurnRetryStatecontract updated; salvaged 401 test updated to current main's client-retirement contract.Validation
Fixes the failure reported in Discord (GitHub Copilot OAuth: most advertised models 400). Closes #58743.
Infographic