fix(copilot): recover from stale/degraded token 400 AND expired IDE-token 401

[dstkwll]

Summary

GitHub Copilot degrades in two related ways that both abort a turn as a non-retryable error and only clear on a gateway restart (a cold process re-runs the token exchange). This PR prevents the degraded state and self-heals from both at runtime. It has been rebased onto current main and its scope expanded from the original 400-only fix to also cover the clean-401 case.

Bug 1 — HTTP 400 model_not_available_for_integrator / model_not_supported

A raw/degraded token routes the request to the restricted copilot-language-server integrator, whose allowlist omits Enterprise-only models (most visibly claude-opus-4.8). Because it is a 400, not a 401, the existing Copilot 401 credential-refresh path never fired and the turn aborted.

Deterministic truth table (verified live against the API) — varying only the token on the wire and whether Copilot-Integration-Id: vscode-chat is present:

Token on wire	vscode-chat header	Result
exchanged (~437-char API token)	present	OK
raw ghu_ (40-char OAuth)	present	OK
raw ghu_ (40-char OAuth)	absent	copilot-language-server 400 ← the bug
exchanged (~437-char)	absent	400 missing Editor-Version (different error)

The fallback integrator is only reached when a raw token and a missing integration header coincide. Two mechanisms produce that state: (1) the token exchange degrades silently to the raw token on any exception and — because the exchanged-JWT cache was in-process only — sticks for the whole process lifetime; (2) two client-rebuild paths (primary_recovery, restore_primary) reconstruct the client from the _primary_runtime snapshot without re-applying Copilot headers.

Bug 2 — HTTP 401 IDE token expired: unauthorized: token expired

The short-TTL exchanged IDE token (minted from the stable raw ghu_ token) expires mid-turn. A heavy/long turn whose request straddles that expiry gets a clean 401. The clean-401 path did fire and call _try_refresh_copilot_client_credentials() — but that method only re-resolved the stable raw token and rebuilt the client; it never evicted the cached exchanged JWT or forced a fresh exchange. So the retry put the same expired IDE token back on the wire, 401'd again, and the single-shot guard aborted the turn as non-retryable. Only a gateway restart helped, because a cold process re-runs the exchange.

This is the same failure class the merged auxiliary-path fix (#59837, closing #20832/#20837/#23379) already solved for compression/title-generation — but the main conversation loop was left with the weaker refresh. Note #63204 was closed implemented_on_main on the assumption that "conversation_loop.py routes Copilot 401 recovery through _try_refresh_copilot_client_credentials()" — it reaches the method, but the method was too weak to actually refresh the expired exchanged JWT. This PR makes the method do what that review already assumed it did.

The fix (layered — prevent + recover)

hermes_cli/copilot_auth.py

• exchange_copilot_token(): retry-with-backoff (3 attempts) instead of failing on the first blip.
• Persist the last-good exchanged JWT to disk (~/.hermes/.copilot_jwt.json, 0o600, profile-aware, expired entries pruned, read bounded to 1 MiB). A fresh process reuses the still-valid ~30-min token before any network call.
• evict_cached_exchanged_token(): drop both cache tiers so a recovery can force a fresh mint.

agent/credential_pool.py — WARNING when the copilot seed degrades to the raw token, so a recurrence is visible instead of silent.

agent/agent_runtime_helpers.py — defense-in-depth header guard at create_openai_client() (the documented single chokepoint every primary client passes through). For githubcopilot.com hosts it fills any missing Copilot headers (never overrides caller-set ones; operates on the local per-call dict copy, so it never mutates _client_kwargs and can't break prompt caching).

agent/conversation_loop.py + agent/turn_retry_state.py + run_agent.py

• 400 recovery: classify the integrator / model_not_supported 400 as a refreshable stale-credential error (narrow match: 400 and a specific body marker), then — copilot-scoped and single-shot — force a fresh exchange, rebuild the client, and retry once on the same provider before falling through to the fallback chain. Aborts cleanly if the exchange stays degraded, so a genuinely unavailable model can't loop.
• 401 recovery (new): _try_refresh_copilot_client_credentials() now evicts the cached exchanged JWT and forces a fresh exchange (via evict_cached_exchanged_token + get_copilot_api_token) before rebuilding the client, so the retry carries a valid IDE token. Falls back to the resolved token if the exchange endpoint is unreachable; picks up the Enterprise base_url on re-exchange. The clean-401 path is already single-shot-guarded (copilot_auth_retry_attempted).

Prior art appropriated (credit)

• Auxiliary-path Copilot 401 refresh (JWT-exchange-cache eviction + fresh exchange): pattern from fix(auxiliary): auth recovery for auto routes and stale fallback candidates #59837 (salvage of fix(auxiliary): refresh auto-routed Copilot credentials on 401 #20837 by @fanyangCS). This PR brings the main conversation loop to parity, using the newer on-disk-aware evict_cached_exchanged_token helper the auxiliary path predates.
• Runtime stale-credential-400 classification + refresh: pattern from fix(auxiliary_client): treat Copilot 400 model_not_supported as a refreshable stale credential #51313 (extended from the auxiliary path to the main loop, and to the model_not_available_for_integrator marker).
• Bounded auth JSON reads: fix(copilot): bound auth JSON response reads #54750.
• SDK header-attribute gotcha awareness (_custom_headers vs _default_headers): fix(primary): preserve routed client headers on auto-routed provider init #9065 — our guard sidesteps it by reconstructing headers rather than reading them back off a live client.

Testing

• Targeted assertions: exchange retry/persist round-trip, restart-blip disk reuse (zero network), bounded oversized-file read, the stale-credential classifier (integrator-400 & model_not_supported → true; wrong-model 400, 401, 500 → false), 400 recovery (aborts when still degraded; rebuilds + re-exchanges on success), and 3 new 401 cases (fresh exchanged token — not the raw token — goes on the wire; network-blip fallback to the resolved token; token-unchanged still rebuilds).
• 58 copilot tests green on current main; existing suites unaffected (test_copilot_token_exchange, test_copilot_auth, test_credential_pool, test_create_openai_client_reuse, test_turn_retry_state).

No new HERMES_* config env vars; no change-detector tests; prompt caching, role alternation, and the caller's _client_kwargs are all preserved.

Related

• Companion (merged): fix(auxiliary): auth recovery for auto routes and stale fallback candidates #59837 (auxiliary-path recovery), which this brings to main-loop parity.
• Same family (open): fix(auxiliary_client): treat Copilot 400 model_not_supported as a refreshable stale credential #51313, fix(copilot): refresh base_url via live token exchange during credential rotation #62689, fix(runtime): exchange raw Copilot OAuth token in runtime resolution #58830, [Bug]: I always get HTTP 400: Bad Request with Github Copilot provider #45813 (400 bug report).
• Supersedes the concern in the closed fix(run): apply Copilot-Integration-Id header on codex_responses credential refresh to fix integrator routing (#63188) #63204 (401 path reached the method but the method was too weak).

[teknium1]

Thanks for tracing the 400 path. The premise is present on current main: agent/conversation_loop.py:2851-2859 only refreshes Copilot on 401, while hermes_cli/copilot_auth.py:415-434 falls back to the raw token when exchange fails.

Problems

• hermes_cli/copilot_auth.py:338 and :399 parse the persisted JWT cache with unbounded read_text(). The proposed 1 MiB check exists only in _load_jwt_from_disk() at :371-374, so recovery eviction and a later save can still load an oversized cache.
• The PR changes no test file (gh api repos/NousResearch/hermes-agent/pulls/58743/files?per_page=100), leaving the new persistence and recovery paths without committed regression coverage.

Suggested changes

• Use one bounded cache-read helper for load, eviction, and save; discard oversized/malformed content before parsing or rewriting.
• Add focused network-free tests for retry/disk reuse, oversized-cache handling, 400 classification, and both recovery outcomes.

Automated hermes-sweeper review.

[teknium1]

The 1 MiB guard is only in _load_jwt_from_disk(). Recovery reaches this unbounded read_text() path, so an oversized cache can still be fully loaded; share a bounded-read helper with eviction and _save_jwt_to_disk().

[dstkwll]

Rebased onto current main (was ~89 commits behind; now MERGEABLE) and expanded scope to also fix the clean HTTP 401 IDE token expired case in the main conversation loop.

The 401 path already reached _try_refresh_copilot_client_credentials(), but that method only re-resolved the stable raw ghu_ token and rebuilt the client — it never evicted the cached exchanged JWT or forced a fresh exchange, so the retry put the same expired IDE token back on the wire and the turn aborted as non-retryable. It now evicts + re-exchanges before rebuild, bringing the main loop to parity with the merged auxiliary-path fix (#59837). 3 new 401 tests added; 58 copilot tests green on current main.

[teknium1]

Merged via PR #75864 (#75864) — your commit was cherry-picked onto current main with your authorship preserved in git log (7779409). Thanks for the deterministic truth-table investigation and the layered prevent+recover design; we added small follow-ups on top (one bounded reader for all JWT-store reads, alias-spelling coverage for the provider gates, UTF-8 encoding per a new repo-wide lint rule). This fixes the widely-reported "most Copilot models return HTTP 400 while gpt-4.1/gpt-4o work" failure. Great work!