Repository navigation
credential pool: adopt the rotated token when the agent's stale key matches no entry (xai-oauth fallback 2026-09-19) - #730
Merged
Conversation
…atches no entry Incident 2026-09-19 03:16 PDT: a long-lived gateway session fell back xai-oauth/grok-4.6 -> claude-apx-1/claude-opus-5 on a 403 while a cron session succeeded on the same provider three seconds later. The pool held a valid, freshly rotated pair; the agent held the pre-rotation access token in agent.api_key (xAI refresh tokens are single-use, so another process had legitimately rotated the pair underneath it). Recovery path failure: the stale key matched no pool entry, so try_refresh_matching() returned None (no refresh attempted), the unmatched rotation branch hit the single-entry escape and returned None, and the fallback fired despite a usable token sitting in the pool. Fix (agent/credential_pool.py): - New _adopt_rotated_entry_unlocked(): when the failing key matches no entry but an entry holds a DIFFERENT, unexpired token, adopt it for one retry (xai-oauth entries are resynced from the auth store first). Nothing is marked exhausted; the unmatched-rotation streak is NOT reset, so a second unmatched failure still trips the existing bound and lets the fallback proceed. - Single-entry escape in mark_exhausted_and_rotate() now tries adoption before giving up. - try_refresh_matching(): when the id-bound entry already holds a different token than the hint, adopt it instead of force-refreshing (which would burn the fresh single-use refresh token). Tests: new tests/agent/test_credential_pool_adopt_rotated_entry.py (8 tests; 3 fail on fork/main without the fix) and test_unmatched_key_does_not_retry_only_pool_entry updated to the new bounded adopt-once contract. tests/agent -k "credential or auth or pool or refresh or fallback": 934 passed; tests/run_agent -k "credential or pool or auth": 135 passed.
Kyzcreig
enabled auto-merge
September 19, 2026 11:11
૮ >ﻌ< ა ci reviewrunning on 441b6f4 — credential pool: adopt the rotated token when the agent's st Still running 13 jobs: ❌ Job failuresPython tests / Run tests slice 3/16 · View jobJob Python tests / Run tests slice 3/16 failed.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Incident
2026-09-19 03:16 PDT, Discord banner
🔄 Model fallback (auth refresh): xai-oauth/grok-4.6 → claude-apx-1/claude-opus-5.A long-lived gateway session 403'd on xai-oauth at 03:49:52 while a cron session succeeded on grok-4.6 at 03:49:55.
The pool held a valid, freshly rotated pair; the agent held the pre-rotation access token in
agent.api_key(xAI refresh tokens are single-use, so another process had legitimately rotated the pair).
Root cause (recovery path)
try_refresh_matching()returnedNone(no refresh attempted)NoneFix —
agent/credential_pool.py_adopt_rotated_entry_unlocked()(new): failing key matches no entry, but an entry holds a different, unexpired token → adopt it for one retry. xai-oauth entries are resynced from the auth store first. Nothing is marked exhausted; the unmatched-rotation streak is not reset, so a second unmatched failure still trips the existing> max(available_count,1)bound and fallback proceeds.mark_exhausted_and_rotate()single-entry escape tries adoption before giving up.try_refresh_matching(): id-bound entry already holds a different token than the hint → adopt instead of force-refreshing (a force refresh would burn the fresh single-use refresh token).Multi-entry behavior unchanged. Nous-style wrapper keys (hint never matches by design) get exactly one extra bounded retry in single-entry pools before fallback.
Tests
tests/agent/test_credential_pool_adopt_rotated_entry.py— 8 tests; 3 fail on fork/main without the fix, pass with it.test_unmatched_key_does_not_retry_only_pool_entryupdated to the bounded adopt-once contract (docstring cites the incident).tests/agent -k "credential or auth or pool or refresh or fallback"→ 934 passed, 2 skippedtests/run_agent -k "credential or pool or auth"→ 135 passedLayer 2 (proactive
xai-token-keeper+ launchd) and Layer 3 (provider parity across profiles + lint) land separately as fleet scripts, not in this repo.