Skip to content

fix(agent): consult credential pool in _restore_primary_runtime to avoid stale snapshot key - #25730

Closed
dusterbloom wants to merge 1 commit into
NousResearch:mainfrom
dusterbloom:fix/credential-pool-bypass-restore-primary
Closed

dusterbloom wants to merge 1 commit into
NousResearch:mainfrom
dusterbloom:fix/credential-pool-bypass-restore-primary

Conversation

@dusterbloom

Copy link
Copy Markdown
Contributor

Summary

Fixes #25205

_restore_primary_runtime() blindly restores api_key from the init-time snapshot, bypassing any credential pool rotation that occurred during the previous turn. After a 401/429/402 triggers pool rotation, the next turn restores the exhausted key, immediately fails again, and cascades into unnecessary cross-provider fallback.

Root Cause

  1. Init: resolve_runtime_provider() selects credential A → snapshotted in _primary_runtime["api_key"]
  2. Turn N: API returns 401 → pool marks A exhausted → _swap_credential() updates self.api_key to credential B → but _primary_runtime is never updated
  3. Turn N+1: _restore_primary_runtime() restores stale credential A from snapshot → immediate 401 again

The same bug exists in _try_recover_primary_transport().

Fix

Both _restore_primary_runtime() and _try_recover_primary_transport() now consult the credential pool after restoring snapshot state:

pool_entry = pool.current() or pool.select()
if pool_entry is not None:
    self._swap_credential(pool_entry)  # fresh key + client rebuild
else:
    # fall back to snapshot behavior (no pool available)
  • pool.current() is tried first (same entry used last turn)
  • pool.select() is the fallback (picks best available)
  • Exceptions from pool access are caught gracefully — falls back to snapshot
  • When no pool exists, behavior is unchanged

Testing

  • All 31 existing test_primary_runtime_restore.py tests pass
  • All 47 test_credential_pool.py tests pass
  • 4 new tests in TestCredentialPoolBypass:
    • test_restore_uses_pool_current_entry — verifies pool entry used instead of snapshot
    • test_restore_falls_back_to_snapshot_when_no_pool — no-pool backward compat
    • test_restore_handles_pool_exception_gracefully — pool error → snapshot fallback
    • test_restore_prefers_current_over_selectcurrent() called first

…oid stale snapshot key (NousResearch#25205)

_restore_primary_runtime() blindly restores api_key from the init-time
snapshot, bypassing any credential pool rotation that occurred during
the previous turn. After a 401/429/402 triggers pool rotation, the next
turn restores the exhausted key, immediately fails again, and cascades
into unnecessary fallback.

Now both _restore_primary_runtime and _try_recover_primary_transport
consult pool.current() / pool.select() after restoring snapshot state.
When a pool entry is available, _swap_credential applies it (rebuilding
the client with the fresh key). Falls back to snapshot behavior when
no pool exists or pool access fails.

Closes NousResearch#25205
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/auth Authentication, OAuth, credential pools labels May 14, 2026
@alt-glitch

Copy link
Copy Markdown
Contributor

Competing fix: PR #25206 also targets the same _restore_primary_runtime stale-key issue (#25205). Both consult the credential pool after snapshot restore. #25733 takes a broader approach (centralized snapshots + align_current_to_runtime). These three PRs overlap significantly on the restore path — likely only one should land.

Related: #15434 (same function, cooldown check) and #25277 (pool in switch_model).

@dusterbloom

Copy link
Copy Markdown
Contributor Author

Closing as stale: branch has fallen behind main and is conflicting, with no active review. Will reopen with a clean rebase if the change is still wanted.

@teknium1

Copy link
Copy Markdown
Collaborator

Closing as a duplicate of #25206 (by @jmmaloney4, submitted first, who also reported the underlying issue #25205). Both PRs fixed the same bug — _restore_primary_runtime restoring a stale credential snapshot instead of consulting the credential pool.

Merged via #53913 (commit f0de4c6 on main). Thanks for the contribution and for independently catching this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/auth Authentication, OAuth, credential pools comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P1 High — major feature broken, no workaround type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] _restore_primary_runtime bypasses credential pool, reuses stale (revoked) api_key

3 participants