Skip to content

fix(pool): re-select from credential pool on primary runtime restore - #25206

Closed
jmmaloney4 wants to merge 1 commit into
NousResearch:mainfrom
jmmaloney4:fix/restore-primary-reselect-from-pool
Closed

fix(pool): re-select from credential pool on primary runtime restore#25206
jmmaloney4 wants to merge 1 commit into
NousResearch:mainfrom
jmmaloney4:fix/restore-primary-reselect-from-pool

Conversation

@jmmaloney4

@jmmaloney4 jmmaloney4 commented May 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #25205.

_restore_primary_runtime restores api_key from a construction-time snapshot. When the credential pool has rotated (exhaustion, revocation, rate-limit cooldown), the snapshot key is stale. The next turn immediately hits the same error, re-exhausts entries, and falls through to cross-provider fallback.

After restoring the snapshot, re-select from the credential pool if one exists. Falls back gracefully to the snapshot key when the pool is absent or empty.

Test plan

  • 7 new test cases in test_restore_primary_pool_reselect.py
    • Rotation picks new entry (not stale snapshot key)
    • Pool absent → snapshot key used (existing behavior)
    • Pool empty (all exhausted) → snapshot key used
    • Entry with empty key → snapshot key used
    • Base URL updated from pool entry
    • Client rebuild triggered after reselect
  • 53 existing credential pool tests pass
  • Manual: run multi-turn session with codex pool, verify restore log shows "Restore re-selected pool entry" and fallback does NOT activate on next turn

Risks

  • Low: pool.select() is idempotent and already used at session init. The restore path just calls it again.
  • If select() fails or returns None, we fall back to the snapshot key (same as pre-fix behavior).

@jmmaloney4
fix(pool): re-select from credential pool on primary runtime restore
d727ede 
_restore_primary_runtime restores api_key from a construction-time
snapshot.  When the credential pool has rotated (due to exhaustion,
revocation, or rate-limit cooldown) the snapshot key is stale.  The
next turn immediately hits the same error, re-exhausts entries, and
falls through to cross-provider fallback.

Fix: after restoring the snapshot, re-select from the credential pool
if one exists.  Falls back gracefully to the snapshot key when the pool
is absent or empty.

Tests: 7 new cases covering rotation, empty pool, no pool, missing
key, base_url update, and client rebuild.

Related: NousResearch#24159
@jmmaloney4

jmmaloney4 commented May 13, 2026

Copy link
Copy Markdown
Contributor Author

/gemini review

@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels May 13, 2026
This was referenced May 13, 2026
Closed
Closed
@teknium1 teknium1 mentioned this pull request Jun 28, 2026
Merged
teknium1 pushed a commit that referenced this pull request Jun 28, 2026
@jmmaloney4 @teknium1
fix(pool): re-select from credential pool on primary runtime restore
759474a 
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.

After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the #33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.

Salvaged from #25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.

Fixes #25205
teknium1 pushed a commit that referenced this pull request Jun 28, 2026
@jmmaloney4 @teknium1
fix(pool): re-select from credential pool on primary runtime restore
3703a83 
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.

After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the #33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.

Salvaged from #25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.

Fixes #25205
teknium1 pushed a commit that referenced this pull request Jun 28, 2026
@jmmaloney4 @teknium1
fix(pool): re-select from credential pool on primary runtime restore
677f8da 
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.

After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the #33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.

Salvaged from #25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.

Fixes #25205
teknium1 pushed a commit that referenced this pull request Jun 28, 2026
@jmmaloney4 @teknium1
fix(pool): re-select from credential pool on primary runtime restore
f0de4c6 
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.

After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the #33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.

Salvaged from #25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.

Fixes #25205
@teknium1

teknium1 commented Jun 28, 2026

Copy link
Copy Markdown
Collaborator

Merged via #53913 — your fix was cherry-picked onto current main with your authorship preserved in git history (commit f0de4c6, "fix(pool): re-select from credential pool on primary runtime restore").

The restore logic had moved from the monolithic method in run_agent.py into agent/agent_runtime_helpers.py:restore_primary_runtime() since you opened this, so the salvage rebased onto that and collapsed the inlined client-rebuild onto the existing _swap_credential helper (which already carries the base_url #33163 / OAuth-detection fixes) — same behavior, less duplication. Your 7 tests came along (one assertion updated to reason="credential_rotation" to match the helper). Verified E2E on a real AIAgent + real CredentialPool: a revoked entry no longer gets restored from the snapshot.

Thanks for the clear repro and the fix.

@teknium1 teknium1 closed this Jun 28, 2026
pai-scaffolde pushed a commit to Scaffolde/hermes-agent that referenced this pull request Jun 28, 2026
@jmmaloney4 @pai-scaffolde
fix(pool): re-select from credential pool on primary runtime restore
02ee6bf 
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.

After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the NousResearch#33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.

Salvaged from NousResearch#25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.

Fixes NousResearch#25205
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
@jmmaloney4 @teknium1
fix(pool): re-select from credential pool on primary runtime restore
688e9d2 
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.

After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the NousResearch#33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.

Salvaged from NousResearch#25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.

Fixes NousResearch#25205
Jasper6439 pushed a commit to Jasper6439/hermes-agent that referenced this pull request Jul 5, 2026
@jmmaloney4 Jasper6439
fix(pool): re-select from credential pool on primary runtime restore
7fd0cc2 
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.

After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the NousResearch#33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.

Salvaged from NousResearch#25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.

Fixes NousResearch#25205
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
@jmmaloney4 @teknium1
fix(pool): re-select from credential pool on primary runtime restore
888a070 
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.

After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the NousResearch#33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.

Salvaged from NousResearch#25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.

Fixes NousResearch#25205
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
@jmmaloney4 @teknium1
fix(pool): re-select from credential pool on primary runtime restore
6fde5ff 
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.

After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the NousResearch#33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.

Salvaged from NousResearch#25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.

Fixes NousResearch#25205
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
@jmmaloney4 @teknium1
fix(pool): re-select from credential pool on primary runtime restore
8983b15 
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.

After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the NousResearch#33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.

Salvaged from NousResearch#25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.

Fixes NousResearch#25205
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
@jmmaloney4 @teknium1
fix(pool): re-select from credential pool on primary runtime restore
ab5cc18 
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.

After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the NousResearch#33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.

Salvaged from NousResearch#25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.

Fixes NousResearch#25205
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
@melon-xf
fix(pool): re-select from credential pool on primary runtime restore
d53295a 
_restore_primary_runtime restored the construction-time api_key snapshot and
never consulted the credential pool. After the pool rotated away from a
revoked/exhausted entry mid-session, every new turn restored the dead key,
re-failed instantly, burned the remaining entries, and fell through to
cross-provider fallback.

After restoring the snapshot, re-select the pool's current best entry and
swap the live credential in via _swap_credential (which already rebuilds the
OpenAI/Anthropic client, reapplies base-url headers, and carries the NousResearch#33163
base_url / OAuth-detection fixes). Falls back to the snapshot key when the
pool is absent, empty, or the entry has no usable key.

Salvaged from NousResearch#25206 onto current main: the original targeted the pre-refactor
monolithic method in run_agent.py; the logic now lives in
agent/agent_runtime_helpers.py and is collapsed onto _swap_credential instead
of re-inlining the client rebuild.

Fixes NousResearch#25205
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P1 High — major feature broken, no workaround type/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug] _restore_primary_runtime bypasses credential pool, reuses stale (revoked) api_key

3 participants

@jmmaloney4 @teknium1 @alt-glitch