fix(agent): check credential pool exhaustion before restoring primary runtime (#15298)

[Tranquil-Flow]

What does this PR do?

_restore_primary_runtime() checks a 60-second rate-limit timer (_rate_limited_until) but does not consult the credential pool's exhaustion state. After the 60-second timer expires, it attempts to restore the primary provider every turn even if the credential pool still marks that provider as exhausted. This burns retries and generates noise during extended outages.

Related Issue

Fixes #15298

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• run_agent.py — after the _rate_limited_until check in _restore_primary_runtime(), added a guard that consults self._credential_pool.has_available(). If the pool exists for the primary provider and reports all credentials exhausted, restoration is skipped. This integrates with the existing CredentialPool architecture (which already tracks per-credential exhaustion with cooldowns) rather than adding another independent timer.
• tests/run_agent/test_primary_runtime_restore.py — 4 new tests in TestCredentialPoolExhaustionGate

How to Test

1. pytest tests/run_agent/test_primary_runtime_restore.py::TestCredentialPoolExhaustionGate -v — 4 new tests cover:
   • Pool exhausted, same provider: restoration blocked
   • Pool has credentials: restoration proceeds
   • Pool for different provider: not blocked
   • No pool at all: not blocked
2. Tested on macOS (Python 3.11)

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS (Python 3.11)

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — or N/A
• I've updated cli-config.yaml.example if I added/changed config keys — or N/A
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A
• I've updated tool descriptions/schemas if I changed tool behavior — or N/A

Screenshots / Logs

N/A — see commit description and PR diff.

[teknium1]

Thanks for targeting the repeated primary-restore retry path. The pool-aware gap still exists on current main: after the fixed cooldown expires, agent/agent_runtime_helpers.py:1163-1199 restores the snapshot key and client; the pool is only considered when has_available() is true at :1220-1222.

Problems

• The new direct comparison at agent/agent_runtime_helpers.py:897-900 does not cover named custom providers. Custom pools are keyed custom:<name> while the agent runtime stays custom (agent/credential_pool.py:117-120). Current main resolves this safely through get_custom_provider_pool_key() at agent/agent_runtime_helpers.py:1235-1249.
• Current HEAD deliberately tests the inverse empty-pool behavior: tests/agent/test_restore_primary_pool_reselect.py:162-175 expects the snapshot key to be restored when no pool credential is available. This PR needs to reconcile that contract, not just add parallel mock coverage.

Suggested changes

• Rework the gate against the current helper and reuse its canonical custom-pool matching logic.
• Add a real exhausted-pool regression case, including custom:<name>, and update the existing empty-pool expectation if the intended contract is now to remain on fallback.

This is an automated hermes-sweeper review.

[teknium1]

This direct provider equality misses named custom pools: their pool key is custom:<name> while the runtime provider remains custom. Please reuse the canonical custom-pool matching approach now used in restore_primary_runtime so an exhausted named custom endpoint is also gated.