fix(agent): notify gateway users when credential pool auth exhaustion causes provider switch

[aangelinsf]

What does this PR do?

When a provider fails with repeated 401/403 errors, Hermes marks its credentials as "exhausted"
in the credential pool and silently falls back to the next available provider — with no notification
to the user. In a gateway deployment (Telegram, Discord, etc.), this means the user unknowingly
receives responses from a completely different AI provider, potentially with different capabilities,
different privacy terms, and unexpected cost implications.

This is a behavioral inconsistency: the configured fallback_model path already calls
_emit_status() when it switches providers, which propagates to all gateway platforms. The
credential pool exhaustion path — which produces the same end-user outcome — does not.

This PR fixes the inconsistency by calling _emit_status() in _recover_with_credential_pool()
when mark_exhausted_and_rotate() returns None (all credentials exhausted, rotation impossible).
Same-provider key rotation — where a new key for the same provider is swapped in transparently —
remains silent, as that is legitimately low-signal infrastructure behavior the user doesn't need
to know about.

The message is written in plain language: it names the failing provider, avoids exposing raw HTTP
status codes or internal jargon, and gives an actionable recovery command.

Related Issue

Fixes #10476

Note: PR #10058 (closed 2026-04-15) addressed a related issue and contained a working
implementation of the notification. That PR was closed due to disagreement over the 24-hour
cooldown duration change it also included. This PR implements only the notification change,
which stands on its own regardless of cooldown policy.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• run_agent.py — In _recover_with_credential_pool(), added _emit_status() call in the
  FailoverReason.auth branch after mark_exhausted_and_rotate() returns None. The message
  includes the provider label and the recovery command (hermes auth reset <provider>). The call
  is placed after rotation fails, so same-provider key rotation (where next_entry is not None)
  continues silently.

• tests/agent/test_credential_pool_routing.py — Added TestAuthExhaustionNotification class
  with two tests:

  • test_emits_notification_when_all_credentials_exhausted — verifies _emit_status is called
    once with a message containing the provider name, HTTP status, and hermes auth reset
    when rotation returns None
  • test_silent_when_rotation_to_next_credential_succeeds — verifies _emit_status is not
    called when rotation succeeds (same-provider key swap), preserving the silent behavior for
    that path

How to Test

Unit tests (fast):

pytest tests/agent/test_credential_pool_routing.py::TestAuthExhaustionNotification -v

Manual gateway verification:

1. Configure Hermes with a primary provider using a deliberately invalid API key (e.g. set
   KIMI_API_KEY=sk-invalid in ~/.hermes/.env with provider: kimi-coding in config.yaml)
2. Ensure a fallback provider key is also present (e.g. OPENROUTER_API_KEY)
3. Send any message via the gateway (Telegram/Discord)
4. Before this fix: the response arrives silently from the fallback provider with no indication
   of the switch
5. After this fix: a notification is sent before the response:

   ⚠️ Primary AI (kimi-coding) is unavailable — the API key may be invalid or expired (HTTP 401).
   Switched to fallback if one is configured. To restore: check your API key and run
   `hermes auth reset kimi-coding`.
   

6. If the credential pool exhaustion state is cached from a previous session, clear it first
   with hermes auth reset <provider> before testing

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(agent):, test(agent):)
• I searched for existing PRs to make sure this isn't a duplicate (see PR fix: 24h cooldown for 401/403 auth failures + user notification #10058, closed)
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: macOS 15.4 (Sequoia), gateway deployment via Telegram

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — N/A (no user-visible features added)
• I've updated cli-config.yaml.example if I added/changed config keys — N/A (no new config keys)
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A (no architecture changes)
• I've considered cross-platform impact (Windows, macOS) — change is pure Python string formatting inside an existing method, no OS-specific code
• I've updated tool descriptions/schemas if I changed tool behavior — N/A (no tools changed)

Screenshots / Logs

Before (silent fallback — no notification sent to Telegram):

User:   Which LLM are you?
Hermes: I'm Claude Sonnet 4 (anthropic/claude-sonnet-4), running through OpenRouter.

After (notification emitted to gateway before response):

⚠️ Primary AI (kimi-coding) is unavailable — the API key may be invalid or expired (HTTP 401).
Switched to fallback if one is configured. To restore: check your API key and run
`hermes auth reset kimi-coding`.

User:   Which LLM are you?
Hermes: I'm Claude Sonnet 4 (anthropic/claude-sonnet-4), running through OpenRouter.

[mxnstrexgl]

🤖 Automated PR Review

Security Scan

✓ No security concerns
✓ Safe string formatting
✓ No credential leakage in messages

Code Quality

✓ Good UX - notifies users on auth failure
✓ Clear actionable message (hermes auth reset)
✓ Tests verify both notification and silent cases
✓ Correctly only notifies when rotation fails, not on successful rotation

Summary

Status: APPROVE

Clean fix. Good user experience improvement - gateway users now get clear notification when credential pool is exhausted.

---

Reviewed by Hermes Agent

[alt-glitch]

Competes with #10718 and #10716 for same fix (#10476). All three address credential pool exhaustion notification.

[alt-glitch]

Competes with #10718 and #10716 for same fix (#10476). All three address credential pool exhaustion notification.

[teknium1]

Thanks for the focused notification fix. This is an automated hermes-sweeper review; current main now provides the requested gateway-visible fallback notification through a later, broader implementation.

• agent/conversation_loop.py:3224 escalates unrecoverable auth failures to the configured fallback chain.
• agent/chat_completion_helpers.py:1544-1556 records a one-shot notice identifying the old and replacement model/provider whenever fallback activates.
• agent/conversation_loop.py:5065-5070 emits that notice on successful fallback before retry status is cleared; run_agent.py:875-893 routes it to gateway status_callback consumers.
• Commit 3fe7f6d27a005a0a75644c1d1d42cef8c8750c5f added this behavior and regression coverage in tests/run_agent/test_retry_status_buffer.py:138-164.

The PR's original insertion point is now only a forwarder to agent.agent_runtime_helpers, so its patch is superseded.