Skip to content

fix: emit status notification on credential pool rotation - #10718

Closed
nightq wants to merge 1 commit into
NousResearch:mainfrom
nightq:fix/issue-10476-credential-pool-notification
Closed

fix: emit status notification on credential pool rotation#10718
nightq wants to merge 1 commit into
NousResearch:mainfrom
nightq:fix/issue-10476-credential-pool-notification

Conversation

@nightq

@nightq nightq commented Apr 16, 2026

Copy link
Copy Markdown

Summary

Fixes inconsistent notification behavior when credential pool rotation causes a provider switch.

Root Cause

When fallback_model triggers a provider switch, Hermes correctly notifies the user via _emit_status(). However, when credential pool exhaustion causes the same provider switch, no notification was sent. The user had no idea they were talking to a different AI.

Fix

Add _emit_status() calls in _recover_with_credential_pool() when credentials are rotated:

  • Billing exhaustion (402): "🔄 Credential exhausted (402) — rotated to: {entry_id}"
  • Rate limit (429): "🔄 Rate limited (429) — rotated to: {entry_id}"
  • Auth failure (401): "🔄 Auth failed (401) — rotated to: {entry_id}"

Also fixed _emit_status() to use getattr() for status_callback to handle edge cases where the attribute doesn't exist (e.g., in test agents).

Test Plan

  • All credential pool routing tests pass
  • Manual verification of status emission

Closes #10476

Fixes NousResearch#10476

Root cause: When credential pool exhaustion caused a provider switch,
no notification was sent to the user. This was inconsistent with the
fallback_model behavior which correctly notifies via _emit_status().

Fix: Add _emit_status() calls in _recover_with_credential_pool() when:
- Billing exhaustion (402) causes rotation
- Rate limit (429) causes rotation after retry
- Auth failure (401) causes rotation after refresh fails

Also fixed _emit_status() to use getattr() for status_callback to handle
test agents that don't have this attribute set.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels Apr 25, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the focused report and implementation. This is an automated hermes-sweeper review; the linked fallback-notification behavior is now implemented on current main.

  • agent/conversation_loop.py:3166-3207 escalates rate-limit/billing failures to the configured fallback chain when credential-pool recovery cannot proceed.
  • agent/chat_completion_helpers.py:1544-1557 records a durable old-provider/model → new-provider/model notification whenever fallback activates.
  • agent/conversation_loop.py:5065-5070 emits that notification after successful recovery, before clearing transient retry status; run_agent.py:984-1002 delivers it exactly once through _emit_status().
  • tests/run_agent/test_retry_status_buffer.py:138-163 covers the successful fallback-notice path.
  • This landed in 3fe7f6d27a005a0a75644c1d1d42cef8c8750c5f (fix: preserve fallback switch notice on successful fallback). The reviewed PR's original run_agent.py recovery body is now a forwarder to agent/agent_runtime_helpers.py:696-954.

Current main intentionally keeps successful same-provider credential rotations as logged retry recovery rather than immediate lifecycle chatter; the durable cross-provider/model switch requested by #10476 is surfaced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P2 Medium — degraded but workaround exists sweeper:implemented-on-main Sweeper: behavior already present on current main type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Credential pool provider fallback is silent — inconsistent with fallback_model notification behavior

3 participants