Skip to content

fix(agent): suppress intermediate retry status messages in chat thread - #18064

Closed
shagghiesuperstar wants to merge 3 commits into
NousResearch:mainfrom
shagghiesuperstar:fix/stream-retry-status-messages
Closed

fix(agent): suppress intermediate retry status messages in chat thread#18064
shagghiesuperstar wants to merge 3 commits into
NousResearch:mainfrom
shagghiesuperstar:fix/stream-retry-status-messages

Conversation

@shagghiesuperstar

Copy link
Copy Markdown

Problem

When a streaming request to the LLM provider fails mid-stream (e.g. ReadTimeout, ConnectionError, RemoteProtocolError), Hermes retries with a fresh connection. Each retry emits a status message that accumulates permanently in the chat thread:

⚠️ Connection to provider dropped (RemoteProtocolError). Reconnecting… (attempt 2/3)
⚠️ Connection to provider dropped (RemoteProtocolError). Reconnecting… (attempt 3/3)
<the actual successful response>

On self-hosted providers with frequent transient errors (backend restarts, rate limits, OOM recovery), these warnings accumulate and don't reflect the final outcome.

Root Cause

Source: run_agent.py — the retry loop calls _emit_status() for intermediate retry messages. _emit_status() sends directly to the platform adapter, which posts a new message (not an update to an existing message).

Fix

Replace _emit_status() with logger.info() for intermediate retry messages. Only the final outcome (success via "Reconnected — resuming…" or failure via "Connection to provider failed after N attempts") is emitted to the user. Intermediate attempts go to logger.info and stay in logs.

Changes

References

Impact

  • Before: 6+ warning messages accumulate in a session with 3 transient retries
  • After: Only the final "Reconnected — resuming…" or "Connection to provider failed" is visible
  • Users can still see intermediate retries by checking agent.log
  • Zero behavioral change — retries still work identically, only the message delivery changes

Scott Scheferman and others added 3 commits April 30, 2026 14:35
…ry exists

When list_authenticated_providers() processes section 1, the native `openai`
provider (8 models from _PROVIDER_MODELS) now checks if a custom provider
already claims the same base_url (https://api.openai.com/v1). If so, the
built-in entry is skipped, preventing duplicate OpenAI buttons in the
/model picker.

Fixes: user-defined OpenAI (18 models) and built-in openai (8 models) both
pointing at api.openai.com/v1 were appearing as separate buttons.
When a streaming request to the LLM provider fails mid-stream (e.g.
ReadTimeout, ConnectionError, RemoteProtocolError), Hermes retries with
a fresh connection. Each retry emits a status message that accumulates
permanently in the chat thread via _emit_status().

For self-hosted providers with frequent transient errors (backend
restarts, rate limits, OOM recovery), these warnings accumulate and
don't reflect the final outcome.

Fix: Replace _emit_status() with logger.info() for intermediate retry
messages. Only the final outcome (success or failure) is emitted to the
user. Intermediate attempts go to logger.info and stay in logs.

See: NousResearch#5151
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint labels Apr 30, 2026
@teknium1

Copy link
Copy Markdown
Contributor

This looks implemented on current main by the later retry-status buffering work. Automated hermes-sweeper review.

Evidence:

  • PR feat(agent): buffer retry/fallback status, surface only on terminal failure #33816 merged as 67011cc0d76b7047320b2760e948b4e4488c24ca and explicitly buffers retry/fallback status messages, surfacing them only on terminal failure.
  • run_agent.py:828 defines the retry status buffer helpers; the comments state retry/fallback messages are dropped on successful recovery and flushed only when all retry/fallback paths exhaust.
  • agent/stream_diag.py:261 now sends stream-drop reconnect notices through agent._buffer_status(...), not direct _emit_status(...).
  • agent/chat_completion_helpers.py:2288 and agent/chat_completion_helpers.py:2338 route both mid-tool-call and normal stream retry call sites through _emit_stream_drop(...).
  • agent/conversation_loop.py:4079 clears the retry status buffer once successful assistant content is reached, so intermediate retry chatter is not left in the chat thread after recovery.
  • Covered by tests/run_agent/test_retry_status_buffer.py.

Fixed in v2026.5.28 and later.

@teknium1 teknium1 closed this Jun 10, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have sweeper:implemented-on-main Sweeper: behavior already present on current main type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants