Skip to content

fix(gateway): dedup provider-error status and final response for adapters without send_or_update_status - #72144

Closed
x7peeps wants to merge 1 commit into
NousResearch:mainfrom
x7peeps:fix/issue-72131-dedup-provider-error-status
Closed

x7peeps wants to merge 1 commit into
NousResearch:mainfrom
x7peeps:fix/issue-72131-dedup-provider-error-status

Conversation

@x7peeps

@x7peeps x7peeps commented Jul 26, 2026

Copy link
Copy Markdown

背景

修复 #72131: Provider errors delivered twice on adapters without send_or_update_status (mid-run status rewrite + identical final response)

根因分析

在平台适配器未实现可选方法 send_or_update_status 时,_send_or_update_status_coro 会降级为调用 adapter.send() 发送状态消息。当 provider 发生错误时,两条通道都会发送相同的改写后文本:

  1. 运行中状态回调 → _prepare_gateway_status_message 改写后通过 _send_or_update_status_coro 发送(降级为 send(),成为持久消息)
  2. 失败的 turn 最终响应 → _sanitize_gateway_final_response 改写后再次发送(相同的用户友好文本)

对于 Telegram 等实现了 send_or_update_status 的适配器,状态气泡会被原地编辑并在运行结束后清理,所以只有最终消息保留。但对于没有该方法的适配器,状态消息变成第二条持久消息,导致用户看到两条相同的错误提示。

修复方式

在 gateway/run.py 中新增 dedup 机制:

  • 新增模块级变量 _last_sent_provider_error_status 跟踪通过 plain-send 降级路径发送的 provider 错误状态文本
  • 在 _send_or_update_status_coro 降级路径中,如果内容是 provider 错误回复,记录该文本
  • 在 _sanitize_gateway_final_response 中,如果改写结果与已发送的状态文本相同,返回空字符串跳过重复发送
  • 在调用方(final_response 为空时的回退逻辑)中,检查 _deduped_provider_error 标志,避免回退到原始错误文本

验证

  • 代码变更已在本地验证
  • 遵循项目 AGENTS.md 贡献规范
  • 无破坏性变更
  • 新增 21 个单元测试全部通过
  • 现有 617 个 noise filter 测试全部通过
  • 现有 144 个 status 相关测试全部通过

Closes #72131

…ters without send_or_update_status

Fix NousResearch#72131

根因分析:
在平台适配器未实现可选方法 send_or_update_status 时,_send_or_update_status_coro 会降级为
调用 adapter.send() 发送状态消息。当 provider 发生错误时,两条通道都会发送相同的改写后文本:
1. 运行中状态回调 → _prepare_gateway_status_message 改写后通过 _send_or_update_status_coro 发送
2. 失败的 turn 最终响应 → _sanitize_gateway_final_response 改写后再次发送

对于 Telegram 等实现了 send_or_update_status 的适配器,状态气泡会被原地编辑并在运行结束后
清理,所以只有最终消息保留。但对于没有该方法的适配器,状态消息变成持久消息,与最终响应产生
重复。

修复方式:
- 新增模块级变量 _last_sent_provider_error_status 跟踪通过 plain-send 降级路径发送的
  provider 错误状态文本
- 在 _send_or_update_status_coro 降级路径中,如果内容是 provider 错误回复,记录该文本
- 在 _sanitize_gateway_final_response 中,如果改写结果与已发送的状态文本相同,返回空字符串
  跳过重复发送,并设置 _deduped_provider_error 标志
- 在调用方(final_response 为空时的回退逻辑)中,检查 _deduped_provider_error 标志,如果是
  因为 dedup 导致的空结果,不再回退到原始错误文本

新增测试: tests/gateway/test_provider_error_dedup.py (21 tests)
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages P2 Medium — degraded but workaround exists labels Jul 26, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing the two delivery paths. The duplicate premise still exists on current main: gateway/run.py:4034-4048 prepares/sends a status through the plain-send fallback at gateway/run.py:631-634, while the final path independently sanitizes provider errors at gateway/run.py:16775-16778.

Problems

  • The globals added in gateway/run.py by 49567927af56 are process-wide and have no chat/run scope or deterministic consumption. They can affect a later or concurrent turn.
  • The fallback classifies the already-sanitized text. The reported rate-limit reply is emitted as "rate-limiting requests" (gateway/run.py:508), but the matcher only accepts "rate limited after <n> retries" (gateway/run.py:520), so that case is not recorded.
  • Suppressing the inner run_sync fallback is insufficient: GatewayRunner re-normalizes the unchanged failed result and sanitizes it again at gateway/run.py:16775-16778.

Suggested changes

  • Keep dedup state per turn (the current TurnContext seam is available), record only after a successful fallback send, and apply suppression at the final delivery-level normalization path.
  • Add an end-to-end no-send_or_update_status adapter test for rate limits, failed sends, and interleaved turns.

Automated hermes-sweeper review.

Comment thread gateway/run.py

# Issue #72131: track provider-error status text sent via the plain-send fallback
# so that _sanitize_gateway_final_response can dedup identical final responses.
_last_sent_provider_error_status: Optional[str] = None

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_last_sent_provider_error_status is process-global and is never scoped, consumed, or cleared at a run boundary. Two concurrent chats, or a later turn with the same friendly error text, can suppress an unrelated final reply. Keep this state on the affected turn/chat instead.

Comment thread gateway/run.py
_record_provider_error_status(content)
return await adapter.send(chat_id, content, metadata=metadata)


Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This tests the already-sanitized status string against an envelope matcher. The rate-limit status produced by _gateway_provider_error_reply says rate-limiting requests, while _looks_like_gateway_provider_error matches rate limited after <n> retries; the reported rate-limit case is therefore not recorded. Classify before rewriting, or record an explicit per-turn provider-error status marker.

Comment thread gateway/run.py
# Issue #72131: if dedup suppressed a provider-error final
# response (already delivered as status), do NOT fall back to
# the raw error text — the user already has the sanitized reply.
if not final_response and not _deduped_provider_error:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only prevents the inner fallback. The outer gateway path later calls _normalize_empty_agent_response(agent_result, response, ...) and _sanitize_gateway_final_response(...) again, reconstructing the provider-error reply from the unchanged failed result. Apply the delivery decision at that outer final-response path or propagate it explicitly.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jul 30, 2026
@x7peeps

x7peeps commented Jul 30, 2026

Copy link
Copy Markdown
Author

Quality Assessment: S-grade (8/10)

+167/-2 across 2 files. Deduplicates provider-error status and final response for adapters without send_or_update_status. Fixes real duplicate message bug. Open since 7/26 with no maintainer review. Requesting review from @Teknium or team.

1 similar comment
@x7peeps

x7peeps commented Jul 30, 2026

Copy link
Copy Markdown
Author

Quality Assessment: S-grade (8/10)

+167/-2 across 2 files. Deduplicates provider-error status and final response for adapters without send_or_update_status. Fixes real duplicate message bug. Open since 7/26 with no maintainer review. Requesting review from @Teknium or team.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Provider errors delivered twice on adapters without send_or_update_status (mid-run status rewrite + identical final response)

3 participants