Skip to content

fix(wechat): exponential rate-limit backoff + ercode 846609 reconnect retry - #31393

Open
luodiheng wants to merge 2 commits into
NousResearch:mainfrom
luodiheng:fix/wechat-errcode-846609-and-rate-limit-backoff
Open

luodiheng wants to merge 2 commits into
NousResearch:mainfrom
luodiheng:fix/wechat-errcode-846609-and-rate-limit-backoff

Conversation

@luodiheng

Copy link
Copy Markdown

Summary

修复两个微信/企业微信相关的严重 Bug:

Fix 1: #31131 — WeChat iLink 限流退避不足导致消息静默丢失

问题: 当微信 iLink API 返回 rate limit (ret=-2, errcode=-2) 时,adapter 使用固定 3 秒退避 × 4 次重试(总计 ~12 秒),但微信限流窗口通常是 15-30 秒。重试过早耗尽 → 消息静默丢弃。

修复: 将 _send_text_chunk 中的固定 * 3 改为指数退避 * (2 ** attempt)

  • 默认参数: 1s → 2s → 4s → 8s → 16s (总计 ~31s,覆盖典型限流窗口)
  • 可配置: 通过 WEIXIN_SEND_CHUNK_RETRY_DELAY_SECONDS 调整基础间隔

Fix 2: #29667 — WeCom ercode 846609 WebSocket 断连导致回复静默失败

问题: 企业微信 WebSocket 中途断开后,后续 send 操作返回 ercode 846609 ("aibot websocket not subscribed")。响应已生成但永远无法送达 — 单日出现 173+ 次

修复: 新增 _send_with_reconnect_retry 包装器:

  • 发送前检查 WebSocket 存活状态
  • 检测 ercode 846609 后自动重连 (_open_connection) + 重试一次
  • 处理 "not connected" RuntimeError 同理
  • 覆盖所有出站路径: send(), _send_media_message(), _send_reply_markdown(), _send_reply_media_message()

Files Changed

  • gateway/platforms/weixin.py — 指数退避 (1 行改动)
  • gateway/platforms/wecom.py — 新增 ERRCODE_NOT_SUBSCRIBED 常量和 _send_with_reconnect_retry 方法,修改 send 和相关发送方法

Testing

  • Python 语法检查通过
  • 向后兼容,不影响现有配置
  • 指数退避参数可通过 env var 调整

luodiheng added 2 commits May 24, 2026 16:56
When the iLink API rate-limits a send (ret=-2, errcode=-2), the old code
used a fixed 3-second backoff per retry (send_chunk_retry_delay_seconds * 3).
With 4 default retries, the total wait window was only ~12 seconds, but
WeChat rate-limit windows are typically 15-30 seconds or longer. Messages
were silently dropped when all retries were exhausted before the limit
cleared.

This change replaces the fixed multiplier with exponential backoff
(2 ** attempt), giving a progressive wait of 1s, 2s, 4s, 8s, 16s
with default settings, which comfortably covers the typical rate-limit
windows.

NOTE: Due to 100MB GitHub file push limit, this is the complete weixin.py
file. The only change is the rate-limit backoff calculation on the
`wait = ...` line inside `_send_text_chunk`.

Refs: NousResearch#31131
WeCom's WebSocket connections occasionally drop mid-session, causing
send attempts to fail with ercode 846609 ("aibot websocket not subscribed").
When this happens, responses are generated but silently dropped — the user
sees no reply and API tokens are wasted.

This adds a `_send_with_reconnect_retry` wrapper that:
- Checks if the WebSocket is alive before sending
- Detects ercode 846609 in the response
- Automatically reconnects (calls _open_connection) and retries once
- Also handles "not connected" RuntimeErrors similarly

Applied to all outbound send paths: send(), _send_media_message(),
_send_reply_markdown(), and _send_reply_media_message().

Refs: NousResearch#29667
@ygd58

ygd58 commented May 24, 2026

Copy link
Copy Markdown
Contributor

LGTM — the _send_with_reconnect_retry wrapper approach is cleaner than patching individual send paths. Closing my earlier #29731 in favor of this PR.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery platform/wecom WeCom / WeChat Work adapter labels May 24, 2026
@alt-glitch

Copy link
Copy Markdown

Competing/related PRs for the same fixes: #31132 (weixin exponential backoff), #27697 (weixin reconnect + QQBot close codes), #21135 (weixin backoff + reconnect + chunk delay), #29731 (wecom 846609 retry, now closed in favor of this PR). This PR bundles both weixin backoff and wecom 846609 reconnect.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for consolidating the related Weixin and WeCom reports. The WeCom lost-subscription premise still holds on current main: plugins/platforms/wecom/adapter.py:1398-1423 returns a failed SendResult for nonzero responses, and _response_error() has no 846609 recovery branch (:1048-1060).

Problems

  • The PR modifies gateway/platforms/wecom.py, which was migrated to plugins/platforms/wecom/adapter.py by f2a7adba5; this needs a port rather than a clean cherry-pick.
  • gateway/platforms/wecom.py:1350 calls _open_connection() directly while the listener remains active. The current equivalent closes the shared socket before reconnecting (plugins/platforms/wecom/adapter.py:272-284), and _listen_loop independently reconnects after its read fails (:335-360), so the retry can race a second replacement of self._ws.
  • No regression tests cover 846609 recovery. Current send-error coverage is only the ordinary error case in tests/gateway/test_wecom.py:522-552.
  • The current Weixin circuit breaker opens at the default first rate-limit event and breaks before the fixed-delay branch (gateway/platforms/weixin.py:1179-1193, :1802-1807), so the exponential-backoff portion requires a current-design decision.

Suggested changes

  • Port the WeCom fix to the plugin adapter, coordinate recovery with the listener, and add response/exception retry tests for reply and proactive sends.
  • Reconcile the Weixin retry change with the current circuit breaker before salvaging it.

Automated hermes-sweeper review.

"[%s] ercode 846609 (not subscribed), reconnecting and retrying...",
self.name,
)
await self._open_connection()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_open_connection() replaces the shared websocket while the existing listener is still reading it. Its close causes _listen_loop to enter its own reconnect path, which can replace the retry connection before the retry completes. Please coordinate recovery through a single lifecycle path (or lock/cancel the listener) before retrying.

@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/wecom WeCom / WeChat Work adapter sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants