Skip to content

fix(weixin): exponential backoff for iLink rate limiting - #31132

Open
awei321 wants to merge 1 commit into
NousResearch:mainfrom
awei321:fix/weixin-rate-limit-backoff
Open

awei321 wants to merge 1 commit into
NousResearch:mainfrom
awei321:fix/weixin-rate-limit-backoff

Conversation

@awei321

@awei321 awei321 commented May 23, 2026

Copy link
Copy Markdown

Summary

Fixes an issue where the WeChat iLink adapter's rate-limit retry strategy
used a fixed 3-second backoff per attempt, causing messages to be silently
dropped when the rate-limit window exceeded ~12 seconds (total retry time).

Closes #31131

Changes

gateway/platforms/weixin.py

Change Description
_parse_retry_after() (new) Extracts retry_after/retry_after_seconds/retry_after_ms/backoff hints from iLink rate-limit responses
Exponential backoff Rate-limit retries now use base x 2^attempt with +-25% jitter
Chat-level cooldown _rate_limited_at dict tracks per-chat rate-limit timestamps; send() increases inter-chunk delay after rate-limiting is detected
New config keys WEIXIN_RATE_LIMIT_BACKOFF_BASE / rate_limit_backoff_base_seconds, WEIXIN_RATE_LIMIT_CHUNK_COOLDOWN / rate_limit_chunk_cooldown_seconds

Comparison

Before: 3s -> 3s -> 3s -> 3s   (12s total window)
After:  1s -> 2s -> 4s -> 8s -> 16s   (31s window, prioritizing server hints)

Screenshots

Web-UI (reply was generated) WeChat (missing reply)
webui weixin

The iLink adapter used a fixed 3-second backoff on rate-limit errors
(ret=-2 / errcode=-2), which was shorter than typical WeChat rate-limit
windows. Messages that should have succeeded after a longer wait were
instead dropped after exhausting all retries.

Changes:
- Rate-limit retries now use exponential backoff (base × 2^attempt)
  with ±25% jitter to avoid thundering-herd
- New _parse_retry_after() helper extracts server-supplied retry hints
  (retry_after / retry_after_seconds / retry_after_ms / backoff fields)
- Chat-level cooldown: when rate-limiting is detected on a chat,
  inter-chunk delay increases from 1.5s to 5s to prevent cascading
  failures across multi-chunk messages
- Configurable via WEIXIN_RATE_LIMIT_BACKOFF_BASE and
  WEIXIN_RATE_LIMIT_CHUNK_COOLDOWN env vars / config.yaml keys

Before: 3s → 3s → 3s → 3s (12s window, typically too short)
After:  1s → 2s → 4s → 8s → 16s (31s window with server hints)
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery platform/wecom WeCom / WeChat Work adapter P2 Medium — degraded but workaround exists labels May 23, 2026
@alt-glitch

Copy link
Copy Markdown

Competes with several open PRs addressing the same Weixin rate-limit backoff: #21135 (exponential backoff + session reconnect + dynamic chunk delay), #27697 (supersedes #21135 with QQBot fixes), #22540 (iLink exponential backoff), #26904 (circuit breaker approach). Related issue: #21126.

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for investigating Weixin delivery loss. The retry concern remains relevant, but this patch needs reconciliation with current main before it can be safely salvaged.

Problems

  • Current main's circuit breaker opens on the first rate-limit response by default: _rate_limit_circuit_threshold defaults to 1 at gateway/platforms/weixin.py:1179-1185, and _record_rate_limit_event() runs before the retry wait at gateway/platforms/weixin.py:1802-1807. The proposed exponential wait would therefore not run under the default current behavior.
  • The diff removes the native-attachment path filters. Current Weixin send() retains them at gateway/platforms/weixin.py:1853-1856; the helpers validate paths at gateway/platforms/base.py:3497-3520 as part of the safety fix in 41d2c758c.
  • The new WEIXIN_RATE_LIMIT_* behavioral environment variables conflict with the repository config policy, and the new retry/cooldown behavior has no tests.

Suggested changes

  • Reconcile exponential retry with the current circuit-breaker policy and add deterministic tests around that transition.
  • Keep both media-path filters and expose any needed tuning through gateway.platforms.weixin.extra only.

Automated hermes-sweeper review.

@@ -1210,6 +1234,14 @@ def __init__(self, config: PlatformConfig):
extra.get("send_chunk_retry_delay_seconds")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not add these new user-facing behavioral environment variables. Keep any necessary tuning under gateway.platforms.weixin.extra and validate it there; the repository policy reserves .env variables for secrets.

# Extract MEDIA: tags and bare local file paths before text delivery.
media_files, cleaned_content = self.extract_media(content)
media_files = self.filter_media_delivery_paths(media_files)
_, image_cleaned = self.extract_images(cleaned_content)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retain both attachment-path filters removed by this hunk. Current main uses them to validate native-upload paths (gateway/platforms/base.py:3497-3520); removing them would undo the protection from 41d2c758c.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/wecom WeCom / WeChat Work adapter sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Weixin] Messages silently dropped under iLink rate limiting due to insufficient backoff

3 participants