Skip to content

fix(weixin): exponential backoff for iLink rate limit (-2) - #27234

Closed
aha-lin wants to merge 1 commit into
NousResearch:mainfrom
aha-lin:fix/weixin-rate-limit-backoff
Closed

aha-lin wants to merge 1 commit into
NousResearch:mainfrom
aha-lin:fix/weixin-rate-limit-backoff

Conversation

@aha-lin

@aha-lin aha-lin commented May 17, 2026

Copy link
Copy Markdown

Problem

When the iLink sendmessage API returns ret=-2 / errcode=-2 (frequency control / rate limit), the chunk retry loop only waits send_chunk_retry_delay_seconds * 3 per attempt — about 3s by default. With the default send_chunk_retries=4, that's 5 attempts spread across roughly 15 seconds total, which almost always exhausts inside a single iLink throttle window.

This was hit in production by a scheduled cron push: the gateway logged

delivery error: Weixin send failed: iLink sendmessage rate limited:
ret=-2 errcode=None errmsg=rate limited

and the message was silently dropped — the user never received their daily digest. iLink's frequency window is on the order of minutes, not seconds, so a 15s burst-retry has essentially no chance of recovering.

Fix

Switch rate-limit retries to exponential backoff with a 10-minute cap, and bump the default retry count from 4 to 5.

With defaults (send_chunk_retry_delay_seconds=1.0, send_chunk_retries=5), the wait series becomes:

attempt wait before retry
1 10s
2 30s
3 90s
4 270s
5 600s

Total ~17 minutes, last attempt ~10 minutes after the first failure — long enough to outlast iLink's frequency window in practice.

The cap (600.0 seconds) is intentionally hard-coded to prevent runaway waits if send_chunk_retry_delay_seconds is ever set unusually high. The backoff multiplier (3 ** attempt) is also hard-coded for now; if anyone needs to tune that I'm happy to expose it as another config knob, but in our case the empirical 10s/30s/90s/270s/600s ladder works well.

Other error paths (session-expired retries, generic exception retries) are unchanged.

Behaviour matrix

Before After
Default retries 4 (5 attempts) 5 (6 attempts)
Wait per rate-limit retry fixed delay * 3 (~3s) max(delay*10, 10s) * 3^attempt, capped at 600s
Total window ~15s ~17 min
Last attempt offset ~12s ~10 min

Tunables

Already exposed (unchanged keys):

  • send_chunk_retries / WEIXIN_SEND_CHUNK_RETRIES
  • send_chunk_retry_delay_seconds / WEIXIN_SEND_CHUNK_RETRY_DELAY_SECONDS

Operators who don't want the longer wait window can set send_chunk_retries=0 to disable rate-limit retries entirely, or pick a smaller value to stop earlier in the series.

Tests

No new tests — the rate-limit branch isn't currently covered by the suite, and the existing send-chunk tests still pass. Happy to add coverage if reviewers want it; the relevant scaffolding would be a fake _send_message that returns {"ret": -2, "errmsg": "rate limited"} and asserts the wait sequence via freezegun / asyncio.sleep patching.

Manual verification

Patched the local gateway, restarted, watched ~/.hermes/logs/gateway.log during a deliberate burst send. Saw the new log line on each retry:

[weixin] rate limited for u****; backing off 10.0s before retry (attempt 1/6)
[weixin] rate limited for u****; backing off 30.0s before retry (attempt 2/6)
...

and delivery succeeded on the third attempt once iLink's window cleared.

When iLink returns ret=-2 / errcode=-2 (frequency control), the chunk
retry loop now waits with exponential backoff capped at 10 minutes,
instead of a fixed 3× delay (~3s) that almost always exhausted retries
inside a single iLink throttle window.

Defaults: 5 retries (6 attempts), waits 10s, 30s, 90s, 270s, 600s —
total ~17 minutes, last attempt ~10 minutes after the first failure.
Triggered in production when scheduled cron pushes hit the gateway's
own rate limit and silently failed delivery (last_delivery_error =
'iLink sendmessage rate limited: ret=-2').

Tunable via send_chunk_retries / send_chunk_retry_delay_seconds in
platform config or WEIXIN_SEND_CHUNK_RETRIES /
WEIXIN_SEND_CHUNK_RETRY_DELAY_SECONDS env vars.
@aha-lin

aha-lin commented May 17, 2026

Copy link
Copy Markdown
Author

Closing for now — refocusing on a related voice-handling PR first; will reopen this rate-limit fix after that lands so reviewers see one change at a time. Sorry for the churn.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants