Conversation
When iLink returns ret=-2 / errcode=-2 (frequency control), the chunk retry loop now waits with exponential backoff capped at 10 minutes, instead of a fixed 3× delay (~3s) that almost always exhausted retries inside a single iLink throttle window. Defaults: 5 retries (6 attempts), waits 10s, 30s, 90s, 270s, 600s — total ~17 minutes, last attempt ~10 minutes after the first failure. Triggered in production when scheduled cron pushes hit the gateway's own rate limit and silently failed delivery (last_delivery_error = 'iLink sendmessage rate limited: ret=-2'). Tunable via send_chunk_retries / send_chunk_retry_delay_seconds in platform config or WEIXIN_SEND_CHUNK_RETRIES / WEIXIN_SEND_CHUNK_RETRY_DELAY_SECONDS env vars.
Author
|
Closing for now — refocusing on a related voice-handling PR first; will reopen this rate-limit fix after that lands so reviewers see one change at a time. Sorry for the churn. |
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the iLink
sendmessageAPI returnsret=-2/errcode=-2(frequency control / rate limit), the chunk retry loop only waitssend_chunk_retry_delay_seconds * 3per attempt — about 3s by default. With the defaultsend_chunk_retries=4, that's 5 attempts spread across roughly 15 seconds total, which almost always exhausts inside a single iLink throttle window.This was hit in production by a scheduled cron push: the gateway logged
and the message was silently dropped — the user never received their daily digest. iLink's frequency window is on the order of minutes, not seconds, so a 15s burst-retry has essentially no chance of recovering.
Fix
Switch rate-limit retries to exponential backoff with a 10-minute cap, and bump the default retry count from 4 to 5.
With defaults (
send_chunk_retry_delay_seconds=1.0,send_chunk_retries=5), the wait series becomes:Total ~17 minutes, last attempt ~10 minutes after the first failure — long enough to outlast iLink's frequency window in practice.
The cap (
600.0seconds) is intentionally hard-coded to prevent runaway waits ifsend_chunk_retry_delay_secondsis ever set unusually high. The backoff multiplier (3 ** attempt) is also hard-coded for now; if anyone needs to tune that I'm happy to expose it as another config knob, but in our case the empirical 10s/30s/90s/270s/600s ladder works well.Other error paths (session-expired retries, generic exception retries) are unchanged.
Behaviour matrix
delay * 3(~3s)max(delay*10, 10s) * 3^attempt, capped at 600sTunables
Already exposed (unchanged keys):
send_chunk_retries/WEIXIN_SEND_CHUNK_RETRIESsend_chunk_retry_delay_seconds/WEIXIN_SEND_CHUNK_RETRY_DELAY_SECONDSOperators who don't want the longer wait window can set
send_chunk_retries=0to disable rate-limit retries entirely, or pick a smaller value to stop earlier in the series.Tests
No new tests — the rate-limit branch isn't currently covered by the suite, and the existing send-chunk tests still pass. Happy to add coverage if reviewers want it; the relevant scaffolding would be a fake
_send_messagethat returns{"ret": -2, "errmsg": "rate limited"}and asserts the wait sequence viafreezegun/asyncio.sleeppatching.Manual verification
Patched the local gateway, restarted, watched
~/.hermes/logs/gateway.logduring a deliberate burst send. Saw the new log line on each retry:and delivery succeeded on the third attempt once iLink's window cleared.