Skip to content

fix(gateway): do not resend a message the platform rate-limited - #79043

Open
s905060 wants to merge 1 commit into
NousResearch:mainfrom
s905060:fix/telegram-flood-duplicate
Open

fix(gateway): do not resend a message the platform rate-limited#79043
s905060 wants to merge 1 commit into
NousResearch:mainfrom
s905060:fix/telegram-flood-duplicate

Conversation

@s905060

@s905060 s905060 commented Aug 5, 2026

Copy link
Copy Markdown

A 429 with a server-requested retry_after is not proof of non-delivery. Telegram commonly ACCEPTS the message and rate-limits the acknowledgement, so _send_with_retry re-sending the whole payload puts a second full copy in the chat.

Observed in a live gateway log, one user turn:

sendRichMessage transient failure (no legacy resend): Flood control exceeded. Retry in 13 seconds
Send failed (attempt 1/2, retrying in 13.7s)
Send succeeded on retry 1

One Sending response (1700 chars), two identical messages delivered to the chat.

Cause

The Telegram adapter already refuses its own internal legacy-resend on this exact error, for this exact reason:

# Transient / network / unknown: the request may have reached
# Telegram. Do NOT legacy-resend (duplicate risk)

It then returns retryable=True with the parsed retry_after, which hands the same duplicate risk to the shared retry layer in gateway/platforms/base.py — which re-sends the whole message. The adapter closed the door and the base layer opened it again.

Change

Treat retry_after as delivery-ambiguous, the way a timeout already is. _send_with_retry returns the failure instead of resending, in both the pre-loop check and the in-loop check (a 429 can arrive on a later attempt after a genuine ConnectError).

That makes the first-send server_retry_after pacing branch unreachable, so it is removed rather than left as dead code.

Fixed in the shared base adapter rather than in Telegram: every platform that reports retry_after routes through this one path.

Ordinary transient errors (ConnectError and friends) retry exactly as before — the guard is scoped to a server-requested backoff.

Tests

TestSendWithRetryAfter previously asserted that retry_after merely paced a retry, which is the bug encoded as a contract, so those two cases are rewritten to assert send-count instead:

  • test_retry_after_is_not_resent — 429 sends exactly once
  • test_retry_after_on_a_later_attempt_stops_resendingConnectError still retries, the 429 on that retry stops it (no third copy)
  • test_plain_network_error_still_retries — new, pins that the guard did not disable ordinary retries

Both rewritten tests were confirmed to fail on unpatched code before the fix was trusted, reproducing the duplicate (assert not result.success / AssertionError: assert not True).

tests/gateway/test_send_retry.py                 13 passed
+ 21 sibling _send_with_retry tests (slack, ephemeral, photon overflow)  24 passed

The suite also drops 32.13s → 1.16s, because the old tests were really sleeping through the flood waits they asserted on.

Not run: the full tests/gateway suite is red at baseline in this environment (213 failed / 1389 errors on an unmodified checkout), so its totals carry no signal. Three test_telegram_thread_fallback cases fail when the _send_with_retry files are run as one batch — they fail identically on unpatched code and pass in isolation, so that is a pre-existing cross-test state leak, not this change.

A 429 with a server-requested retry_after is not proof of non-delivery.
Telegram commonly ACCEPTS the message and rate-limits the acknowledgement,
so _send_with_retry re-sending the whole payload put a second full copy in
the chat.

Observed in a live gateway log, one user turn:

  sendRichMessage transient failure (no legacy resend): Flood control
    exceeded. Retry in 13 seconds
  Send failed (attempt 1/2, retrying in 13.7s)
  Send succeeded on retry 1

One 1700-char response, two identical messages delivered.

The Telegram adapter already refuses its own internal legacy-resend on this
exact error for this exact reason, then returned retryable=True and handed
the same duplicate risk to the shared retry layer. Treat retry_after as
delivery-ambiguous the way a timeout already is, in both the pre-loop and
in-loop checks, so the one copy the platform holds stays the only one.

This makes the first-send server_retry_after pacing branch unreachable, so
it is removed rather than left as dead code.

Fixed in the shared base adapter, not in Telegram: every platform reporting
retry_after routes through here.
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Aug 5, 2026
@spfcraze

spfcraze commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

This was generated by AI during triage.

A defect in the change:

The guard keyed on retry_after in the shared _send_with_retry also suppresses Slack's retry of HTTP 429 rate limits, where the message was not delivered, undoing the behavior PR #52436 added.

Problems:

  • The new if result.retry_after is not None: return result runs before the retry loop and before the plain-text fallback, so any adapter whose send() sets retry_after — not just Telegram — stops retrying entirely.
  • Slack's send() sets retryable=True and retry_after from the Retry-After header on HTTP 429 (plugins/platforms/slack/adapter.py:2611-2620), and _is_retryable_upload_error classifies 429 as retryable (adapter.py:3373). A Slack 429 is a rate-limit rejection before the request is processed — the message is not posted — so waiting the server-requested delay and retrying is the correct behavior, which is what the server_retry_after pacing in base.py (5080-5086) does on main today.
  • That pacing is exactly what PR fix(slack): surface retryable + Retry-After on send() rate-limit errors (#46762) #52436 ("fix(slack): surface retryable + Retry-After on send() rate-limit errors") wired Slack's retry_after into: it fixed Slack 429s being silently dropped. This guard returns Slack to the no-retry case fix(slack): surface retryable + Retry-After on send() rate-limit errors (#46762) #52436 fixed.

Suggested changes:


Checked against 1d51f78 — the tip of fix/telegram-flood-duplicate when this was written — and aec3318, main at the same moment. Verified mechanically: the diff grep at the former; the code on main at the latter; the state of the pull requests referenced as it stood then. The reading of intended scope is inference. If I have misread the intent here, please say so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants