Skip to content

fix(slack): surface retryable + Retry-After on send() rate-limit errors (#46762) - #52436

Closed
srojk34 wants to merge 1 commit into
NousResearch:mainfrom
srojk34:fix/slack-send-429-retry
Closed

fix(slack): surface retryable + Retry-After on send() rate-limit errors (#46762)#52436
srojk34 wants to merge 1 commit into
NousResearch:mainfrom
srojk34:fix/slack-send-429-retry

Conversation

@srojk34

@srojk34 srojk34 commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Repro scenario

  1. Bot sends a multi-chunk response to a busy Slack workspace
  2. Slack returns 429 ratelimited on a middle chunk with Retry-After: 30
  3. send() catches the exception → SendResult(success=False, retryable=False) ← no retry
  4. Base _send_with_retry() sees retryable=False, "ratelimited" not in _RETRYABLE_ERROR_PATTERNS → gives up
  5. Remaining chunks silently dropped ❌

Fix

Reuse the existing _is_retryable_upload_error() helper (which already detects 429, 500+, and connection errors for the upload path) to set retryable=True on the send() path. Extract the Retry-After header from the SlackApiError response when present.

Test plan

  • test_429_returns_retryable_with_retry_after — 429 + Retry-After: 30 → retryable=True, retry_after=30.0
  • test_429_without_retry_after_header — 429 without header → retryable=True, retry_after=None
  • test_500_is_retryable_no_retry_after — 500 → retryable=True
  • test_403_is_not_retryable — 403 → retryable=False (unchanged)
  • test_connection_error_is_retryable — ConnectionError → retryable=True
  • All 209 existing Slack tests pass (zero regression)
$ pytest tests/gateway/test_slack_send_retry.py tests/gateway/test_slack.py -q
214 passed in 3.30s

…rs (NousResearch#46762)

Slack's send() caught all exceptions and returned a bare
SendResult(success=False) — never setting retryable=True or extracting
the server's Retry-After header.  When Slack returned a 429 rate-limit
error, the base _send_with_retry() layer saw retryable=False and did
not retry, silently dropping remaining message chunks.

Reuse the existing _is_retryable_upload_error() helper (which already
detects 429, 500+, and connection-type errors) to set retryable=True,
and extract the Retry-After header from the SlackApiError response
when present so the base retry layer honors Slack's backoff schedule
instead of its own default.

Sibling of the Telegram FloodWait fix (PR NousResearch#46762 / commit 404b06a)
which added the SendResult.retry_after plumbing to the base layer.

Adds five regression tests covering 429 with/without Retry-After,
500 server errors, 403 non-retryable errors, and connection errors.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery platform/slack Slack app adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jun 25, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for tracing the missing Slack rate-limit classification; current main still returns a bare failed SendResult from SlackAdapter.send() at plugins/platforms/slack/adapter.py:1435-1437.

Problems

  • The new retryable=True path would cause BasePlatformAdapter._send_with_retry() to call send() again with the complete original content (gateway/platforms/base.py:4148). SlackAdapter.send() posts split responses sequentially (plugins/platforms/slack/adapter.py:1396-1410), so a 429 on a middle chunk would replay already delivered prefix chunks and duplicate them.

Suggested changes

  • Retry only the failed Slack chunk, honoring Retry-After, instead of returning a retryable whole-message failure to the base retry wrapper.
  • Add a multi-chunk regression where chunk one succeeds and chunk two receives 429; assert chunk one is emitted once.

Automated hermes-sweeper review.

except Exception as e: # pragma: no cover - defensive logging
logger.error("[Slack] Send error: %s", e, exc_info=True)
return SendResult(success=False, error=str(e))
_retryable = self._is_retryable_upload_error(e)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marking a failure from this whole-message send() retryable makes _send_with_retry() invoke send() again with the full original content. Since send() posts chunks sequentially, a 429 after earlier chunks succeeded will replay those prefixes and duplicate them. Retry the failed chunk locally (and honor its Retry-After) rather than escalating this result to the whole-message retry layer.

@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 15, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Merged via #69479 — your commit was cherry-picked/reapplied onto current main with your authorship preserved in git history: your retryable + Retry-After surfacing was cherry-picked directly.

Thanks for the contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/slack Slack app adapter sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants