Skip to content

fix(discord): scale slash sync timeout to actual write count (#16713) - #16739

Closed
Tranquil-Flow wants to merge 1 commit into
NousResearch:mainfrom
Tranquil-Flow:fix/16713-discord-slash-sync-timeout
Closed

Tranquil-Flow wants to merge 1 commit into
NousResearch:mainfrom
Tranquil-Flow:fix/16713-discord-slash-sync-timeout

Conversation

@Tranquil-Flow

Copy link
Copy Markdown
Contributor

What does this PR do?

_run_post_connect_initialization in gateway/platforms/discord.py wraps the entire _safe_sync_slash_commands call in asyncio.wait_for(..., 30). Discord's per-app command-management bucket allows roughly 5 writes / 20-second window, so a mass-prune-plus-upsert reconcile reliably blows the 30 s budget under back-pressure. The reported case had 77 orphans + 30 desired = 107 writes; two consecutive 30 s timeouts, then a clean 22 s sync only after the bucket fully recovered ~60 minutes later.

This PR splits the read-only diff from the writes and sizes the execute-phase timeout to the actual workload, so a heavy reconcile gets enough budget to finish under bucket pressure but pathological loads still get a hard cap.

Related Issue

Fixes #16713

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • gateway/platforms/discord.py — extracted two helpers from _safe_sync_slash_commands:
    • _plan_slash_command_sync() — read-only; diffs desired vs existing global commands, returns a list of typed actions (create / update / recreate) plus the orphan deletes and a write_count.
    • _execute_slash_command_sync(plan) — applies the plan, returns the existing summary dict.
    • _safe_sync_slash_commands is now a thin wrapper that calls plan + execute, preserving the single-shot signature for callers and existing tests.
  • gateway/platforms/discord.py:_run_post_connect_initialization — now plans first under a 30 s budget, then computes a write-count-aware execute budget (30 + 5 × write_count, capped at 600 s) before running the plan. recreate actions count as two writes (delete + upsert) so the budget covers Discord's actual rate-limit math. Timeout log message updated to point at saturated rate-limit bucket as the cause instead of a flat "after 30 s".
  • Three new module-level constants on the adapter (_SLASH_SYNC_BASE_BUDGET_SECONDS, _SLASH_SYNC_PER_WRITE_SECONDS, _SLASH_SYNC_MAX_BUDGET_SECONDS) and a static _estimate_slash_sync_budget(write_count) so the budget formula is testable and trivially tunable.
  • tests/gateway/test_discord_connect.py — added five tests:
    • _estimate_slash_sync_budget scales with write count, is monotonic, caps at the maximum.
    • _plan_slash_command_sync counts a recreate as two writes plus an orphan delete (matches Discord's bucket math).
    • Plan + execute produces the same summary dict as the existing _safe_sync_slash_commands single-shot path (so callers/tests that read the summary stay correct).

How to Test

Reproduction (matches the issue body): trigger a session that produces ≥ ~30 orphan slash commands (e.g., a command-registry refactor that drops or renames commands), restart the gateway, and watch the post-connect initialization. Before this fix, wait_for(..., 30) raised asyncio.TimeoutError and subsequent reconnects within the rate-limit cooldown also timed out. After this fix, the budget scales with the planned write count and the reconcile completes under bucket pressure.

Automated:

pytest tests/gateway/test_discord_connect.py -q

Result on macOS 15.6.1 / Python 3.14.2: 16 passed (11 pre-existing + 5 new). All five new tests fail on main without the production change.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run pytest tests/ -q and all tests pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS 15.6.1 (Python 3.14.2)

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — or N/A (new helpers carry inline docstrings explaining the bucket math)
  • I've updated cli-config.yaml.example if I added/changed config keys — or N/A (N/A — budget tuning lives on adapter class constants; not user-facing yaml)
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — or N/A (N/A)
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide — or N/A (N/A — pure asyncio + discord.py HTTP calls, no platform-specific syscalls)
  • I've updated tool descriptions/schemas if I changed tool behavior — or N/A (N/A — gateway adapter, not a tool)

Screenshots / Logs

$ pytest tests/gateway/test_discord_connect.py -q
................                                                         [100%]
16 passed, 3 warnings in 3.17s

…earch#16713)

``_run_post_connect_initialization`` wrapped the entire
``_safe_sync_slash_commands`` call in ``asyncio.wait_for(..., 30)``.
Discord's per-app command-management bucket allows ~5 writes / 20-second
window, so a mass-prune-plus-upsert reconcile (the reported case had 77
orphans + 30 desired = 107 writes) reliably blew the 30 s budget under
back-pressure. The post-connect handler then returned a TimeoutError
and the gateway retried on the next reconnect, only succeeding ~60 min
later once the bucket fully recovered.

Split ``_safe_sync_slash_commands`` into ``_plan_slash_command_sync``
(read-only diff) + ``_execute_slash_command_sync`` (writes). The
post-connect path now plans first under a 30 s budget, computes a
write-count-aware execute budget (30 s base + 5 s per write, capped at
600 s), then runs the plan under that budget. ``_safe_sync_slash_commands``
keeps its existing single-shot signature for callers/tests, layered on
top of the plan + execute helpers.

Recreate operations count as two writes (delete + upsert) so the budget
covers Discord's actual rate-limit math. Logs the timeout cause as a
saturated rate-limit bucket instead of a flat "after 30 s" message.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery platform/discord Discord bot adapter labels Apr 27, 2026
teknium1 added a commit that referenced this pull request Apr 28, 2026
…ure (#16713) (#17029)

Discord's per-app command-management bucket is ~5 writes / 20 s. A
mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes
in the reported case) can't finish under the old flat 30 s budget, and
the subsequent reconnect retries inside the rate-limit cooldown also
time out — leaving slash commands broken for ~60 min until the bucket
fully recovers.

Bump the timeout to 600 s so realistic bursts drain, update the warning
message to point at the saturated bucket instead of a hardcoded 30 s.
The 600 s cap still guards against a true hang.

Credit to @Tranquil-Flow for PR #16739 and @davidbordenwi for reporting
#16713 with the bucket-math diagnosis.

Closes #16713.

Co-authored-by: Teknium <teknium@nousresearch.com>
@teknium1

Copy link
Copy Markdown
Collaborator

Thanks for the clean write-up and the bucket-math diagnosis — both were load-bearing for the fix.

Merged a simpler variant via #17029 / commit on main: same root cause (flat 30 s budget vs Discord's ~5-writes/20-s command-management bucket), but just widens the outer asyncio.wait_for to 600 s instead of introducing the plan/execute split and write-count-derived budget. The 600 s ceiling does the load-bearing work here; the adaptive formula in your PR extrapolates from the same bucket math, so in practice both fixes land in the same regime, and the flat ceiling keeps _safe_sync_slash_commands as a single unit.

Credit to @Tranquil-Flow and @davidbordenwi in the merge commit. Closing in favor of #17029.

@teknium1 teknium1 closed this Apr 28, 2026
cluricaun28 referenced this pull request in cluricaun28/Logos Apr 28, 2026
…ure (#16713) (#17029)

Discord's per-app command-management bucket is ~5 writes / 20 s. A
mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes
in the reported case) can't finish under the old flat 30 s budget, and
the subsequent reconnect retries inside the rate-limit cooldown also
time out — leaving slash commands broken for ~60 min until the bucket
fully recovers.

Bump the timeout to 600 s so realistic bursts drain, update the warning
message to point at the saturated bucket instead of a hardcoded 30 s.
The 600 s cap still guards against a true hang.

Credit to @Tranquil-Flow for PR #16739 and @davidbordenwi for reporting
#16713 with the bucket-math diagnosis.

Closes #16713.

Co-authored-by: Teknium <teknium@nousresearch.com>
donald131 pushed a commit to donald131/hermes-agent that referenced this pull request May 2, 2026
…ure (NousResearch#16713) (NousResearch#17029)

Discord's per-app command-management bucket is ~5 writes / 20 s. A
mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes
in the reported case) can't finish under the old flat 30 s budget, and
the subsequent reconnect retries inside the rate-limit cooldown also
time out — leaving slash commands broken for ~60 min until the bucket
fully recovers.

Bump the timeout to 600 s so realistic bursts drain, update the warning
message to point at the saturated bucket instead of a hardcoded 30 s.
The 600 s cap still guards against a true hang.

Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting
NousResearch#16713 with the bucket-math diagnosis.

Closes NousResearch#16713.

Co-authored-by: Teknium <teknium@nousresearch.com>
@bgmbgm94

bgmbgm94 commented May 4, 2026

Copy link
Copy Markdown

Sharing a related data point from a Windows Discord gateway deployment.

We saw intermittent Discord send 429 pressure around long/chunked replies. A local mitigation that helped was pacing outbound Discord channel.send(...) calls with a small await asyncio.sleep(0.5) before chunk sends and before the plain-text fallback resend path.

This did not look related to a separate silent Process exited with code 0 investigation, but it did reduce burst pressure during Discord replies. If this area is revisited, a configurable Discord send throttle / adapter pacing helper might be worth considering alongside the write-count-aware slash-sync budgeting discussed here.

02356abc pushed a commit to 02356abc/hermes-agent that referenced this pull request May 14, 2026
…ure (NousResearch#16713) (NousResearch#17029)

Discord's per-app command-management bucket is ~5 writes / 20 s. A
mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes
in the reported case) can't finish under the old flat 30 s budget, and
the subsequent reconnect retries inside the rate-limit cooldown also
time out — leaving slash commands broken for ~60 min until the bucket
fully recovers.

Bump the timeout to 600 s so realistic bursts drain, update the warning
message to point at the saturated bucket instead of a hardcoded 30 s.
The 600 s cap still guards against a true hang.

Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting
NousResearch#16713 with the bucket-math diagnosis.

Closes NousResearch#16713.

Co-authored-by: Teknium <teknium@nousresearch.com>
dannyJ848 pushed a commit to dannyJ848/hermes-agent that referenced this pull request May 17, 2026
…ure (NousResearch#16713) (NousResearch#17029)

Discord's per-app command-management bucket is ~5 writes / 20 s. A
mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes
in the reported case) can't finish under the old flat 30 s budget, and
the subsequent reconnect retries inside the rate-limit cooldown also
time out — leaving slash commands broken for ~60 min until the bucket
fully recovers.

Bump the timeout to 600 s so realistic bursts drain, update the warning
message to point at the saturated bucket instead of a hardcoded 30 s.
The 600 s cap still guards against a true hang.

Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting
NousResearch#16713 with the bucket-math diagnosis.

Closes NousResearch#16713.

Co-authored-by: Teknium <teknium@nousresearch.com>
gweeteve pushed a commit to gweeteve/hermes-agent that referenced this pull request Jun 2, 2026
…ure (NousResearch#16713) (NousResearch#17029)

Discord's per-app command-management bucket is ~5 writes / 20 s. A
mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes
in the reported case) can't finish under the old flat 30 s budget, and
the subsequent reconnect retries inside the rate-limit cooldown also
time out — leaving slash commands broken for ~60 min until the bucket
fully recovers.

Bump the timeout to 600 s so realistic bursts drain, update the warning
message to point at the saturated bucket instead of a hardcoded 30 s.
The 600 s cap still guards against a true hang.

Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting
NousResearch#16713 with the bucket-math diagnosis.

Closes NousResearch#16713.

Co-authored-by: Teknium <teknium@nousresearch.com>
Seven74AI pushed a commit to Seven74AI/hermes-agent that referenced this pull request Jun 13, 2026
…ure (NousResearch#16713) (NousResearch#17029)

Discord's per-app command-management bucket is ~5 writes / 20 s. A
mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes
in the reported case) can't finish under the old flat 30 s budget, and
the subsequent reconnect retries inside the rate-limit cooldown also
time out — leaving slash commands broken for ~60 min until the bucket
fully recovers.

Bump the timeout to 600 s so realistic bursts drain, update the warning
message to point at the saturated bucket instead of a hardcoded 30 s.
The 600 s cap still guards against a true hang.

Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting
NousResearch#16713 with the bucket-math diagnosis.

Closes NousResearch#16713.

Co-authored-by: Teknium <teknium@nousresearch.com>
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
…ure (NousResearch#16713) (NousResearch#17029)

Discord's per-app command-management bucket is ~5 writes / 20 s. A
mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes
in the reported case) can't finish under the old flat 30 s budget, and
the subsequent reconnect retries inside the rate-limit cooldown also
time out — leaving slash commands broken for ~60 min until the bucket
fully recovers.

Bump the timeout to 600 s so realistic bursts drain, update the warning
message to point at the saturated bucket instead of a hardcoded 30 s.
The 600 s cap still guards against a true hang.

Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting
NousResearch#16713 with the bucket-math diagnosis.

Closes NousResearch#16713.

Co-authored-by: Teknium <teknium@nousresearch.com>
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…ure (NousResearch#16713) (NousResearch#17029)

Discord's per-app command-management bucket is ~5 writes / 20 s. A
mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes
in the reported case) can't finish under the old flat 30 s budget, and
the subsequent reconnect retries inside the rate-limit cooldown also
time out — leaving slash commands broken for ~60 min until the bucket
fully recovers.

Bump the timeout to 600 s so realistic bursts drain, update the warning
message to point at the saturated bucket instead of a hardcoded 30 s.
The 600 s cap still guards against a true hang.

Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting
NousResearch#16713 with the bucket-math diagnosis.

Closes NousResearch#16713.

Co-authored-by: Teknium <teknium@nousresearch.com>
prmartinow pushed a commit to prmartinow/hermes-agent that referenced this pull request Aug 26, 2026
…ure (NousResearch#16713) (NousResearch#17029)

Discord's per-app command-management bucket is ~5 writes / 20 s. A
mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes
in the reported case) can't finish under the old flat 30 s budget, and
the subsequent reconnect retries inside the rate-limit cooldown also
time out — leaving slash commands broken for ~60 min until the bucket
fully recovers.

Bump the timeout to 600 s so realistic bursts drain, update the warning
message to point at the saturated bucket instead of a hardcoded 30 s.
The 600 s cap still guards against a true hang.

Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting
NousResearch#16713 with the bucket-math diagnosis.

Closes NousResearch#16713.

Co-authored-by: Teknium <teknium@nousresearch.com>
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
…ure (NousResearch#16713) (NousResearch#17029)

Discord's per-app command-management bucket is ~5 writes / 20 s. A
mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes
in the reported case) can't finish under the old flat 30 s budget, and
the subsequent reconnect retries inside the rate-limit cooldown also
time out — leaving slash commands broken for ~60 min until the bucket
fully recovers.

Bump the timeout to 600 s so realistic bursts drain, update the warning
message to point at the saturated bucket instead of a hardcoded 30 s.
The 600 s cap still guards against a true hang.

Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting
NousResearch#16713 with the bucket-math diagnosis.

Closes NousResearch#16713.

Co-authored-by: Teknium <teknium@nousresearch.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/discord Discord bot adapter type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

_safe_sync_slash_commands times out under post-prune rate-limit pressure (30s budget too tight)

4 participants