fix(discord): scale slash sync timeout to actual write count (#16713) - #16739
Tranquil-Flow wants to merge 1 commit into
Conversation
…earch#16713) ``_run_post_connect_initialization`` wrapped the entire ``_safe_sync_slash_commands`` call in ``asyncio.wait_for(..., 30)``. Discord's per-app command-management bucket allows ~5 writes / 20-second window, so a mass-prune-plus-upsert reconcile (the reported case had 77 orphans + 30 desired = 107 writes) reliably blew the 30 s budget under back-pressure. The post-connect handler then returned a TimeoutError and the gateway retried on the next reconnect, only succeeding ~60 min later once the bucket fully recovered. Split ``_safe_sync_slash_commands`` into ``_plan_slash_command_sync`` (read-only diff) + ``_execute_slash_command_sync`` (writes). The post-connect path now plans first under a 30 s budget, computes a write-count-aware execute budget (30 s base + 5 s per write, capped at 600 s), then runs the plan under that budget. ``_safe_sync_slash_commands`` keeps its existing single-shot signature for callers/tests, layered on top of the plan + execute helpers. Recreate operations count as two writes (delete + upsert) so the budget covers Discord's actual rate-limit math. Logs the timeout cause as a saturated rate-limit bucket instead of a flat "after 30 s" message.
…ure (#16713) (#17029) Discord's per-app command-management bucket is ~5 writes / 20 s. A mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes in the reported case) can't finish under the old flat 30 s budget, and the subsequent reconnect retries inside the rate-limit cooldown also time out — leaving slash commands broken for ~60 min until the bucket fully recovers. Bump the timeout to 600 s so realistic bursts drain, update the warning message to point at the saturated bucket instead of a hardcoded 30 s. The 600 s cap still guards against a true hang. Credit to @Tranquil-Flow for PR #16739 and @davidbordenwi for reporting #16713 with the bucket-math diagnosis. Closes #16713. Co-authored-by: Teknium <teknium@nousresearch.com>
|
Thanks for the clean write-up and the bucket-math diagnosis — both were load-bearing for the fix. Merged a simpler variant via #17029 / commit on main: same root cause (flat 30 s budget vs Discord's ~5-writes/20-s command-management bucket), but just widens the outer Credit to @Tranquil-Flow and @davidbordenwi in the merge commit. Closing in favor of #17029. |
…ure (#16713) (#17029) Discord's per-app command-management bucket is ~5 writes / 20 s. A mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes in the reported case) can't finish under the old flat 30 s budget, and the subsequent reconnect retries inside the rate-limit cooldown also time out — leaving slash commands broken for ~60 min until the bucket fully recovers. Bump the timeout to 600 s so realistic bursts drain, update the warning message to point at the saturated bucket instead of a hardcoded 30 s. The 600 s cap still guards against a true hang. Credit to @Tranquil-Flow for PR #16739 and @davidbordenwi for reporting #16713 with the bucket-math diagnosis. Closes #16713. Co-authored-by: Teknium <teknium@nousresearch.com>
…ure (NousResearch#16713) (NousResearch#17029) Discord's per-app command-management bucket is ~5 writes / 20 s. A mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes in the reported case) can't finish under the old flat 30 s budget, and the subsequent reconnect retries inside the rate-limit cooldown also time out — leaving slash commands broken for ~60 min until the bucket fully recovers. Bump the timeout to 600 s so realistic bursts drain, update the warning message to point at the saturated bucket instead of a hardcoded 30 s. The 600 s cap still guards against a true hang. Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting NousResearch#16713 with the bucket-math diagnosis. Closes NousResearch#16713. Co-authored-by: Teknium <teknium@nousresearch.com>
|
Sharing a related data point from a Windows Discord gateway deployment. We saw intermittent Discord send 429 pressure around long/chunked replies. A local mitigation that helped was pacing outbound Discord This did not look related to a separate silent |
…ure (NousResearch#16713) (NousResearch#17029) Discord's per-app command-management bucket is ~5 writes / 20 s. A mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes in the reported case) can't finish under the old flat 30 s budget, and the subsequent reconnect retries inside the rate-limit cooldown also time out — leaving slash commands broken for ~60 min until the bucket fully recovers. Bump the timeout to 600 s so realistic bursts drain, update the warning message to point at the saturated bucket instead of a hardcoded 30 s. The 600 s cap still guards against a true hang. Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting NousResearch#16713 with the bucket-math diagnosis. Closes NousResearch#16713. Co-authored-by: Teknium <teknium@nousresearch.com>
…ure (NousResearch#16713) (NousResearch#17029) Discord's per-app command-management bucket is ~5 writes / 20 s. A mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes in the reported case) can't finish under the old flat 30 s budget, and the subsequent reconnect retries inside the rate-limit cooldown also time out — leaving slash commands broken for ~60 min until the bucket fully recovers. Bump the timeout to 600 s so realistic bursts drain, update the warning message to point at the saturated bucket instead of a hardcoded 30 s. The 600 s cap still guards against a true hang. Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting NousResearch#16713 with the bucket-math diagnosis. Closes NousResearch#16713. Co-authored-by: Teknium <teknium@nousresearch.com>
…ure (NousResearch#16713) (NousResearch#17029) Discord's per-app command-management bucket is ~5 writes / 20 s. A mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes in the reported case) can't finish under the old flat 30 s budget, and the subsequent reconnect retries inside the rate-limit cooldown also time out — leaving slash commands broken for ~60 min until the bucket fully recovers. Bump the timeout to 600 s so realistic bursts drain, update the warning message to point at the saturated bucket instead of a hardcoded 30 s. The 600 s cap still guards against a true hang. Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting NousResearch#16713 with the bucket-math diagnosis. Closes NousResearch#16713. Co-authored-by: Teknium <teknium@nousresearch.com>
…ure (NousResearch#16713) (NousResearch#17029) Discord's per-app command-management bucket is ~5 writes / 20 s. A mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes in the reported case) can't finish under the old flat 30 s budget, and the subsequent reconnect retries inside the rate-limit cooldown also time out — leaving slash commands broken for ~60 min until the bucket fully recovers. Bump the timeout to 600 s so realistic bursts drain, update the warning message to point at the saturated bucket instead of a hardcoded 30 s. The 600 s cap still guards against a true hang. Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting NousResearch#16713 with the bucket-math diagnosis. Closes NousResearch#16713. Co-authored-by: Teknium <teknium@nousresearch.com>
…ure (NousResearch#16713) (NousResearch#17029) Discord's per-app command-management bucket is ~5 writes / 20 s. A mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes in the reported case) can't finish under the old flat 30 s budget, and the subsequent reconnect retries inside the rate-limit cooldown also time out — leaving slash commands broken for ~60 min until the bucket fully recovers. Bump the timeout to 600 s so realistic bursts drain, update the warning message to point at the saturated bucket instead of a hardcoded 30 s. The 600 s cap still guards against a true hang. Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting NousResearch#16713 with the bucket-math diagnosis. Closes NousResearch#16713. Co-authored-by: Teknium <teknium@nousresearch.com>
…ure (NousResearch#16713) (NousResearch#17029) Discord's per-app command-management bucket is ~5 writes / 20 s. A mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes in the reported case) can't finish under the old flat 30 s budget, and the subsequent reconnect retries inside the rate-limit cooldown also time out — leaving slash commands broken for ~60 min until the bucket fully recovers. Bump the timeout to 600 s so realistic bursts drain, update the warning message to point at the saturated bucket instead of a hardcoded 30 s. The 600 s cap still guards against a true hang. Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting NousResearch#16713 with the bucket-math diagnosis. Closes NousResearch#16713. Co-authored-by: Teknium <teknium@nousresearch.com>
…ure (NousResearch#16713) (NousResearch#17029) Discord's per-app command-management bucket is ~5 writes / 20 s. A mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes in the reported case) can't finish under the old flat 30 s budget, and the subsequent reconnect retries inside the rate-limit cooldown also time out — leaving slash commands broken for ~60 min until the bucket fully recovers. Bump the timeout to 600 s so realistic bursts drain, update the warning message to point at the saturated bucket instead of a hardcoded 30 s. The 600 s cap still guards against a true hang. Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting NousResearch#16713 with the bucket-math diagnosis. Closes NousResearch#16713. Co-authored-by: Teknium <teknium@nousresearch.com>
…ure (NousResearch#16713) (NousResearch#17029) Discord's per-app command-management bucket is ~5 writes / 20 s. A mass-prune-plus-upsert reconcile (77 orphans + 30 desired = 107 writes in the reported case) can't finish under the old flat 30 s budget, and the subsequent reconnect retries inside the rate-limit cooldown also time out — leaving slash commands broken for ~60 min until the bucket fully recovers. Bump the timeout to 600 s so realistic bursts drain, update the warning message to point at the saturated bucket instead of a hardcoded 30 s. The 600 s cap still guards against a true hang. Credit to @Tranquil-Flow for PR NousResearch#16739 and @davidbordenwi for reporting NousResearch#16713 with the bucket-math diagnosis. Closes NousResearch#16713. Co-authored-by: Teknium <teknium@nousresearch.com>
What does this PR do?
_run_post_connect_initializationingateway/platforms/discord.pywraps the entire_safe_sync_slash_commandscall inasyncio.wait_for(..., 30). Discord's per-app command-management bucket allows roughly 5 writes / 20-second window, so a mass-prune-plus-upsert reconcile reliably blows the 30 s budget under back-pressure. The reported case had 77 orphans + 30 desired = 107 writes; two consecutive 30 s timeouts, then a clean 22 s sync only after the bucket fully recovered ~60 minutes later.This PR splits the read-only diff from the writes and sizes the execute-phase timeout to the actual workload, so a heavy reconcile gets enough budget to finish under bucket pressure but pathological loads still get a hard cap.
Related Issue
Fixes #16713
Type of Change
Changes Made
gateway/platforms/discord.py— extracted two helpers from_safe_sync_slash_commands:_plan_slash_command_sync()— read-only; diffs desired vs existing global commands, returns a list of typed actions (create/update/recreate) plus the orphan deletes and awrite_count._execute_slash_command_sync(plan)— applies the plan, returns the existing summary dict._safe_sync_slash_commandsis now a thin wrapper that calls plan + execute, preserving the single-shot signature for callers and existing tests.gateway/platforms/discord.py:_run_post_connect_initialization— now plans first under a 30 s budget, then computes a write-count-aware execute budget (30 + 5 × write_count, capped at 600 s) before running the plan.recreateactions count as two writes (delete + upsert) so the budget covers Discord's actual rate-limit math. Timeout log message updated to point at saturated rate-limit bucket as the cause instead of a flat "after 30 s"._SLASH_SYNC_BASE_BUDGET_SECONDS,_SLASH_SYNC_PER_WRITE_SECONDS,_SLASH_SYNC_MAX_BUDGET_SECONDS) and a static_estimate_slash_sync_budget(write_count)so the budget formula is testable and trivially tunable.tests/gateway/test_discord_connect.py— added five tests:_estimate_slash_sync_budgetscales with write count, is monotonic, caps at the maximum._plan_slash_command_synccounts arecreateas two writes plus an orphan delete (matches Discord's bucket math)._safe_sync_slash_commandssingle-shot path (so callers/tests that read the summary stay correct).How to Test
Reproduction (matches the issue body): trigger a session that produces ≥ ~30 orphan slash commands (e.g., a command-registry refactor that drops or renames commands), restart the gateway, and watch the post-connect initialization. Before this fix,
wait_for(..., 30)raisedasyncio.TimeoutErrorand subsequent reconnects within the rate-limit cooldown also timed out. After this fix, the budget scales with the planned write count and the reconcile completes under bucket pressure.Automated:
Result on macOS 15.6.1 / Python 3.14.2:
16 passed(11 pre-existing + 5 new). All five new tests fail onmainwithout the production change.Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests passDocumentation & Housekeeping
docs/, docstrings) — or N/A (new helpers carry inline docstrings explaining the bucket math)cli-config.yaml.exampleif I added/changed config keys — or N/A (N/A — budget tuning lives on adapter class constants; not user-facing yaml)CONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/A (N/A)Screenshots / Logs