customcommands: refine delayed execCC limits

[jo3-l]

This PR proposes two changes to delayed execCC¹ limits.

1. We make the ratelimit per-guild instead of per-channel (with parameters adjusted appropriately.)
2. We allow certain executions that would otherwise be ratelimited (and thus silently dropped) to go forward after a delay if the ratelimit expiry is not too far in the future.
   • Concretely, consider an execCC execution that is ratelimited (and where the ratelimit expires after 1s.) Currently, this execution is silently dropped, but this PR would instead block the handling goroutine for 1 second then execute the custom command as usual.

Current limits on execCC

To provide some additional context between the changes here, we must first describe the status quo. At present, execCC calls with nonzero delay must go through the scheduled event system, and so require additional limits to not overload YAGPDB. A per-channel token bucket (see also the x/time/rate doc) is therefore used to ratelimit delayed execCC executions as they arrive. Each token bucket starts with 10 tokens and is replenished at a rate of 1/10s. Each execution consumes 1 token; if there are no tokens available, the request is ratelimited and does not proceed: it is silently ignored. Generally speaking, then, the current limit can be understood as follows: delayed execCC calls are limited per-channel to 1/10s; however, if some time has gone by with no execCC calls, up to 10 execCC calls are permitted in short succession.²

Why current limits are imperfect

The current approach has two minor flaws:

1. First, the limit is per-channel, not per-guild, leading to an awkward, semi-supported, workaround where users set execCC to use random target channels to reduce the chance that they run afoul of the limit. Ideally, this workaround would not be necessary, motivating the change to a per-guild limit. A per-guild limit is also more robust: we want to cap the load that a single guild can place on YAGPDB, not a single channel.
2. Second, ratelimited requests are silently dropped. This behavior is unintuitive and makes tweaking the limits more difficult, as a slight change might ratelimit previously working execCC calls (causing them to no longer execute and associated user complaints.) We therefore allow executions that only slightly exceed the limit to proceed after an appropriate delay; this change maintains the effectiveness of the limit while minimizing breakage.
   • This change also makes userland code that wishes to execute a custom command but does not care about the specific delay (think mass action custom commands) slightly more robust: if the execution skirts the limit by a little, it will no longer randomly halt but will instead proceed after the proper delay.

---

Points of discussion

• What should the per-guild limit be? I have generously configured the token bucket to replenish at a rate of 5/s and allow a burst of 25, though this limit may be too generous.
• What is the maximum delay that we can tolerate? I have set this parameter to 10 seconds for now (so an execCC call ratelimited for up to 10 seconds will still be processed, but one ratelimited for, say, 15 seconds, would still be dropped.)

Footnotes

1. Note that execCC calls with 0 delay are not affected by this change, nor do future revisions of this change aim to limit them further. Those calls do not go through the scheduled event system and are considerably less costly to handle. ↩

2. This description is imperfect as we do not consider the latency of individual execCC invocations but it suffices as a first approximation. ↩

[jo3-l]

Whoops, sent this before I finished writing the description 😔 Will edit.