fix(qqbot): prevent event-loop busy-spin when WebSocket is half-closed - #29057
Closed
xieyuanqing wants to merge 1 commit into
Closed
fix(qqbot): prevent event-loop busy-spin when WebSocket is half-closed#29057xieyuanqing wants to merge 1 commit into
xieyuanqing wants to merge 1 commit into
Conversation
After a reconnect attempt fails before _open_ws can replace self._ws
(for example when _get_gateway_url returns an empty body), the adapter
is left with a stale, already-closed ClientWebSocketResponse. The next
iteration of _listen_loop re-enters _read_events, whose while-condition
is immediately False, so the coroutine returns normally. _listen_loop
then loops again without any suspending await, and the whole asyncio
event loop is starved at 100% CPU.
Observed in production: a single failed QQ gateway reconnect at 06:35
froze the process for 4+ hours. The main thread was pinned at
~100% CPU, all other platforms (Telegram, cron, heartbeats) stopped
firing, and 'systemctl restart' could not deliver SIGTERM because the
asyncio loop never yielded long enough to run the signal handler. The
process had to be SIGKILL'd.
Fix:
* Treat 'self._ws is None or self._ws.closed' as a precondition
failure in _read_events and raise so _listen_loop reaches its
reconnect path instead of busy-looping.
* Document the invariant in the docstring so future edits do not
silently regress it.
Includes regression tests that fail (hang up to a 1s wait_for) without
the guard.
Collaborator
This was referenced May 22, 2026
Closed
Closed
2 tasks
Contributor
|
This has been implemented on current Evidence from this automated hermes-sweeper review:
Thanks for the detailed production root-cause writeup; it matches the behavior now fixed on main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gateway/platforms/qqbot/adapter.py::QQAdapter._read_eventscan returnnormally on a stale, already-closed
ClientWebSocketResponse. When thathappens the outer
_listen_loopre-enters_read_eventsimmediatelywithout any awaitable actually suspending, the asyncio event loop is
starved, and the whole gateway hangs at 100% CPU. This PR makes
_read_eventsraise in that situation so_listen_loopreaches itsreconnect path, and adds regression tests.
Trigger / observed failure
In production a single failed QQ-Bot reconnect at
06:35:14(gatewayURL request returned an empty body) froze the gateway for 4h 24m until
operator intervention.
Log evidence right before the hang:
py-spy dumpagainst the wedged process (MainThreadat ~100% CPU, allother threads idle):
Side-effects of the hang:
heartbeats) silently stopped firing.
systemctl restartcould not shut the process down — the asynciosignal handler never ran because the loop was starved. The process
had to be
SIGKILL'd.Root cause
When
_reconnect()fails before_open_ws()is entered (e.g._get_gateway_url()raises or returns empty),self._wsis leftpointing at the previous, already-closed
ClientWebSocketResponse— it is not
None, andself._ws.closedisTrue.The next
_listen_loopiteration calls_read_events:The function returns. Back in
_listen_loop:Because no
awaitin this path actually yields (_read_eventsonlysuspends inside
self._ws.receive(), which is unreachable), CPythonholds the GIL and the asyncio loop never runs any other task. From the
outside the process looks alive (PID present,
Active: running) but iscompletely unresponsive.
Fix
self._ws is None or self._ws.closedas a precondition failurein
_read_eventsandraise RuntimeError("WebSocket not connected").This routes control back to
_listen_loop, which already handles theexception path and goes through
_mark_transport_disconnected()+_reconnect().via
raiseor viaself._runningbeing cleared — never returnnormally on a non-functional socket) so future edits don't silently
reintroduce the regression.
The change is intentionally small. The reconnect machinery itself is
already correct; the bug is specifically that
_read_eventsis toopermissive about what counts as a runnable read.
Tests
Added
tests/gateway/test_qqbot.py::TestReadEventsClosedSocket:test_raises_when_ws_is_none— preserves the existing contract.test_raises_when_ws_is_closed— pins the new behaviour and assertsreceive()is never even called on a closed socket.test_does_not_busy_spin_on_closed_socket— wraps the call inasyncio.wait_for(..., timeout=1.0)so a future regression surfacesas a CI timeout instead of an indefinite hang.
Full QQ-Bot test module passes locally:
Risk / compatibility
_ws is not None and _ws.closed is True, which previously caused a silent infinite loop. In everybranch where
_listen_loopalready expected an exception (transientWS error, server CLOSE, etc.) the new path is equivalent.