Skip to content

fix(qqbot): prevent busy-loop when WebSocket is closed but not None - #40497

Closed
xushibo wants to merge 1 commit into
NousResearch:mainfrom
xushibo:fix/qqbot-spin-loop-closed-websocket
Closed

fix(qqbot): prevent busy-loop when WebSocket is closed but not None#40497
xushibo wants to merge 1 commit into
NousResearch:mainfrom
xushibo:fix/qqbot-spin-loop-closed-websocket

Conversation

@xushibo

@xushibo xushibo commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

Bug Description

After QQBot WebSocket disconnect and a failed reconnect attempt, the gateway process enters a tight CPU-bound loop consuming 100% CPU indefinitely with no further reconnect logging.

Fixes #31771
Fixes #31193

Root Cause

In gateway/platforms/qqbot/adapter.py, _read_events() returns silently when self._ws is a closed aiohttp WebSocket (non-None but .closed==True). The outer _listen_loop() sees this as a normal return, resets backoff_idx to 0, and immediately re-enters — a zero-delay spin.

Fix

Add an early guard: if self._ws.closed is True on entry to _read_events(), raise RuntimeError, which is caught by the existing reconnect/backoff handler in _listen_loop().

Change: 2 lines added to gateway/platforms/qqbot/adapter.py

+        if self._ws.closed:
+            raise RuntimeError("WebSocket closed")

How to Verify

  1. Let QQBot WebSocket session expire (happens every ~30 min naturally)
  2. Observe that reconnect attempts continue with proper backoff logging
  3. Verify gateway CPU stays at idle (<5%) even after multiple reconnect cycles

Test Plan

  • Deployed to local gateway — CPU dropped from 99.5% to 0.0% after restart
  • Watchdog script confirms no recurrence over monitoring period

Risk Assessment

Low — the added check only fires when WebSocket is already in a terminal state. The existing exception handler in _listen_loop() already handles RuntimeError with proper backoff and reconnect logic. No change to the happy path.

_read_events() returned silently when self._ws was a closed aiohttp
WebSocket (non-None but .closed==True), causing _listen_loop() to
reset backoff_idx and immediately re-enter — a tight CPU-bound loop
consuming 100% CPU indefinitely.

This fix adds an early check: if self._ws.closed is True on entry,
raise RuntimeError, which is caught by the existing reconnect/backoff
handler in _listen_loop().

Closes NousResearch#31771
Closes NousResearch#31193
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists platform/qqbot QQ Bot adapter comp/gateway Gateway runner, session dispatch, delivery labels Jun 6, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related to the QQBot closed-WebSocket busy-loop cluster (root #17703) with several competing fixes: #31333, #31774, #29057, #27821, #20994. Maintainers should pick one canonical fix to land.

@teknium1

teknium1 commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

Salvaged into #40574 with your authorship credited (Co-authored-by). #40497 and #40291 were byte-identical fixes for this busy-loop — folded both in. Added regression tests. Thanks!

#40574

@wuandx

wuandx commented Jun 8, 2026

Copy link
Copy Markdown

Reproduction:

  1. Hermes gateway with QQ Bot platform connected via aiohttp WebSocket
  2. After the QQ server disconnects (e.g., session timeout at ~30 min intervals), ws.receive() returns a message type not handled by the branch chain — the while loop continues calling receive() without blocking
  3. Measured result: 73,661 receive() calls in 3 seconds, event loop at 100% CPU

Confirmation on production (twice):

Thread MainThread (active+gil):
    _read_events
    _listen_loop

Main thread stuck in _read_events at 99.7% CPU. All other threads sleeping. ~48 hours uptime before each trigger.

Same behavior was fixed in WeCom adapter (commit 8dca287, 2026-05-18): aiohttp WSMsgType.CLOSING was not handled by the message type branch chain, causing the same CPU spin.

Environment where reproduced:

  • Hermes v2026.6.5-202-ge3b8b6d32
  • aiohttp 3.13.3
  • Python 3.11.14
  • Linux 7.0.10-zen1-1-zen x86_64

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/qqbot QQ Bot adapter type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: QQBot adapter busy-loops after WebSocket reconnect failure, causing 100% CPU QQ Bot Reconnect Busy Loop Causes 100% CPU Spin

4 participants