Conversation
The Exception handler (no close code) was missing: 1. Window-based disconnect counting — 5+ disconnects in 300s forces a fresh Identify instead of Resume, breaking the 60s idle-timeout death-loop on QQ's TCP frontend. 2. Quick-disconnect detection (mirrors QQCloseError branch). 3. Backoff with MAX_RECONNECT_ATTEMPTS upper bound — previously backoff_idx was reset to 0 on every successful reconnect, making it impossible to ever give up. Root cause: QQ's TCP-layer idle timeout (~60s) is not reset by WebSocket heartbeat frames on the load-balancer tier, so Resume sessions with no message traffic enter an infinite reconnect cycle.
Contributor
Contributor
jiqfu
added a commit
to jiqfu/hermes-agent
that referenced
this pull request
May 1, 2026
Cherry-pick PR NousResearch#17990 from yunkai: fix exception-disconnect window tracking, quick-disconnect detection, and backoff upper bound for the except Exception branch in _listen_loop(). Fixes: 5+ exception-disconnects within 300s with same session_id now triggers session clear → forced fresh Identify, breaking the infinite Resume death-loop.
teknium1
reviewed
Jul 12, 2026
teknium1
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for isolating the generic exception-path loop. Current main still resets reconnect state after every successful socket reopen (gateway/platforms/qqbot/adapter.py:651-655), so this remains worth salvaging.
Problems
- The added quick-disconnect increment at PR line 579 is undone by the unchanged success arm at PR lines 606-608. A reconnect-success-immediate-disconnect cycle therefore never reaches
MAX_QUICK_DISCONNECT_COUNT. - The advertised backoff cap has the same issue: PR line 607 resets
backoff_idx, so the existing cap cannot bound a cycle with successful socket opens. - The diff changes only the adapter; it adds no regression test. Current
tests/gateway/test_qqbot.py:2220-2241covers the closed-socket raise, not repeated generic failures after successful reconnects. - Current main now requires
_mark_transport_disconnected()for this path (gateway/platforms/qqbot/adapter.py:643; commit8199ec380), so the older_mark_disconnected()context should not be carried forward.
Suggested changes
- Preserve a loop-detection counter until a connection survives the configured threshold, and test successful reconnect followed by immediate generic failure.
- Keep current transport/lifecycle separation while salvaging the recovery policy.
Automated hermes-sweeper review.
| # --- Quick disconnect detection (mirror QQCloseError branch) --- | ||
| duration = time.monotonic() - connect_time | ||
| if duration < QUICK_DISCONNECT_THRESHOLD and connect_time > 0: | ||
| quick_disconnect_count += 1 |
Collaborator
There was a problem hiding this comment.
This increment is reset by the unchanged successful-reconnect arm at lines 606-608. In the reported reconnect-success-immediate-close cycle, the counter therefore stays at 1 and never reaches MAX_QUICK_DISCONNECT_COUNT; retain it until a connection survives the threshold or reset it from a confirmed healthy-session event.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
except Exceptionbranch in QQ Bot's_listen_loophandles WebSocket disconnects without a close code — typically triggered by QQ's TCP-layer idle timeout (~60s). This branch was missing three safety mechanisms present in theQQCloseErrorbranch, causing the gateway to enter an infinite Resume death-loop: disconnect → Resume → disconnect every 60s, forever.Pre-fix logs (April 29, 08:36–08:58)
Same
session_idpersists across all cycles,seqincrements endlessly. QQ's TCP frontend does not reset its idle timer on WebSocket heartbeat frames, so Resume can never succeed without message traffic. Once rate-limited, the bot stays permanently offline — observed in production: 26+ hours of downtime.Fix
Three mechanisms added to the
except Exceptionbranch:Exception-disconnect window tracking + forced Identify: Count disconnects in a 300-second sliding window. After 5 disconnects with the same
session_id, clear the session to force a fresh Identify on the next reconnect — breaking the death-loop.Quick-disconnect detection: Mirror existing logic from the
QQCloseErrorbranch. Three disconnects within 5 seconds of connection triggers a fatal error with actionable guidance.Backoff with upper bound:
backoff_idxnow increments on reconnect failures and is checked againstMAX_RECONNECT_ATTEMPTS(100). Previouslybackoff_idxwas reset to 0 on every successful reconnect, making it impossible to ever give up.Related: #15051
This PR builds on the direction of #15051 (adding quick-disconnect detection to the Exception path) and extends it with window counting + forced Identify + backoff cap — all three mechanisms together provide a complete defense against the death-loop.
Post-fix validation (April 30)
session_idon each Identify