Feishu WebSocket disconnect causes full gateway restart every ~30min

[Seekers2001]

Summary

When Feishu (Lark) is connected via WebSocket, the connection drops approximately every 30 minutes with:

[Lark] ERROR: receive message loop exit, err: no close frame received or sent
→ Stopping gateway for restart...

This triggers a full gateway restart (all adapters disconnect/reconnect, shutdown notification sent to all active sessions), rather than just reconnecting the Feishu WebSocket.

Impact

• 48 restarts per day (every ~30 min, around the clock)
• Each restart sends a verbose notification to all active chat sessions
• Active tasks get interrupted unnecessarily
• DingTalk and other platforms also disconnect/reconnect even though they're fine

Expected behavior

Feishu WebSocket reconnection should be handled at the adapter level (reconnect the WebSocket) without restarting the entire gateway process. Other platforms (DingTalk, Slack, etc.) should be unaffected.

Environment

• Hermes Agent v0.14.0 (2026.5.16)
• macOS 26.5 (Apple Silicon)
• Connected platforms: DingTalk (stream mode) + Feishu (websocket mode)
• Installed via Homebrew

Logs

2026-05-24 12:33:34 [Lark] ERROR: receive message loop exit, err: no close frame received or sent [conn_id=...]
2026-05-24 12:33:34 [Lark] INFO: disconnected to wss://msg-frontier.feishu.cn/...
2026-05-24 12:33:40 INFO: Stopping gateway for restart...
2026-05-24 12:33:40 INFO: ✓ dingtalk disconnected
2026-05-24 12:33:40 INFO: ✓ feishu disconnected
2026-05-24 12:33:42 INFO: Starting Hermes Gateway...
2026-05-24 12:33:42 INFO: ✓ dingtalk connected
2026-05-24 12:33:42 INFO: ✓ feishu connected

The 30-minute cycle is consistent — see restart timestamps from a single day:

00:53, 01:23, 01:53, 02:23, 02:53, 03:24, 03:54, 04:24, 04:54, 05:25, 05:55, 06:25, 06:55, 07:26, 07:56, 08:26, 08:56, 09:26, 09:57, 10:27, 10:57, 11:27, 11:57, 12:33, 13:03, 13:34, 14:04, 14:34, 15:04, 15:35

Suggested fix

Handle Feishu WebSocket no close frame errors at the adapter level:

1. Catch the disconnect in the Feishu adapter
2. Attempt WebSocket reconnection (with backoff)
3. Only escalate to full gateway restart if reconnection fails after N retries

This aligns with how most messaging SDKs handle transient WebSocket disconnects gracefully.

[GottZ]

This was generated by AI during triage.

Summary

Four PRs address the Feishu WebSocket reliability complex. #5500 improves SDK lifecycle and reconnect tuning, #31386 observes silent worker completion but implements recovery on the obsolete module path, #61271 reports worker exits from the current plugin but has a shutdown race and unrelated session changes, and #64741 narrowly routes SDK reconnect exhaustion into the existing per-platform supervisor.

Related pull requests

• Fix/feishu reconnect and shutdown #5500 [closed] partial — (+306/-18) — partial fix: Improves Feishu WebSocket lifecycle cleanup, handler rebuilding, and SDK reconnect/ping configuration, but does not notify the gateway when ws_client.start() returns after reconnect exhaustion. Although closed, it remains relevant as a merged reference implementation via fix: batch gateway/platform fixes — matrix E2EE, CJK input, Windows browser, Feishu reconnect + ACL #5665.
• fix(feishu): adapter-level WS reconnect on receive-loop death (#31367) #31386 partial — (+449/-0) — partial fix: Adds completion observation and a bounded adapter-local reconnect ladder, directly targeting the silent receive-loop exit, but it modifies the obsolete gateway/platforms/feishu.py path and duplicates the existing supervisor retry policy. Despite the keep_open review on fix(feishu): adapter-level WS reconnect on receive-loop death (#31367) #31386, the current diff is superseded by fix(feishu): detect Lark SDK reconnect exhaustion and trigger reconnection #64741 because fix(feishu): detect Lark SDK reconnect exhaustion and trigger reconnection #64741 applies the worker-exit signal to the current plugin and delegates recovery to the established per-platform watcher, matching that review's requested architecture.
• fix: Feishu WS reconnect on disconnect + session context recovery on auto-reset #61271 partial — (+46/-5) — partial fix: Detects exceptional and normal worker exits in the current plugin and reports a retryable fatal error, but the exception path can also fire during intentional teardown; the diff additionally mixes in unrelated session-context recovery without serialization or regression tests. The keep_open review on fix: Feishu WS reconnect on disconnect + session context recovery on auto-reset #61271 is consistent with retaining its salvageable session work, but its Feishu portion should be superseded by the narrower fix(feishu): detect Lark SDK reconnect exhaustion and trigger reconnection #64741 implementation.
• fix(feishu): detect Lark SDK reconnect exhaustion and trigger reconnection #64741 best fix — (+40/-0) — fixes: On the current plugin path, detects ws_client.start() completion after SDK retry exhaustion, guards intentional disconnect with _running, and reports a retryable fatal error to the existing per-platform reconnect watcher instead of restarting the full gateway. The keep_open review identifies one remaining requirement before merge: regression tests for unexpected normal return, notification, and the intentional-disconnect guard.

Duplicates

#31386, the Feishu portion of #61271, and #64741 overlap on detecting an exited Feishu WS worker; #64741 is the narrow current-main implementation, while #31386 adds a redundant local retry ladder and #61271 mixes in a distinct session-recovery change.

Suggested consolidation

Merge #64741 after adding the regression tests requested by its keep_open review; it is the smallest current-path fix and reuses the gateway's existing per-platform supervisor. Close #31386 as superseded despite its keep_open review because its diff targets the removed module path and duplicates recovery policy; supersede the Feishu portion of #61271 with #64741, but split or independently revise its non-duplicate session-context work rather than treating the entire PR as a duplicate. #5500 is already closed and merged through #5665, but remains only a partial reference for lifecycle handling.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I31367(["issue #31367 (open)"])
    I46561(["issue #46561 (open)"])
    I64712(["issue #64712 (open)"])
    P5500["PR #5500 (closed)"]
    P61271["PR #61271 (open)"]
    subgraph Dup31386 ["PRs duplicating each other"]
        P31386["PR #31386 (open)"]
        P64741["PR #64741 (open)"]
    end
    P5500 -.->|partial| I31367
    P31386 -.->|partial| I31367
    P61271 -.->|partial| I31367
    P64741 -->|best fix| I31367
    P5500 -.->|partial| I46561
    P31386 -.->|partial| I46561
    P61271 -.->|partial| I46561
    P64741 -->|best fix| I46561
    P5500 -.->|partial| I64712
    P31386 -.->|partial| I64712
    P61271 -.->|partial| I64712
    P64741 -->|best fix| I64712
    class I31367 open
    class I46561 open
    class I64712 open
    class I31367 target
    class P5500 closed
    class P31386 open
    class P61271 open
    class P64741 open
    class P64741 best
    class P64741 best
    class P64741 best
    click I31367 "https://github.com/NousResearch/hermes-agent/issues/31367"
    click I46561 "https://github.com/NousResearch/hermes-agent/issues/46561"
    click I64712 "https://github.com/NousResearch/hermes-agent/issues/64712"
    click P5500 "https://github.com/NousResearch/hermes-agent/pull/5500"
    click P31386 "https://github.com/NousResearch/hermes-agent/pull/31386"
    click P61271 "https://github.com/NousResearch/hermes-agent/pull/61271"
    click P64741 "https://github.com/NousResearch/hermes-agent/pull/64741"

Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed or no verify verdict yet (state tag in the node label).

Cross-PR triage: Reviewed 4 pull requests and 3 issues in this complex. Each diff was read against this issue; Assessment working set: 54 kB of PR diffs, 22 kB of issue/PR text, 5 kB of discussion (7 comments), 16 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.