Skip to content

fix(feishu): per-IP WS connect failover for msg-frontier DNS black holes - #89929

Open
zmlgit wants to merge 1 commit into
NousResearch:mainfrom
zmlgit:feishu/ws-ip-failover
Open

fix(feishu): per-IP WS connect failover for msg-frontier DNS black holes#89929
zmlgit wants to merge 1 commit into
NousResearch:mainfrom
zmlgit:feishu/ws-ip-failover

Conversation

@zmlgit

@zmlgit zmlgit commented Aug 19, 2026

Copy link
Copy Markdown

Problem

msg-frontier.feishu.cn resolves to a ~12-IP pool in which roughly three are black holes: the TCP or TLS handshake hangs indefinitely with no RST. websockets.connect() walks the resolved addresses sequentially under a single open_timeout, so when the first address is a black hole the entire connect times out and the adapter cannot come up at all until DNS order rotates.

Observed in production (multi-profile gateway, CN network): adapters stuck in connect loops for minutes whenever the frontier DNS round-robin returned a bad IP first.

Fix

Resolve the frontier hostname ourselves (socket.getaddrinfo), then try each unique IP with a short per-IP deadline (_WS_CONNECT_PER_IP_TIMEOUT = 4.0s, module constant), keeping the first TLS+WS handshake that completes. Wired into the existing _connect_with_overrides wrapper of websockets.connect that _run_official_feishu_ws_client already installs — no SDK method patching involved.

Upgrade path: replace with happy eyeballs (RFC 8305) if websockets grows native support; drop entirely if the DNS pool is cleaned up.

Regression test

test_ws_connect_ip_failover_skips_blackhole_ips drives the patched connect through the module-global loop with the first resolved IP hanging forever and asserts the second IP completes the handshake (attempts: black hole first, working IP second).

Scope

Split out of #64247 per review (@Seekers2001) — solves a different problem than the event-loop isolation patch (#89928) and was making the root-cause fix harder to review. Independent of #89928: this PR applies cleanly on plain main without the loop-proxy work.

msg-frontier.feishu.cn resolves to a ~12-IP pool in which roughly three
are black holes: the TCP or TLS handshake hangs indefinitely with no RST.
websockets.connect() walks the resolved addresses sequentially under a
single open_timeout, so when the first address is a black hole the entire
connect times out and the adapter cannot come up at all until DNS order
rotates.

Resolve the frontier hostname ourselves and try each unique IP with a
short per-IP deadline (_WS_CONNECT_PER_IP_TIMEOUT, 4s), keeping the first
TLS+WS handshake that completes. Replace with happy eyeballs (RFC 8305)
if websockets grows native support; drop entirely if the DNS pool is
cleaned up.

Regression test drives the patched connect through the module-global
loop with the first resolved IP hanging and asserts the second IP
completes the handshake (attempts: black hole first, working IP second).

Split out of NousResearch#64247 per review: this solves a different problem than the
event-loop isolation patch and was making the root-cause fix harder to
review.
@alt-glitch alt-glitch added type/bug Something isn't working comp/plugins Plugin system and bundled plugins platform/feishu Feishu / Lark adapter P3 Low — cosmetic, nice to have labels Aug 19, 2026
@Enough1122

Copy link
Copy Markdown
Contributor

AI code review — automated review for reference; please use your judgment.

Reviewed the diff. Well-reasoned workaround with an honest docstring: black-holed IPs in msg-frontier's pool hang TLS handshakes with no RST, so one `open_timeout` over a sequential pool was doomed whenever a bad address sorted first. Per-IP deadlines via `asyncio.wait_for`, deduped resolved addresses, and first-handshake-wins is the right minimal shape, and the test's fake-module harness genuinely exercises attempt ordering through the patched `websockets.connect`.

Two points:

  • plugins/platforms/feishu/adapter.py:~1367 — `_socket.getaddrinfo` runs synchronously on the event loop. DNS resolution is blocking; on the dedicated WS thread's fresh loop the blast radius is limited, but with a slow/broken resolver it can stall the loop for seconds and delay ping scheduling. Use `await loop.getaddrinfo(...)` (or `run_in_executor`) so resolution yields like every other await here.

  • The IP-override path leans on `websockets.connect(host=ip, port=…)` preserving SNI and certificate validation against the original hostname. That is the documented behavior of the library's host/port overrides, but it's load-bearing for TLS security here — worth a comment naming the assumption (and the library version it was verified on), so a future websockets upgrade that changes override semantics gets caught rather than silently connecting with mismatched identity checks.

Nit: worst case is now sequential (N IPs × 4s) versus the old single budget; fine given ~12 IPs and typically ≤3 bad ones, but the docstring could note the bounded worst case as an accepted trade until RFC 8305 lands natively.

No blocking issues found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have platform/feishu Feishu / Lark adapter type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants