fix(feishu): per-IP WS connect failover for msg-frontier DNS black holes - #89929
fix(feishu): per-IP WS connect failover for msg-frontier DNS black holes#89929zmlgit wants to merge 1 commit into
Conversation
msg-frontier.feishu.cn resolves to a ~12-IP pool in which roughly three are black holes: the TCP or TLS handshake hangs indefinitely with no RST. websockets.connect() walks the resolved addresses sequentially under a single open_timeout, so when the first address is a black hole the entire connect times out and the adapter cannot come up at all until DNS order rotates. Resolve the frontier hostname ourselves and try each unique IP with a short per-IP deadline (_WS_CONNECT_PER_IP_TIMEOUT, 4s), keeping the first TLS+WS handshake that completes. Replace with happy eyeballs (RFC 8305) if websockets grows native support; drop entirely if the DNS pool is cleaned up. Regression test drives the patched connect through the module-global loop with the first resolved IP hanging and asserts the second IP completes the handshake (attempts: black hole first, working IP second). Split out of NousResearch#64247 per review: this solves a different problem than the event-loop isolation patch and was making the root-cause fix harder to review.
Reviewed the diff. Well-reasoned workaround with an honest docstring: black-holed IPs in msg-frontier's pool hang TLS handshakes with no RST, so one `open_timeout` over a sequential pool was doomed whenever a bad address sorted first. Per-IP deadlines via `asyncio.wait_for`, deduped resolved addresses, and first-handshake-wins is the right minimal shape, and the test's fake-module harness genuinely exercises attempt ordering through the patched `websockets.connect`. Two points:
Nit: worst case is now sequential (N IPs × 4s) versus the old single budget; fine given ~12 IPs and typically ≤3 bad ones, but the docstring could note the bounded worst case as an accepted trade until RFC 8305 lands natively. No blocking issues found. |
Problem
msg-frontier.feishu.cnresolves to a ~12-IP pool in which roughly three are black holes: the TCP or TLS handshake hangs indefinitely with no RST.websockets.connect()walks the resolved addresses sequentially under a singleopen_timeout, so when the first address is a black hole the entire connect times out and the adapter cannot come up at all until DNS order rotates.Observed in production (multi-profile gateway, CN network): adapters stuck in connect loops for minutes whenever the frontier DNS round-robin returned a bad IP first.
Fix
Resolve the frontier hostname ourselves (
socket.getaddrinfo), then try each unique IP with a short per-IP deadline (_WS_CONNECT_PER_IP_TIMEOUT = 4.0s, module constant), keeping the first TLS+WS handshake that completes. Wired into the existing_connect_with_overrideswrapper ofwebsockets.connectthat_run_official_feishu_ws_clientalready installs — no SDK method patching involved.Upgrade path: replace with happy eyeballs (RFC 8305) if websockets grows native support; drop entirely if the DNS pool is cleaned up.
Regression test
test_ws_connect_ip_failover_skips_blackhole_ipsdrives the patched connect through the module-global loop with the first resolved IP hanging forever and asserts the second IP completes the handshake (attempts: black hole first, working IP second).Scope
Split out of #64247 per review (@Seekers2001) — solves a different problem than the event-loop isolation patch (#89928) and was making the root-cause fix harder to review. Independent of #89928: this PR applies cleanly on plain
mainwithout the loop-proxy work.