fix(feishu,multiplex): WS thread-safety + IP failover + secret scope fallback (re-submission of #53691)

[zmlgit]

Summary

Re-submission of #53691 (closed prematurely). The close reason claimed "upstream solved the Feishu WS thread-safety issue differently" — verification against current origin/main shows that is not the case for 3 of the 4 fixes here. Only _ThreadLocalLoopProxy could theoretically be replaced by upstream's per-thread asyncio.set_event_loop, but that does NOT actually fix the underlying bug because the Lark SDK reads ws_client_module.loop (module global) directly, not asyncio.get_event_loop(). So all 4 fixes are still needed upstream.

Verification that upstream still has the bugs

Fix in this PR	Upstream state (verified on 226e8de82)
_ThreadLocalLoopProxy	ws_client_module.loop = loop (line 1304) — still bare module-global, no thread-local proxy
per-instance _hermes_loop + monkey-patch SDK methods	Upstream has no _hermes_loop, SDK methods still read module global
_connect_ws_with_ip_failover for DNS black-holes	Upstream has no getaddrinfo / IP rotation; single connect with default timeout
_resolve_custom_provider_key via secret_scope with UnscopedSecretError fallback	Upstream _fallback_entry_api_key still calls bare os.getenv(key_env, "")

What this PR fixes

Multiplex feishu gateways (gateway.multiplex_profiles: true) currently break in 4 distinct ways, each addressed by one commit:

1. _ThreadLocalLoopProxy (commit 1)

The Lark SDK stores its asyncio event loop in a module-level global (lark_oapi.ws.client.loop) and reads it via module-global lookup inside Client.start() / _connect() / _receive_message_loop(). With 2+ feishu WS adapters in one process, the second worker thread overwrites the loop the first one registered, and the first adapter crashes with:

Task ... got Future ... attached to a different loop

This commit replaces lark_oapi.ws.client.loop with a _ThreadLocalLoopProxy instance that forwards attribute access to the calling thread's registered loop.

2. per-instance _hermes_loop (commit 2)

Even with the proxy, three SDK methods (_connect, _receive_message_loop, _start) cache the loop reference at method entry, so a later reconnect on a wrong-thread client still misroutes. This commit stashes the per-thread loop on ws_client._hermes_loop and monkey-patches those three methods to read self._hermes_loop instead of the module global. Belt-and-suspenders with commit 1.

3. WS IP failover (commit 3)

msg-frontier.feishu.cn DNS pool returns ~12 IPs, ~3 of which are TCP/TLS black-holes (no RST, just indefinite hang). websockets.connect() tries resolved IPs sequentially under one open_timeout — if the first IP is a black-hole the entire connect times out. This commit resolves the hostname via getaddrinfo, tries each unique IP with a 4s per-IP timeout, returns the first successful TLS+WS handshake.

Upgrade path: replace with happy-eyeballs (RFC 8305) if websockets adds native support, or remove entirely if Feishu cleans their DNS pool.

4. Multiplex secret_scope fallback (commit 4)

_fallback_entry_api_key reads custom-provider API keys via os.getenv(key_env, ""). In multiplex mode without an active profile scope, agent.secret_scope.get_secret() raises UnscopedSecretError. At startup / prewarm no scope exists yet, so the call fails. This commit adds a _resolve_custom_provider_key helper that tries get_secret() first and falls back to os.getenv() on UnscopedSecretError, then routes both call sites through it.

Tests

Local: 217/218 pass on this branch. The single failure (test_websocket_sdk_accepts_channel_ua_tag) is pre-existing on pristine origin/main and unrelated to these changes.

Backward compatibility

• Single-profile feishu configurations are byte-identical: the proxy install is a no-op when only one thread ever registers a loop; the IP failover falls through to a single connect when only one IP is returned; the secret_scope fallback returns the same value as os.getenv when no scope is active.
• All four commits are additive — no public API change.

Why this PR was re-opened

#53691 was closed with the reasoning "upstream solved the Feishu WS thread-safety issue differently". That is inaccurate: upstream uses asyncio.set_event_loop(loop) per-thread, but the Lark SDK reads ws_client_module.loop directly (not via asyncio.get_event_loop()), so upstream's approach does not address the module-global clobbering. The other three fixes were not addressed upstream at all.

Re-submitting as a fresh PR rather than re-opening #53691 to keep the diff context current against latest main (the original PR was rebased onto a much older base).

[Seekers2001]

The latest public reproduction added to #31367 supports the event-loop isolation part of this PR.

For mergeability, could that change be split into a focused current-plugin-path patch with a multiplex regression test? The IP failover and secret-scope changes solve different problems and make the root-cause fix harder to review. The silent-worker recovery should remain in the gateway-owned supervisor path (#73202 / #53508), rather than adding another retry policy here.

[zmlgit]

Split per review (@Seekers2001's ask) — closing in favor of focused follow-ups:

• fix(feishu): thread-local WS loop proxy for multiplex event-loop isolation #89928 — event-loop isolation: thread-local WS loop proxy + per-instance loop patching, current plugin path, with the multiplex regression test (supported by the Feishu WebSocket disconnect causes full gateway restart every ~30min #31367 repro)
• fix(feishu): per-IP WS connect failover for msg-frontier DNS black holes #89929 — per-IP WS connect failover for msg-frontier DNS black holes (independent of fix(feishu): thread-local WS loop proxy for multiplex event-loop isolation #89928, applies on plain main)

The secret-scope fallback part of this PR turned out to be already implemented on current main (_scoped_key_env in agent/auxiliary_client.py, #76573 / Slack pattern #59739) — verified both call sites (resolve_provider_client named-custom branch and _fallback_entry_api_key → resolve_entry_api_key) now route through it, so no replacement PR is needed for that piece.

The adapter-side silent-worker retry is dropped per review — the gateway-owned supervisor path (#73202 / #53508) stays the reconnect authority.