Skip to content

fix(weixin): resolve 'Future attached to a different loop' in aiohttp 3.13+ - #20956

Open
rj-chenlinfeng wants to merge 4 commits into
NousResearch:mainfrom
rj-chenlinfeng:fix/weixin-aiohttp-different-loop
Open

fix(weixin): resolve 'Future attached to a different loop' in aiohttp 3.13+#20956
rj-chenlinfeng wants to merge 4 commits into
NousResearch:mainfrom
rj-chenlinfeng:fix/weixin-aiohttp-different-loop

Conversation

@rj-chenlinfeng

Copy link
Copy Markdown

Problem

All WeChat (iLink) message delivery fails with:

Task got Future attached to a different loop

This happens when send() is invoked via asyncio.run_coroutine_threadsafe() from worker threads — which covers all cron job deliveries and most agent responses through the gateway.

Root Cause

aiohttp 3.13+ uses aiohappyeyeballs for async DNS resolution in TCPConnector._resolve_host_with_throttle(). Internally it creates Futures that can get bound to the wrong event loop when the coroutine is submitted cross-thread via run_coroutine_threadsafe().

This affects:

  • Cron job result delivery
  • Agent responses in gateway mode
  • Any cross-thread send() call

Fix

Two changes in gateway/platforms/weixin.py:

  1. _make_ssl_connector(): Pass resolver=aiohttp.resolver.ThreadedResolver() to TCPConnector. This routes DNS resolution through a thread pool instead of the buggy async aiohappyeyeballs path.

  2. send_weixin_direct(): Add loop compatibility check — compare session._loop with asyncio.get_running_loop() before using the gateway-loop session, and fall back gracefully when they mismatch.

Testing

  • Cross-thread aiohttp test (simulating run_coroutine_threadsafe): ✅ passes
  • Gateway restart + WeChat message delivery: ✅ confirmed working
  • Cron job delivery: ✅ no more "Future attached to a different loop" errors

chenlinfeng added 3 commits April 30, 2026 18:42
aiohttp ClientTimeout uses BaseTimerContext which calls
loop.call_later() internally. When invoked via
asyncio.run_coroutine_threadsafe() from cron jobs, this
triggers "Timeout context manager should be used inside a task"
errors, causing message delivery failures.

Replace all direct ClientTimeout usage with asyncio.wait_for():
- _upload_ciphertext: CDN upload (120s timeout)
- _download_bytes: CDN download (configurable timeout)
- _download_remote_media: remote media fetch (30s timeout)

Also set total=None on _send_session to disable aiohttp built-in
timeout, and change trust_env=True to False to bypass proxy for
WeChat CDN connections.
… 3.13+

aiohttp 3.13+ uses aiohappyeyeballs for async DNS resolution in
TCPConnector._resolve_host_with_throttle().  When send() is invoked
via asyncio.run_coroutine_threadsafe() from worker threads (cron
delivery, agent responses), internal Futures get attached to the wrong
event loop, causing all WeChat message delivery to fail with:

  Task got Future attached to a different loop

Fix:
- Use ThreadedResolver in _make_ssl_connector() so DNS resolution
  happens in a thread pool instead of the buggy async path.
- Add loop compatibility check in send_weixin_direct() to gracefully
  fall back when the session's loop doesn't match the running loop.

Tested: cross-thread aiohttp request succeeds; WeChat messages deliver
normally after restart.

@liuhao1024 liuhao1024 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good fix for the aiohttp 3.13+ event-loop / task-context issues. The asyncio.wait_for approach is well-reasoned and the ThreadedResolver change makes sense.

One concern: the PR also silently changes trust_env=True → False on all four ClientSession instances (lines 1028, 1241, 1248, and the _download_remote_media session at the bottom), but the PR description only mentions the event-loop/timeout fix.

Setting trust_env=False means environment proxy variables (HTTP_PROXY, HTTPS_PROXY, NO_PROXY) will be silently ignored. Any Weixin deployment behind a corporate proxy that relies on env-var-based proxy configuration would lose connectivity with no error — just silent timeouts or connection refused.

This change is independent of the event-loop fix (the ThreadedResolver + explicit SSL context work regardless of trust_env). If the intent is to avoid proxy interference with Weixin's CDN endpoints, consider:

  • Reverting trust_env to True (the real fix is in ThreadedResolver + asyncio.wait_for), or
  • Adding an explicit comment explaining why trust_env=False is required for this platform, plus a follow-up note in the PR description so operators are aware of the behavioral change.

@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery comp/cron Cron scheduler and job management platform/wecom WeCom / WeChat Work adapter P2 Medium — degraded but workaround exists labels May 7, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related: #13520, #12810, #19141 (merged) — prior fixes for cross-event-loop session reuse in Weixin adapter. This PR goes further by addressing aiohttp 3.13+ aiohappyeyeballs Future binding and replacing ClientTimeout with asyncio.wait_for() to avoid "Timeout context manager should be used inside a task" errors.

…on rate-limit fallback

Two bugs caused iLink errcode=-2 to be misidentified as genuine rate
limiting when it was actually a stale/expired session, leading to
infinite retries that always fail.

Bug 1: _is_stale_session_ret() only recognized errmsg "unknown error"
as a session-expiry signal, but iLink also returns empty strings,
"session expired", "token expired", etc. with errcode=-2.  These
were all misclassified as rate-limit errors.

  Before: (errmsg or "").lower() == "unknown error"
  After:  also match empty errmsg, "unknown error", and any string
          containing "expire" (covers locale-dependent variants)

Bug 2: The rate-limit retry branch in _send_text_chunk() never cleared
context_token, so if errcode=-2 was actually a stale session that
slipped through _is_stale_session_ret, every retry would carry the
same expired token and always get -2 again — a dead loop until the
retry budget was exhausted.

  Fix: strip context_token on the first rate-limit hit (with
  retried_without_token guard), same as the explicit session-expired
  branch.  This provides a second safety net: even if the errmsg
  doesn't match any known pattern, the degraded tokenless retry can
  still succeed.

Together these changes ensure that stale iLink sessions are recovered
reliably regardless of the exact errmsg wording returned by the server.
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the detailed Weixin investigation. Current main already contains the headline loop-affinity and timeout protections, but the remaining changes need a focused re-scope and regression evidence.

Problems

  • gateway/platforms/weixin.py:2307-2336 already guards live-session reuse by event-loop identity and falls back to a fresh session; this landed in a22465e07 (merged PR fix(weixin): send_weixin_direct cross-loop session check #19141). Current _api_post/_api_get also use asyncio.wait_for() at gateway/platforms/weixin.py:381-414, with coverage in tests/gateway/test_weixin.py:1106-1207.
  • The final stale-session change conflicts with the current rate-limit contract: tests/gateway/test_weixin.py:926-928 explicitly requires an empty errmsg for -2 not to be considered stale, so it reaches the breaker/backoff path at gateway/platforms/weixin.py:1789-1813. This PR changes that behavior without tests.
  • The trust_env=True to False changes alter environment-proxy behavior independently of the stated loop fix; the submitted review correctly called this out.

Suggested changes

  • Preserve only a demonstrated remaining aiohttp 3.13 failure after the existing loop guard, with a regression test.
  • Validate and test the -2 classification before changing rate-limit behavior.
  • Retain trust_env=True unless the proxy behavior change is intentional and documented.

Automated hermes-sweeper review.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/wecom WeCom / WeChat Work adapter sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants