Skip to content

fix(gateway): skip cross-loop live adapter in _send_via_adapter - #77378

Open
andrexibiza wants to merge 1 commit into
NousResearch:mainfrom
andrexibiza:fix/mattermost-cron-delivery-timeout
Open

fix(gateway): skip cross-loop live adapter in _send_via_adapter#77378
andrexibiza wants to merge 1 commit into
NousResearch:mainfrom
andrexibiza:fix/mattermost-cron-delivery-timeout

Conversation

@andrexibiza

@andrexibiza andrexibiza commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Related #37005 #39836 #39877 #76799 #78791

What / Why

Cron delivery of plugin-platform messages runs _send_to_platform via asyncio.run() on a scheduler worker thread (cron/scheduler.py, _sequential_pool/_parallel_pool). When the gateway runs in the same process, _send_via_adapter (tools/send_message_tool.py:688) prefers the live in-process adapter — whose aiohttp.ClientSession was created on the gateway's main event loop. Reusing that session from the worker-thread loop makes aiohttp's TimerContext call asyncio.current_task(loop=session._loop), see None, and raise:

RuntimeError: Timeout context manager should be used inside a task

which surfaces as delivery error: Plugin platform send failed: Timeout context manager should be used inside a task on cron jobs whose execution itself succeeds (last_status: ok). The same path is hit by the send_message tool when invoked from a running loop: _run_async (model_tools.py:103) spins up a worker thread with a fresh asyncio.new_event_loop() and runs the coroutine there.

The bug is loop-bound session reuse, not the timeout mechanism itself — a fresh ClientSession inside asyncio.run() works fine (the author of #39877 verified this before closing it), which is why the failure is specific to the in-process live-adapter shortcut and affects every plugin platform with a standalone_sender_fn (Mattermost, etc.), not just one adapter.

Fix: gate the live-adapter shortcut on _adapter_session_is_usable() — the adapter's session must be open and bound to the current event loop. When the session belongs to a different loop (or is closed), delivery falls through to the plugin's standalone_sender_fn, which creates a fresh session on the current loop. This mirrors the Weixin fix in commit a22465e07a (send_weixin_direct cross-loop session check), applied to the generic path all plugin platforms share.

How to test

  1. Run the gateway with a Mattermost (or any plugin) platform connected and cron running in-process.
  2. Create a cron job with deliver: "mattermost" targeting a channel.
  3. Before the fix: the job executes with last_status: ok, but last_delivery_error is Timeout context manager should be used inside a task and the message never arrives.
  4. After the fix: delivery falls back to the plugin's standalone sender and the message arrives.

Tests

  • New TestSendViaAdapterCrossLoopSession (3 tests, tests/tools/test_send_message_tool.py):
    • foreign-loop session → falls back to standalone sender, live adapter not called;
    • closed session → falls back to standalone sender;
    • same-loop session → live adapter still used (control).
  • RED on unpatched main (6858e0d): 2 failed, 1 passed (control). GREEN with fix: 3 passed.
  • Full file: 50 passed on the branch.
  • git diff --check clean; scripts/check-windows-footguns.py clean (2 files scanned).

Platforms tested

Windows native (git-bash). The change is event-loop identity logic — platform-agnostic; the identical code path executes on macOS/Linux.

Related issues

Fixes #39836, #76799, #37005 — same root cause (cross-loop aiohttp session reuse in cron / send_message-tool delivery).

Why this matters to users

Before: a Hermes user running cron jobs that deliver to Mattermost or another plugin platform inside the same process as the gateway saw every job succeed but every delivery fail with "Timeout context manager should be used inside a task" — the message never arrived and there was no workaround. The same failure hit the send_message tool when the agent used it from inside a running gateway. After: delivery detects the loop mismatch and uses the platform's standalone sender, which builds a fresh session on the current loop, so messages arrive. Users no longer need to keep the gateway and cron in separate processes to get plugin-platform deliveries.

Part of #37005
Part of #39836
Part of #76799

Part of #55009

Cron delivery runs _send_to_platform via asyncio.run() in a worker
thread (ThreadPoolExecutor), while the gateway's live platform adapter
holds an aiohttp session created on the gateway's main event loop.
Reusing that session from the worker-thread loop makes aiohttp's
TimerContext call asyncio.current_task(loop=session._loop), see None,
and raise 'Timeout context manager should be used inside a task' —
so every plugin-platform cron delivery (Mattermost, etc.) fails at the
final delivery step even though the job itself succeeds.

_send_via_adapter preferred the live adapter whenever the gateway
runner was present, which is always true for in-process cron. Now the
live-adapter shortcut is gated on _adapter_session_is_usable(): the
adapter's session must be open and bound to the current event loop.
When the session belongs to a different loop (or is closed), delivery
falls through to the plugin's standalone_sender_fn, which creates a
fresh session on the current loop. This mirrors the Weixin fix in
commit a22465e, applied here to the generic path used by all
plugin platforms.

Fixes the delivery half of NousResearch#39836, NousResearch#76799, NousResearch#37005.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery comp/tools Tool registry, model_tools, toolsets sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages duplicate This issue or pull request already exists labels Aug 3, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Duplicate of #62956 — both gate the in-process live-adapter shortcut on a current-event-loop session check and fall back to the standalone sender, fixing the same cross-loop aiohttp delivery failure. #62956 is the earlier open canonical.

@andrexibiza

Copy link
Copy Markdown
Contributor Author

Not a duplicate of #62956: that PR is mergeable_state: dirty against current main and fixes a different path — it gates the live adapter in cron/scheduler.py via HERMES_CRON_SESSION and adds a Matrix-specific loop check in _send_matrix_via_adapter. This PR fixes the generic _send_via_adapter path (used by the send_message tool) by checking whether the adapter's aiohttp session is bound to the current event loop and falling back to standalone_sender_fn when it is not — a path #62956 does not cover. Keeping this open.

@GottZ

GottZ commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

This was generated by AI during triage.

Summary

Seven PRs address or reference this complex: six target the plugin-delivery timeout through adapter timeout rewrites, task creation, cross-loop request routing, or generic live-adapter avoidance, while #55011 separately bounds Matrix standalone response reads. The diffs show that #77378 most directly avoids the reported generic cross-loop session reuse, whereas #55011 is the best fix for the unrelated Matrix issue.

Related pull requests

Duplicates

#37599 and closed #39877 duplicate the Mattermost timeout-replacement approach. Closed #38902 substantially duplicates closed #38563 and additionally incorporates #38525; #77378 is not a diff duplicate of those adapter rewrites, subject to resolving its asserted overlap with #62956.

Suggested consolidation

Keep #77378 open with a salvage path for its generic session-loop guard, standalone fallback, and three focused regression tests; first resolve the contributor's #62956 duplicate objection using the concrete path distinction, then close #37599 as duplicate of the already-closed #39877 approach while leaving #38525, #38563, #38902, and #39877 closed under their recorded outcomes. Treat #55011 separately with author action to rebase onto main or split out its Matrix response-cap change; despite #38563's recorded best_fix verdict, its Mattermost-specific cross-loop routing is better retained as a reference than revived alongside #77378's narrower generic dispatch fix.

Complex graph

flowchart LR
    classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
    classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
    classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
    classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
    classDef best stroke-width:3px,stroke:#b45309
    classDef target stroke-width:3px,stroke:#4338ca
    I37005(["issue #37005 (open)"])
    I39836(["issue #39836 (open)"])
    I76799(["issue #76799 (open)"])
    P77378["PR #77378 (open)"]
    P77378 -->|best fix| I37005
    P77378 -->|best fix| I39836
    P77378 -->|best fix| I76799
    class I37005 open
    class I39836 open
    class I76799 open
    class P77378 open
    class P77378 best
    class P77378 best
    class P77378 best
    class P77378 target
    click I37005 "https://github.com/NousResearch/hermes-agent/issues/37005"
    click I39836 "https://github.com/NousResearch/hermes-agent/issues/39836"
    click I76799 "https://github.com/NousResearch/hermes-agent/issues/76799"
    click P77378 "https://github.com/NousResearch/hermes-agent/pull/77378"
Loading

Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).

Cross-PR triage: Reviewed 7 pull requests and 4 issues in this complex. Each diff was read against this issue; Assessment working set: 103 kB of PR diffs, 27 kB of issue/PR text, 13 kB of discussion (15 comments), 25 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

@andrexibiza

Copy link
Copy Markdown
Contributor Author

Thanks for the triage. No code changes were needed in response to this verdict — the current head 97381f5 already carries every salvage-path item listed: the generic _send_via_adapter open/current-loop session gate, the standalone-sender fallback, and the three regression tests (foreign-loop, closed-session, same-loop) in tests/tools/test_send_message_tool.py.

The #62956 duplicate objection was already resolved in-thread above: #62956 gates the live adapter in cron/scheduler.py via HERMES_CRON_SESSION and adds a Matrix-specific check in _send_matrix_via_adapter, whereas this PR fixes the shared generic dispatch point in tools/send_message_tool.py — a material path distinction the triage itself confirms.

Validation on the current head: all required checks pass (8 Python test slices, ruff enforcement + Windows footguns, e2e, both Docker builds); mergeable_state is clean. Ready for maintainer review.

@alt-glitch alt-glitch removed the duplicate This issue or pull request already exists label Aug 3, 2026
@alt-glitch alt-glitch added P3 Low — cosmetic, nice to have comp/cron Cron scheduler and job management comp/plugins Plugin system and bundled plugins P2 Medium — degraded but workaround exists and removed comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have labels Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/cron Cron scheduler and job management comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Mattermost cron job delivery fails with "Timeout context manager should be used inside a task"

3 participants