Skip to content

fix(cron): avoid live adapter reuse across event loops - #62956

Open
snowctl wants to merge 1 commit into
NousResearch:mainfrom
snowctl:fix/cron-live-adapter-event-loop
Open

fix(cron): avoid live adapter reuse across event loops#62956
snowctl wants to merge 1 commit into
NousResearch:mainfrom
snowctl:fix/cron-live-adapter-event-loop

Conversation

@snowctl

@snowctl snowctl commented Jul 12, 2026

Copy link
Copy Markdown

Summary

Cron delivery can reuse the live Matrix adapter from a different asyncio event loop. Because the adapter owns an aiohttp session, this can fail with:

Timeout context manager should be used inside a task

The failure was reproduced in a live Hermes gateway during manual execution of a Matrix-delivered cron job.

Root cause

There are two cron delivery paths involved:

  • cron/scheduler.py routes gateway cron delivery through DeliveryRouter and selects the live adapter by default.
  • Tool-driven Matrix delivery enters _send_to_platform(Platform.MATRIX, ...) and then _send_matrix_via_adapter().

Both paths could reuse a gateway-owned Matrix adapter from cron execution without establishing that the adapter’s HTTP session belonged to the current event loop.

The original revision of this PR changed the generic _send_via_adapter() helper, but review and live tracing showed that this was not the Matrix path. That change has been removed from this revision.

Fix

  • In cron/scheduler.py, cron execution no longer selects a live runtime_adapter; it uses the standalone/ephemeral delivery path instead.
  • In _send_matrix_via_adapter():
    • cron context skips live-adapter reuse;
    • foreign-loop Matrix sessions fall back to an ephemeral adapter;
    • same-loop non-cron calls continue to reuse the live adapter.

The reconnect/liveness problem for an already-connected Matrix gateway remains outside this PR.

Testing

The focused Matrix/send-message suite passes:

155 passed, 0 failed

Coverage includes the real _send_to_platform(Platform.MATRIX, ...) route for:

  • cron-context fallback;
  • foreign-loop fallback;
  • same-loop live-adapter reuse.

Also verified:

  • Python compilation for all changed modules;
  • git diff --check;
  • live delivery through the running Matrix gateway after reloading commit 47d1511;
  • theo-daily-curiosity recorded last_status: ok with last_delivery_error: null;
  • no new timeout or Matrix delivery error appeared in the gateway journal.

Tested on Linux with the repository's Hermes virtual environment.

Related work

This is the Matrix/generic cron counterpart of the event-loop mismatch documented in #18014, although that issue is WeChat-specific and already resolved on current main.

The Matrix cron delivery architecture is related to #3767 and #5271.

Scope

This PR fixes cross-event-loop reuse during cron delivery. It does not implement Matrix reconnect supervision after a dead sync/session; that is a separate failure mode and should be reviewed independently.

@snowctl
snowctl force-pushed the fix/cron-live-adapter-event-loop branch from 8c94557 to d7bf2ad Compare July 12, 2026 02:43
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 12, 2026
@snowctl
snowctl force-pushed the fix/cron-live-adapter-event-loop branch from d7bf2ad to c65a104 Compare July 12, 2026 02:51

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the adapter ownership concern. The current patch is not wired into the Matrix delivery path it describes.

Problems

  • Current main routes Matrix sends through _send_matrix_via_adapter in tools/send_message_tool.py:913-927. That helper directly reuses the live adapter at tools/send_message_tool.py:1693-1700, while this PR changes generic _send_via_adapter instead. The new test directly calls the generic helper, so it does not cover the actual Matrix route.
  • Gateway cron startup passes the gateway loop to the scheduler (gateway/run.py:20831-20845), and live cron delivery is scheduled onto that loop in cron/scheduler.py:1627-1713. The standard gateway path therefore does not establish the reported cross-loop call into this generic helper.

Suggested changes

  • Put any foreign-loop guard in _send_matrix_via_adapter before its live-adapter branch, and test via _send_to_platform(Platform.MATRIX, ...).
  • Add both foreign-loop fallback and same-loop reuse coverage on that real path; then reassess whether the standalone aiohttp timeout change has a reachable caller.

Automated hermes-sweeper review.

Comment thread tools/send_message_tool.py Outdated
"Skipping live %s adapter from a different event loop; using standalone sender",
platform_name,
)
adapter = None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This generic fallback is not reached for Matrix: _send_to_platform dispatches Platform.MATRIX to _send_matrix_via_adapter first, and that helper still directly awaits its live adapter at current-main tools/send_message_tool.py:1693-1700. Move the loop-ownership decision to that Matrix-specific branch and test through _send_to_platform.

@snowctl
snowctl force-pushed the fix/cron-live-adapter-event-loop branch 2 times, most recently from 8cf1932 to a755ad9 Compare July 12, 2026 03:01
@snowctl
snowctl marked this pull request as draft July 12, 2026 03:02
@snowctl
snowctl force-pushed the fix/cron-live-adapter-event-loop branch 2 times, most recently from 44bec38 to 9f87447 Compare July 12, 2026 03:07
@snowctl
snowctl force-pushed the fix/cron-live-adapter-event-loop branch from 9f87447 to 47d1511 Compare July 12, 2026 03:13
@snowctl

snowctl commented Jul 12, 2026

Copy link
Copy Markdown
Author

Sorry about that, still tweaking the permissions and rules on this agent. Launched the PR without finishing our live test 🤦
Thanks for the review anyway. You were of course right: the first revision changed the generic _send_via_adapter() seam, while the Matrix route actually enters _send_matrix_via_adapter() and gateway cron delivery selects runtime_adapter in cron/scheduler.py.

I reworked it accordingly:

  • cron scheduler delivery no longer selects the live adapter;
  • _send_matrix_via_adapter() skips cron-context reuse and guards foreign-loop Matrix sessions;
  • same-loop non-cron reuse remains unchanged;
  • tests now exercise the real _send_to_platform(Platform.MATRIX, ...) route for cron fallback, foreign-loop fallback, and same-loop reuse.

@snowctl
snowctl marked this pull request as ready for review July 12, 2026 03:22
@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 12, 2026
st0aty pushed a commit to st0aty/hermes-agent that referenced this pull request Aug 2, 2026
Five tests for the loop-mismatch behavior added in the parent commit:
same-loop direct send, foreign-loop bridge onto the adapter's own loop,
send-error surfacing as a dict cron can inspect, unknown-loop legacy
fallback, and detection-failure safety. Patterns adapted from NousResearch#62956 and
NousResearch#68359 (fake live adapter reporting a session loop; thread-backed foreign
loop).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants