Skip to content

fix(gateway): store asyncio.create_task refs to prevent GC of background watchers - #65417

Closed
x7peeps wants to merge 1 commit into
NousResearch:mainfrom
x7peeps:fix/gateway-store-task-refs
Closed

fix(gateway): store asyncio.create_task refs to prevent GC of background watchers#65417
x7peeps wants to merge 1 commit into
NousResearch:mainfrom
x7peeps:fix/gateway-store-task-refs

Conversation

@x7peeps

@x7peeps x7peeps commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Register 11 fire-and-forget asyncio.create_task() calls in gateway/run.py with the existing _background_tasks set to prevent garbage collection of critical background watchers.

Problem

asyncio.create_task() only keeps a weak reference to the created task. If no strong reference is held, the task can be garbage-collected before it completes, causing the coroutine to silently disappear.

In gateway/run.py, 11 background watchers were created with asyncio.create_task() but never stored anywhere:

  • _run_process_watcher (crash-recovery + mid-run batch, 2 sites)
  • _session_expiry_watcher
  • _kanban_notifier_watcher
  • _kanban_dispatcher_watcher
  • _platform_reconnect_watcher
  • _handoff_watcher
  • _async_delegation_watcher
  • _scale_to_zero_watcher
  • _drain_control_watcher
  • runner.stop() in signal handler

Any of these being GC'd would cause the corresponding gateway functionality to silently stop working with no error logged.

Fix

Use the existing _background_tasks set (already present at line 3177) with the established pattern:

task = asyncio.create_task(...)self._background_tasks.add(task)task.add_done_callback(self._background_tasks.discard)```

This is the same pattern already used at line 6793-6794 for `_run_startup_resume_event`.

## Testing
- Syntax check passed (py_compile)
- Behavior verified

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the unretained watcher-task pattern. The ownership needs a separate lifecycle lane rather than reusing _background_tasks.

Problems

  • gateway/run.py:21229 adds the signal-handler task that awaits runner.stop() to _background_tasks. The shutdown loop cancels tracked tasks at gateway/run.py:8427-8437; this wrapper is neither _stop_task nor _restart_task. Commit 1ce5d6d97 documents that cancelling a task awaiting _stop_task propagates CancelledError into _stop_impl and skips cleanup.
  • Adding permanent watcher loops to _background_tasks also blocks scale-to-zero: gateway/run.py:4208 treats any unfinished tracked task as active work, while _session_expiry_watcher runs for self._running (gateway/run.py:7766-7779).

Suggested changes

  • Keep the signal-handler stop wrapper out of _background_tasks.
  • Use a distinct strong-reference collection for service watchers, with explicit shutdown handling, and add regressions for signal shutdown and scale-to-zero idle detection.

Automated hermes-sweeper review.

Comment thread gateway/run.py
# Start background session expiry watcher to finalize expired sessions
asyncio.create_task(self._session_expiry_watcher())
task = asyncio.create_task(self._session_expiry_watcher())
self._background_tasks.add(task)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

_background_tasks is also the scale-to-zero active-work signal (gateway/run.py:4208). This watcher runs for the lifetime of self._running, so tracking it here makes an otherwise idle scale-to-zero gateway permanently non-idle. Please use a separate strong-reference collection for service loops, or exclude those loops from that predicate.

Comment thread gateway/run.py
logger.debug("spawn_async_diagnostic failed: %s", _e)
asyncio.create_task(runner.stop())
task = asyncio.create_task(runner.stop())
runner._background_tasks.add(task)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not add the outer runner.stop() task to this cancellation set. _stop_impl cancels every member except _stop_task and _restart_task (gateway/run.py:8427-8437), so this task is cancelled while awaiting _stop_task; commit 1ce5d6d97 documents that this propagation aborts shutdown cleanup.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 16, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #45372 and #6790. This direct-registration implementation overlaps their watcher-task retention work, but also covers additional active watcher and shutdown call sites. Please consolidate the competing implementations rather than treating either as a duplicate.

@teknium1 teknium1 added sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit labels Jul 16, 2026
@x7peeps x7peeps closed this Jul 16, 2026
@x7peeps

x7peeps commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Closed as duplicate per maintainer feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-broad Sweeper blast radius: broad — a core path most sessions hit sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants