fix(gateway): track all background tasks in _background_tasks set - #11630
Closed
teknium1 wants to merge 1 commit into
Closed
fix(gateway): track all background tasks in _background_tasks set#11630teknium1 wants to merge 1 commit into
teknium1 wants to merge 1 commit into
Conversation
Five long-running asyncio.create_task() call sites in gateway/run.py
were fire-and-forget — no reference stored anywhere, not added to
self._background_tasks, invisible to _stop_impl's cancel loop.
The impact:
* Python could GC the task object prematurely under memory pressure.
* Tasks survived stop()/restart because cancel() never reached them.
* Each bound-method coroutine held a reference to self (GatewayRunner),
pinning every adapter, cached AIAgent, and session object until
the process actually exited — even though stop() had 'completed'.
Fixed sites (gateway/run.py):
* start(): _session_expiry_watcher — forever loop
* start(): _platform_reconnect_watcher — forever loop
* start(): _run_process_watcher — per recovered watcher
* _handle_message(): _run_process_watcher — per mid-run watcher
* _schedule_update_notification_watch(): _watch_update_progress
All now use the established pattern:
_task = asyncio.create_task(...)
self._background_tasks.add(_task)
_task.add_done_callback(self._background_tasks.discard)
_update_notification_task keeps its self attribute too (the 'is a
watcher already running?' idempotency check in
_schedule_update_notification_watch depends on it).
Tests: 7 cases in tests/gateway/test_background_task_tracking.py
covering each tracking site, done-callback deregistration on normal
completion, idempotency of the update-notification scheduler, and
the full spawn+cancel+cleanup cycle.
No changes to _stop_impl — its existing cancel loop already handles
everything in _background_tasks correctly.
Contributor
Author
|
Skipping this one. Low ROI relative to the rest of the audit — Python GC risk is documented but rarely observed, and the forever-loop watchers already poll |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does (zoomed out)
Five long-running
asyncio.create_task()call sites ingateway/run.pywere fire-and-forget — no reference stored anywhere, not added toself._background_tasks, invisible to_stop_impl's cancel loop. They kept running (and pinningself, every cached agent, every adapter) past gateway shutdown.The leak
Each call site looked like:
The bound-method coroutine holds an implicit reference to
self(theGatewayRunner). That reference keeps the whole runner and everything it owns (adapters,_agent_cache,_session_db, etc.) alive until the process itself exits — even afterstop()'completed'. Gateway restarts looked clean but actually leaked the previous runner until Python exited.Python GC could also reap the task object early under memory pressure, at which point the coroutine silently disappears (task was only held by the task variable we threw away).
Fixed sites
All in
gateway/run.py:start()startup drain loop_run_process_watcher(per recovered watcher from crash-recovery checkpoint)start()_session_expiry_watcher(forever loop, per runner)start()_platform_reconnect_watcher(forever loop, per runner)_handle_message()drain_run_process_watcher(mid-run per-process check_interval)_schedule_update_notification_watch()_watch_update_progressEvery site now uses the pattern already established elsewhere:
_update_notification_taskkeeps itsselfattribute so the idempotency check (if existing and not existing.done(): return) still works. It's now stored in both places._stop_implis unchanged — its existingfor _task in list(self._background_tasks): _task.cancel()loop already handles everything it sees.Tests
tests/gateway/test_background_task_tracking.py— 7 cases:_background_tasks_schedule_update_notification_watchtwice while one is running doesn't spawn a duplicate or leak into_background_tasksContext
Part of the memory-leak audit series. PR 2 of 5, following #11565 (bounded agent cache). Next up:
SessionStore._entriespruning (PR 3).