fix(yuanbao): anchor the reconnect task and drain background tasks on disconnect - #83418
Conversation
`ConnectionManager.schedule_reconnect` launched the backoff loop with a bare
`asyncio.create_task(...)` and dropped the handle. The event loop holds only a
weak reference to a task nobody else references, so a still-pending reconnect
can be garbage-collected mid-flight. When that happens the Yuanbao adapter
never reconnects and the bot is silently offline until the gateway is
restarted — there is no retry left to fire, because the retry *was* the task
that vanished.
`YuanbaoAdapter._track_task` exists for exactly this case ("Register a
fire-and-forget task so it won't be GC'd prematurely") and is already used for
the inbound pipeline 40 lines above, at the `_flush_inbound_buffer` dispatch.
Route the reconnect through it and give the task a `name=`, matching every
other named task in this file.
The `_running` / `_reconnecting` guards are unchanged: this only fixes who
holds the reference, not when a reconnect is scheduled.
Asserts that a scheduled reconnect is reachable from the adapter's `_background_tasks` set, which is what stops the event loop from collecting it while it is still pending. The test fails against a bare `asyncio.create_task` (the set stays empty) and passes once the task is routed through `_track_task`. Also pins the two guards the fix runs through, so a later refactor cannot quietly start scheduling reconnects on a stopped adapter or on top of an in-progress one.
`disconnect()` is documented as "Cancel background tasks and close the WebSocket connection", but it only ever cancelled `_inbound_tasks`. Everything registered through `_track_task` lands in `_background_tasks` — the inbound pipeline, the recall-redaction job, and the reconnect — and that set has no teardown path at all: its only removal path is the per-task done callback. So every one of those jobs was abandoned mid-flight when the adapter shut down, with its `except`/`finally` cleanup never running. It also closes the ordering hole the anchoring fix would otherwise open. Now that a reconnect is held in `_background_tasks`, an in-flight one could outlive `disconnect()` and go on trying to revive an adapter that was deliberately stopped. `disconnect()` clears `_running` early, so the `schedule_reconnect` guard covers *new* schedules — it does nothing for a reconnect already past that check. Cancel-then-gather mirrors `ConnectionManager.close()`, which already cancels and awaits `_heartbeat_task` / `_recv_task`. Details: - The set is snapshotted first, because cancelling mutates it through the same done callback. - `return_exceptions=True` keeps one task raising from its cleanup path from aborting teardown for the rest. - `asyncio.current_task()` is excluded so a shutdown driven from inside a tracked task cannot await itself. `_inbound_tasks` handling is left byte-identical.
Four cases against the teardown path: - an in-flight `_track_task` job is cancelled *and* awaited by `disconnect()`, proven by asserting its `except asyncio.CancelledError` arm actually ran before `disconnect()` returned — a cancel without the await would leave the task not yet unwound; - a task whose cleanup raises does not abort teardown for the others; - a shutdown driven from inside a tracked task completes instead of awaiting itself; - the pre-existing `_inbound_tasks` cancellation is unchanged. The first two fail against the old `disconnect()`, which left `_background_tasks` untouched.
There was a problem hiding this comment.
Pull request overview
This PR fixes background task lifecycle handling in the Yuanbao gateway adapter by ensuring reconnect tasks are strongly referenced (preventing GC mid-flight) and by draining tracked fire-and-forget tasks during adapter disconnect, with regression tests added to lock in the behavior.
Changes:
- Anchor
ConnectionManager.schedule_reconnect()’s backoff loop task viaYuanbaoAdapter._track_task(...)(namedyuanbao-reconnect) to prevent premature garbage collection. - Extend
YuanbaoAdapter.disconnect()to cancel and await in-flight_background_tasks(tracked via_track_task) so teardown doesn’t leave tasks running post-disconnect. - Add a new focused test suite validating reconnect anchoring and disconnect task draining semantics.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| gateway/platforms/yuanbao.py | Tracks reconnect backoff task via _track_task and drains _background_tasks during disconnect(). |
| tests/test_yuanbao_task_lifecycle.py | Adds regression coverage for task anchoring and disconnect draining behavior. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| current = asyncio.current_task() | ||
| background = [ | ||
| task for task in self._background_tasks | ||
| if task is not current and not task.done() | ||
| ] |
There was a problem hiding this comment.
The snapshot is the list(...) call:
background = [
task for task in list(self._background_tasks)
if task is not current and not task.done()
]list() materialises a copy of the set before the comprehension starts walking it, so the sequence being iterated is that copy, not _background_tasks. The _track_task done-callback can only discard() from the underlying set, which is no longer what the loop is reading.
The same holds for both later uses — the for task in background cancel loop and asyncio.gather(*background, return_exceptions=True) — since both take the list. That gather is precisely where the callbacks fire (it suspends while the cancellations land), and it cannot raise RuntimeError: Set changed size during iteration for the same reason. The trailing self._background_tasks.clear() is what reconciles the set afterwards.
tests/test_yuanbao_task_lifecycle.py::test_disconnect_drains_in_flight_background_tasks and ::test_disconnect_survives_a_background_task_that_raises both exercise this path with callbacks firing mid-drain and assert _background_tasks == set() on the way out.
The drain built its work list straight off `self._background_tasks` while the comment above it claimed to snapshot first. The comprehension is in fact a snapshot and cannot be interrupted — asyncio schedules done callbacks through `loop.call_soon`, so none of them can run partway through synchronous code — but the code did not say so, and the `_inbound_tasks` loop directly above already spells the snapshot out as `list(...)`. Make it explicit and match that neighbour, and correct the comment to state the real reason the snapshot matters: the done callbacks mutate `_background_tasks` while the `gather` is suspended. No behaviour change.
|
@copilot Addressed in
No behaviour change, and the drain is still covered by |
fix(yuanbao): anchor the reconnect task and drain background tasks on disconnect
|
What does this PR do?
Fixes two defects in the Yuanbao adapter's fire-and-forget task lifecycle. Both leave a task unowned at a lifecycle boundary, and the second one is created by fixing the first — which is why they belong in one PR.
Symptom: the bot goes silently offline and never comes back.
ConnectionManager.schedule_reconnectlaunched the backoff loop with a bareasyncio.create_task(...)and dropped the handle:The event loop keeps only a weak reference to a task nobody else holds, so the still-pending reconnect can be garbage-collected mid-flight. When that happens there is no retry left to fire — the retry was the task that vanished — and the adapter stays down until the gateway is restarted.
_reconnect_with_backoffsleeps up to 60s between attempts, so the window in which the task is pending and unreferenced is a long one.This is the repo's own stated invariant, in two places:
gateway/platforms/yuanbao.py—_track_task's docstring: "Register a fire-and-forget task so it won't be GC'd prematurely."gateway/run.py— "a bareasyncio.create_task()keeps only a weak reference, so the event loop may garbage-collect a still-pending task mid-flight."And the correct idiom is already used in this same file, 40 lines above at the
_flush_inbound_bufferdispatch:The second half — teardown.
YuanbaoAdapter.disconnect()is documented as "Cancel background tasks and close the WebSocket connection", but it only ever cancelled_inbound_tasks. Everything registered through_track_tasklands in_background_tasks, and that set had no teardown path at all — its only removal path is the per-task done callback. So the inbound pipeline and the recall-redaction job were abandoned mid-flight at shutdown, with theirexcept/finallycleanup never running.That also closes the ordering hole the first fix would otherwise open: once the reconnect is anchored into
_background_tasks, an in-flight one could outlivedisconnect()and go on trying to revive an adapter that was deliberately stopped.disconnect()clears_runningearly, so theschedule_reconnectguard covers new schedules — it does nothing for a reconnect already past that check. Fixing the GC bug alone would trade it for a shutdown-ordering bug.Cancel-then-gather mirrors
ConnectionManager.close(), which already cancels and awaits_heartbeat_task/_recv_task. Three details worth calling out:return_exceptions=Truestops one task raising from its cleanup path from aborting teardown for the rest;asyncio.current_task()is excluded, so a shutdown driven from inside a tracked task cannot await itself._inbound_taskshandling is left byte-identical.Sibling-site sweep
Every task-creation site in
gateway/platforms/yuanbao.pywas enumerated (create_task/_track_task/_background_tasks), andschedule_reconnectis the only one missing a strong reference. The rest are already anchored and are deliberately untouched::1160,:1825,:3602_background_tasks/_track_task:3002,:3009_inbound_tasks:3212,:3215,:3701,:3708_heartbeat_task/_recv_task:4225,:4305_reply_heartbeat_tasks[chat_id]/_tasks[chat_id]:3644schedule_reconnectThe same construct appears in other platform adapters (
weixin.py,qqbot/adapter.py,bluebubbles.py) and ingateway/run.py. Those are deliberately out of scope here — each already has an open PR against it (#65966/#26049for weixin,#38848/#74775for qqbot,#18395for bluebubbles,#45372for run.py), and this PR stays inside one file and one adapter's lifecycle.Duplicate check
Searched open PRs for
schedule_reconnect,yuanbao reconnect,yuanbao disconnect,yuanbao background tasks,yuanbao GC task, and_track_task: no open PR touches Yuanbao's reconnect or disconnect path. The_track_task/_background_tasksmatches are all other files —weixin.py(#65966, #26049),qqbot/adapter.py(#41298),gateway/run.py(#17966, #6790, #45372),gateway/platforms/base.py(#29216, #43903) — verified by reading each one's changed-file list. The nearest open PRs that do touchgateway/platforms/yuanbao.pysit in unrelated regions (#74860 around_sender_may_designate_home, #65112, #29077, #62031, #78166, #55782), none withinschedule_reconnectordisconnect().Related Issue
No filed issue — found by auditing task lifecycle at shutdown/reconnect boundaries in
gateway/platforms/yuanbao.py.Type of Change
Changes Made
gateway/platforms/yuanbao.py—ConnectionManager.schedule_reconnectroutes its task throughself._adapter._track_task(...)and names ityuanbao-reconnect. The_running/_reconnectingguards are unchanged; only ownership of the reference changes.gateway/platforms/yuanbao.py—YuanbaoAdapter.disconnectcancels and awaits the tasks in_background_tasksalongside the existing_inbound_taskscancellation, then clears the set.tests/test_yuanbao_task_lifecycle.py— new; 7 regression tests.How to Test
pytest tests/test_yuanbao_task_lifecycle.py -q→ 7 passed.schedule_reconnecthunk to its previous form and re-run:test_schedule_reconnect_anchors_task_against_gcfails withassert 0 == 1— the task is not in_background_tasks. Nothing else fails.disconnect()hunk and re-run:test_disconnect_drains_in_flight_background_tasksandtest_disconnect_survives_a_background_task_that_raisesfail, and pytest additionally logsunhandled exception during asyncio.run() shutdownfor the abandoned task. Nothing else fails.pytest tests/test_yuanbao_task_lifecycle.py tests/test_yuanbao_shutdown.py tests/test_yuanbao_reconnect_set_active.py tests/test_yuanbao_pipeline.py tests/test_yuanbao_integration.py tests/test_yuanbao_proto.py tests/test_yuanbao_markdown.py tests/gateway/test_yuanbao_forwarded_heartbeat.py tests/gateway/test_yuanbao_media_ssrf.py tests/gateway/platforms/test_yuanbao_recall_db_only.py tests/gateway/platforms/test_yuanbao_state_cleanup.py -q→ 147 passed.Each of the four commits was also checked out on its own and its touched suite run, so the history bisects cleanly.
Checklist
Code
fix(scope):,feat(scope):, etc.)pytest tests/ -qand all tests pass — I ran the full Yuanbao surface plus the reconnect/shutdown suites (147 tests, listed above), not the whole repo suiteDocumentation & Housekeeping
docs/, docstrings) — or N/A (disconnect()'s existing docstring already described the behaviour this PR makes true)cli-config.yaml.exampleif I added/changed config keys — or N/ACONTRIBUTING.mdorAGENTS.mdif I changed architecture or workflows — or N/A