fix(gateway): run adapter connect detach-on-timeout so a swallowed cancel can't wedge reconnect - #70345
fix(gateway): run adapter connect detach-on-timeout so a swallowed cancel can't wedge reconnect#70345VaitaR wants to merge 1 commit into
Conversation
…ncel can't wedge reconnect _connect_adapter_with_timeout used a plain asyncio.wait_for(adapter.connect()), which cancels an overdue child and then awaits it. A connect() whose teardown swallows CancelledError would keep the background reconnect watcher blocked indefinitely, leaving the gateway alive but never rebuilding a deaf adapter. Run connect as a detached task (the pattern already used by _await_adapter_cleanup_with_timeout): on deadline the task is cancelled and drained via consume_detached_task_result and TimeoutError is raised, so the watcher's existing failure/backoff path is unchanged. A timed-out connect is never installed in self.adapters, so a late-completing swallowed-cancel connect stays unowned and is disposed rather than leaving a half-registered adapter.
cdcaae3 to
9cc85ae
Compare
Related to #70344: the live diff hardens a swallowed-cancellation connect timeout, while the reported incident froze before the reconnect watcher started. This is not yet a confirmed fix for that incident. |
|
suggesting changes The detach-on-timeout path releases the watcher, but it also drops lifecycle On a clean replay onto current Current Security evidence:
Please retain ownership of the timed-out connect long enough to arrange a final Signed: GPT-5.6-sol-xhigh in Codex |
|
Merged via #71176 — your commit was cherry-picked with authorship preserved (rebase-merge). Thanks for the clean single-concern fix! |
What does this PR do?
Hardens
GatewayRunner._connect_adapter_with_timeoutagainst aconnect()thatswallows
CancelledError.Every other bounded teardown/cleanup await in the gateway goes through
_await_adapter_cleanup_with_timeout(gateway/run.py:3796), which runs the child as adetached task so that when the deadline fires the runner is released even if the child
traps its cancellation. The connect path was the one exception: it used plain
asyncio.wait_for(adapter.connect(...)).wait_forcancels an overdue child and thenawaits it — so a
connect()whose teardown swallowsCancelledErrorwould keep thebackground reconnect watcher blocked indefinitely, leaving the gateway alive but never
rebuilding a deaf adapter.
This converts connect to the same detach-on-timeout pattern: on deadline the connect task
is cancelled and detached (drained via
consume_detached_task_result), and the callraises
TimeoutErrorexactly as before, so the watcher's existing failure/backoff path isunchanged.
Honesty note: this was found while investigating a ~22h silent-deaf wedge on 0.19.0
(issue linked below). That incident's freeze happened before the reconnect watcher ever
ran, so this is not a proven fix for it — it closes a real, separate
plain-wait_forinconsistency (defense-in-depth). The incident itself needs a stack dumpto localize, as described in the issue.
Related Issue
Fixes #70344
Type of Change
Changes Made
gateway/run.py—_connect_adapter_with_timeoutnow runsadapter.connect()as adetached task via
asyncio.wait({task}, timeout=...), returningtask.result()onsuccess (same result/exception surface as the old
await wait_for) and, on deadline,cancelling + detaching the task and raising
TimeoutError. A timed-out connect is neverinstalled in
self.adapters, so a late-completing swallowed-cancel connect staysunowned and is disposed by the existing failure path rather than leaving a
half-registered adapter (the race class addressed by Telegram polling silently dies after network error + self-restart #55992/fix(telegram): close reconnect races that leave the adapter half-destroyed (#55992) #56200).
tests/gateway/test_platform_reconnect.py— regression testtest_connect_timeout_releases_when_connect_swallows_cancel: aconnect()that trapsCancelledErrorand keeps running. It hangs the old plain-wait_forimplementation(verified: the 2s outer bound trips with a bare
asyncio.TimeoutError) and returnspromptly with the bounded
"... connect timed out"error after this change.How to Test
pytest tests/gateway/test_platform_reconnect.py -q— 39 passed.git stashthegateway/run.pychange andre-run just
test_connect_timeout_releases_when_connect_swallows_cancel— it fails(inner call hangs, outer 2s guard trips); restore and it passes.
Checklist
Code
fix(gateway): ...)pytest tests/ -qand all tests pass — ran the affected gateway suite(
tests/gateway/test_platform_reconnect.py, 39 passed); did not run the fulltests/(heavy optional deps, e.g. torch, not installed in my env)Documentation & Housekeeping