Skip to content

fix(gateway): detach-on-timeout for adapter connect so swallowed cancel can't wedge reconnect - #71176

Closed
kshitijk4poor wants to merge 1 commit into
NousResearch:mainfrom
kshitijk4poor:salvage/70345-connect-detach-timeout
Closed

fix(gateway): detach-on-timeout for adapter connect so swallowed cancel can't wedge reconnect#71176
kshitijk4poor wants to merge 1 commit into
NousResearch:mainfrom
kshitijk4poor:salvage/70345-connect-detach-timeout

Conversation

@kshitijk4poor

Copy link
Copy Markdown
Collaborator

Summary

Switches _connect_adapter_with_timeout from asyncio.wait_for() to the detach-on-timeout pattern, preventing a connect() that swallows CancelledError from wedging the reconnect watcher indefinitely.

Root cause: asyncio.wait_for cancels an overdue child task but then awaits it — a connect() whose teardown traps CancelledError blocks the caller forever, leaving the gateway alive but never rebuilding a deaf adapter.

Changes

  • gateway/run.py: _connect_adapter_with_timeout now uses asyncio.wait({task}, timeout=...) + consume_detached_task_result (same pattern as _await_adapter_cleanup_with_timeout)
  • tests/gateway/test_platform_reconnect.py: regression test test_connect_timeout_releases_when_connect_swallows_cancel — a connect() that traps CancelledError and keeps running

Credit: @VaitaR's original commit cherry-picked with authorship preserved.

Validation

Before After
connect() swallows CancelledError watcher wedged forever TimeoutError raised, watcher freed
Normal connect() works works
Targeted tests 39 passed

Closes #70344 (detach-on-timeout concern). The reconnect watcher respawn concern is addressed in a separate PR.

Infographic

Image generation is unavailable in this environment (no FAL_KEY / portal credits). The infographic will be attached once available.

…ncel can't wedge reconnect

_connect_adapter_with_timeout used a plain asyncio.wait_for(adapter.connect()),
which cancels an overdue child and then awaits it. A connect() whose teardown
swallows CancelledError would keep the background reconnect watcher blocked
indefinitely, leaving the gateway alive but never rebuilding a deaf adapter.

Run connect as a detached task (the pattern already used by
_await_adapter_cleanup_with_timeout): on deadline the task is cancelled and
drained via consume_detached_task_result and TimeoutError is raised, so the
watcher's existing failure/backoff path is unchanged. A timed-out connect is
never installed in self.adapters, so a late-completing swallowed-cancel connect
stays unowned and is disposed rather than leaving a half-registered adapter.
@kshitijk4poor
kshitijk4poor enabled auto-merge (rebase) July 25, 2026 03:29
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists needs-decision Awaiting maintainer decision before any implementation sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 25, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to #70344 and broader sibling #71177. The current combined salvage contains this same detach-on-timeout repair plus watcher respawn; maintainers should choose the focused or combined integration.

@teknium1

Copy link
Copy Markdown
Contributor

Closing as superseded — the detach-on-timeout piece already landed on main via #70987, and the supervision restore merged via #72366. Thanks!

@teknium1 teknium1 closed this Jul 27, 2026
auto-merge was automatically disabled July 27, 2026 02:32

Pull request was closed

@kshitijk4poor
kshitijk4poor deleted the salvage/70345-connect-detach-timeout branch August 5, 2026 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery needs-decision Awaiting maintainer decision before any implementation P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gateway silently deaf after generic network loss — reconnect watcher never starts post-escalation (0.19.0)

4 participants