fix(gateway): prevent reconnect watcher wedge after network-loss fatal error - #70919
Closed
kshitijk4poor wants to merge 2 commits into
Closed
fix(gateway): prevent reconnect watcher wedge after network-loss fatal error#70919kshitijk4poor wants to merge 2 commits into
kshitijk4poor wants to merge 2 commits into
Conversation
…l error (NousResearch#70344) Three-part fix for the gateway going silently deaf after a retryable fatal adapter error (e.g. httpx.ConnectError on Telegram): 1. **Detach-on-timeout in _connect_adapter_with_timeout** — Replaced plain asyncio.wait_for with the task-detach pattern used by _await_adapter_cleanup_with_timeout. asyncio.wait_for cancels the overdue task but then waits for it to exit, so a connect() that catches CancelledError can block recovery forever. The detach pattern releases the runner at the deadline via consume_detached_task_result. 2. **Ensure reconnect watcher always runs after escalation** — Added _ensure_reconnect_watcher_running(), called after queueing a retryable fatal error. If the reconnect watcher task has died (exhausted restart budget, terminal exception), it is respawned so queued platforms are never permanently stranded. 3. **Faulthandler at gateway startup** — Enabled faulthandler + SIGUSR2 dump to a rotating file under HERMES_HOME/logs/ for post-mortem diagnosis of future event-loop freezes. Tests added for _ensure_reconnect_watcher_running (alive, dead, not-started, not-running), fatal-error integration (retryable calls ensure, non-retryable does not), and _connect_adapter_with_timeout (timeout raises, success returns).
alt-glitch
added
type/bug
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Prevents gateway reconnect watcher from wedging permanently after a network-loss fatal error by respawning the watcher task and using a detach-on-timeout pattern for adapter connect.
Changes
gateway/run.py: Add_ensure_reconnect_watcher_running()— detects dead watcher task and respawns itgateway/run.py: Replaceasyncio.wait_forwith detach-on-timeout pattern in_connect_adapter_with_timeout()— prevents CancelledError-catching adapters from blocking recoverygateway/run.py: Addfaulthandlersetup at gateway start for post-mortem stack dumpsgateway/run.py: Track_reconnect_watcher_taskas instance attribute for liveness checkstests/gateway/test_platform_reconnect.py: 8 new tests covering watcher respawn, fatal error integration, and detach-on-timeoutFixes #70344.
Salvaged from @webtecnica's PR #70530 (commit 2 of 3).
Validation