fix(gateway): prevent reconnect watcher wedge after network-loss fatal error - #70987
Merged
Conversation
…l error (#70344) Three-part fix for the gateway going silently deaf after a retryable fatal adapter error (e.g. httpx.ConnectError on Telegram): 1. **Detach-on-timeout in _connect_adapter_with_timeout** — Replaced plain asyncio.wait_for with the task-detach pattern used by _await_adapter_cleanup_with_timeout. asyncio.wait_for cancels the overdue task but then waits for it to exit, so a connect() that catches CancelledError can block recovery forever. The detach pattern releases the runner at the deadline via consume_detached_task_result. 2. **Ensure reconnect watcher always runs after escalation** — Added _ensure_reconnect_watcher_running(), called after queueing a retryable fatal error. If the reconnect watcher task has died (exhausted restart budget, terminal exception), it is respawned so queued platforms are never permanently stranded. 3. **Faulthandler at gateway startup** — Enabled faulthandler + SIGUSR2 dump to a rotating file under HERMES_HOME/logs/ for post-mortem diagnosis of future event-loop freezes. Tests added for _ensure_reconnect_watcher_running (alive, dead, not-started, not-running), fatal-error integration (retryable calls ensure, non-retryable does not), and _connect_adapter_with_timeout (timeout raises, success returns).
…ailures When connected_count == 0 and at least one platform failed with a non-retryable error, the runner exited with GATEWAY_FATAL_CONFIG_EXIT_CODE (78) even if OTHER platforms failed for merely transient reasons. Real-world shape (NS-609, hosted instance): WhatsApp enabled but never paired (non-retryable whatsapp_not_paired) + Telegram TimedOut during polling startup (retryable) => exit 78 => the gateway either goes permanently down (supervisors honoring the exit-78 contract via RestartPreventExitStatus / the s6 finish->125 translation from #51228) or crash-loops (anything else). Either way Telegram never gets its retry and the dashboard drops with every exit, so a single unpaired platform plus one network blip disconnected every channel on the instance. Now exit 78 is reserved for the case where ALL startup failures are non-retryable (true config error, nothing to wait for). With mixed failures the gateway stays alive in degraded state: the reconnect watcher recovers the retryable platforms and the misconfigured ones stay fatal-parked and visible in runtime status.
signal.SIGUSR2 and faulthandler.register() don't exist on Windows; the bare reference raised AttributeError at import time per the windows-footgun checker. faulthandler.enable() still covers fatal-error dumps on all platforms.
alt-glitch
added
type/bug
Contributor
૮ >ﻌ< ა ci reviewran on c841846 all good! |
This was referenced Jul 24, 2026
Contributor
This was referenced Jul 27, 2026
Closed
fix(gateway): detach-on-timeout for adapter connect so swallowed cancel can't wedge reconnect
#71176
Closed
18 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The gateway no longer goes silently deaf after a network loss escalates to a fatal error — the reconnect watcher is (re)started post-escalation, and mixed retryable + non-retryable startup failures no longer kill the process.
Changes
Validation
Targeted gateway reconnect/startup suites green; ruff clean.
Supersedes #70502 (byte-identical core diff, bundled unrelated kanban changes, 163 commits stale).
Fixes #70344
Infographic