fix(gateway): respawn dead reconnect watcher + faulthandler for event-loop freeze diagnosis - #71177
Closed
kshitijk4poor wants to merge 2 commits into
Closed
Conversation
…esearch#70344) Cherry-picked from @webtecnica's PR NousResearch#70502. The reconnect watcher task can die after exhausting its _spawn_supervised restart budget, leaving platforms permanently stranded. _ensure_reconnect_watcher_running detects a dead task and respawns it.
…th + file handle
- _ensure_reconnect_watcher_running now uses _spawn_supervised instead of
bare asyncio.create_task, preserving crash-restart supervision (backoff,
crash-loop detection, healthy-run reset)
- Faulthandler log path uses _hermes_home (already available at module level)
instead of os.environ.get('HERMES_HOME'), respecting profile overrides
- Faulthandler file handle stored in self._faulthandler_file for clean shutdown
- _connect_adapter_with_timeout: use task.result() instead of bool(result)
to preserve the original return value from adapter.connect()
kshitijk4poor
enabled auto-merge (rebase)
July 25, 2026 03:29
alt-glitch
added
type/bug
Collaborator
This was referenced Jul 25, 2026
fix(gateway): detach-on-timeout for adapter connect so swallowed cancel can't wedge reconnect
#71176
Closed
Contributor
|
Closing as superseded. The detach-on-timeout + |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ensures the platform reconnect watcher is always alive after queueing a retryable fatal adapter error, and adds faulthandler at gateway startup for post-mortem diagnosis of event-loop freezes.
Root cause (#70344): if the reconnect watcher task dies (exhausts its
_spawn_supervisedrestart budget), platforms queued for reconnection are permanently stranded — the gateway stays alive but never rebuilds deaf adapters.Changes
gateway/run.py:_ensure_reconnect_watcher_running()detects a dead reconnect watcher task and respawns it via_spawn_supervised(preserving crash-restart supervision). Called from_handle_adapter_fatal_error_implafter queueing a retryable fatal error.gateway/run.py:faulthandler.enable()+SIGUSR2dump togateway_faulthandler.logfor post-mortem diagnosis of event-loop freezes.tests/gateway/test_platform_reconnect.py: 8 new tests (4 for_ensure_reconnect_watcher_running, 2 for fatal error integration, 2 for detach-on-timeout)Salvage fixes on top of contributor's commit:
_ensure_reconnect_watcher_runninguses_spawn_supervisedinstead of bareasyncio.create_task(preserves backoff, crash-loop detection, healthy-run reset)_hermes_home(module-levelget_hermes_home()) instead ofos.environ.get("HERMES_HOME")(respects profile overrides)self._faulthandler_filefor clean shutdown (no leaked file handle)_connect_adapter_with_timeoutusestask.result()instead ofbool(result)(preserves original return value)Credit: @webtecnica's original commit cherry-picked with authorship preserved.
Validation
Related: #70345 (detach-on-timeout, same issue's defense-in-depth concern)
Infographic
Image generation is unavailable in this environment (no FAL_KEY / portal credits). The infographic will be attached once available.