fix(gateway): recover Telegram after fatal disconnect wedge (#80598) - #80700
Closed
HexLab98 wants to merge 2 commits into
Closed
fix(gateway): recover Telegram after fatal disconnect wedge (#80598)#80700HexLab98 wants to merge 2 commits into
HexLab98 wants to merge 2 commits into
Conversation
…earch#80598) After a network outage the Telegram fatal handler could hang inside disconnect() and never populate _failed_platforms, so the reconnect watcher had nothing to retry and the process stayed permanently deaf. Queue retryable platforms before any disconnect await, bound the fatal handler with an outer detach deadline, and release the Telegram token lock / PTB close steps with detach-on-timeout so recovery cannot stall.
alt-glitch
added
type/bug
kshitijk4poor
added a commit
that referenced
this pull request
Aug 7, 2026
…laim keys Follow-up to #80700: 1. _await_disconnect_step was missing the try/except CancelledError around asyncio.wait() that _await_adapter_cleanup_with_timeout already has. When the outer fatal-handler timeout cancels disconnect() mid-step, asyncio.wait does NOT cancel its inner task — the task was orphaned with no observer. Add the same cancel+detach+re-raise pattern. 2. _queue_retryable_fatal_platform omitted credential_claim/listener_claim keys that all 3 startup-path queue sites include. These are consumed by the multiplex reservation logic to prevent secondary profiles from taking the endpoint while a primary is queued. Pre-existing latent bug — now fixed since the extraction makes it trivial.
Collaborator
|
Merged via #81036 — your commits were cherry-picked onto current Your fix was correct and well-tested. During review we found two small follow-ups that we applied on top:
Thanks for the thorough bug report (#80598) and the clean fix! |
ma1138569845
pushed a commit
to ma1138569845/dechnicAuditor-agent
that referenced
this pull request
Aug 10, 2026
…laim keys Follow-up to NousResearch#80700: 1. _await_disconnect_step was missing the try/except CancelledError around asyncio.wait() that _await_adapter_cleanup_with_timeout already has. When the outer fatal-handler timeout cancels disconnect() mid-step, asyncio.wait does NOT cancel its inner task — the task was orphaned with no observer. Add the same cancel+detach+re-raise pattern. 2. _queue_retryable_fatal_platform omitted credential_claim/listener_claim keys that all 3 startup-path queue sites include. These are consumed by the multiplex reservation logic to prevent secondary profiles from taking the endpoint while a primary is queued. Pre-existing latent bug — now fixed since the extraction makes it trivial.
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…laim keys Follow-up to NousResearch#80700: 1. _await_disconnect_step was missing the try/except CancelledError around asyncio.wait() that _await_adapter_cleanup_with_timeout already has. When the outer fatal-handler timeout cancels disconnect() mid-step, asyncio.wait does NOT cancel its inner task — the task was orphaned with no observer. Add the same cancel+detach+re-raise pattern. 2. _queue_retryable_fatal_platform omitted credential_claim/listener_claim keys that all 3 startup-path queue sites include. These are consumed by the multiplex reservation logic to prevent secondary profiles from taking the endpoint while a primary is queued. Pre-existing latent bug — now fixed since the extraction makes it trivial.
This was referenced Aug 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
disconnect()await, so a wedged Telegram close cannot leave_failed_platformsempty and the reconnect watcher with nothing to retry (#80598).TelegramAdapter.disconnect()(release the bot-token lock immediately; bound lifecycle/PTBstop/shutdownwith detach-on-timeout) so recovery cannot stall on a half-dead socket after a network outage.Test plan
scripts/run_tests.sh tests/gateway/test_runner_fatal_adapter.py tests/gateway/test_telegram_network_reconnect.py tests/gateway/test_platform_reconnect.py tests/gateway/test_safe_adapter_disconnect.py -qapi.telegram.orgunreachability untiltelegram_network_errorfatal escalation; confirm logs showqueued for background reconnection/gateway staying aliveand that Telegram reconnects after the network returns without a manual gateway restartdisconnect()no longer strands the platform while cron/MCP keep the process aliveFixes #80598