fix: bound DoH fallback-IP discovery with wall-clock deadline - #74260
Closed
rkfshakti wants to merge 1 commit into
Closed
fix: bound DoH fallback-IP discovery with wall-clock deadline#74260rkfshakti wants to merge 1 commit into
rkfshakti wants to merge 1 commit into
Conversation
The discover_fallback_ips() call in the Telegram gateway connect path was not wrapped in _await_with_thread_deadline. While the DoH client has its own timeout, the system-DNS leg (asyncio.to_thread(socket.getaddrinfo)) can hang for minutes on broken VPN/DNS configs, blocking the event loop and preventing the 'Connecting to Telegram' retry ladder from ever running. The issue reporter confirmed the fix from NousResearch#64370 (_await_with_thread_deadline on initialize()) is present, but the gateway still hangs at 'Connecting to Telegram (attempt 1/8)' — because the hang starts earlier, in the DoH discovery that precedes the retry loop. Fix: wrap discover_fallback_ips() in _await_with_thread_deadline using the same HERMES_TELEGRAM_INIT_TIMEOUT (default 30s). On timeout, fall back to the seed fallback IPs instead of hanging indefinitely. Fixes NousResearch#74140
1 task
Contributor
|
Thanks for tracing the fallback-IP discovery path. Automated hermes-sweeper review found that current
Closing as implemented on main. |
|
@teknium1 was this an automated sweep and not a your human review? Not trying to be rude however, I'm running v2026.7.20 — the exact version cited as containing the fix. The code referenced is confirmed present in my image (verified via grep). The hang still reproduces every time. The existing fix wraps initialize() but discover_fallback_ips() hangs before initialize() is ever reached — the 'Auto-discovered Telegram fallback IPs' log line never appears in failing runs, only in the intermittent successes. Requesting another review if possible? |
This was referenced Aug 3, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Telegram gateway hangs indefinitely at
Connecting to Telegram (attempt 1/8)…afterDiscovering Telegram API fallback IPs via DNS-over-HTTPS…. No timeout fires, no retry, no error — the process must be killed manually.The fix from #64370 (
_await_with_thread_deadlineoninitialize()) is confirmed present in the running image, but the issue still reproduces.Root cause
The
discover_fallback_ips()call in the connect path is not wrapped in_await_with_thread_deadline. While the DoH client has its own timeout, the system-DNS leg (asyncio.to_thread(socket.getaddrinfo)) can hang for minutes on broken VPN/DNS configs, blocking the event loop and preventing the "Connecting to Telegram" retry ladder from ever running.The issue reporter sees the "Discovering Telegram API fallback IPs via DNS-over-HTTPS…" log, but the "Auto-discovered Telegram fallback IPs" log never appears — meaning
discover_fallback_ips()itself is hanging, notinitialize().Fix
Wrap
discover_fallback_ips()in_await_with_thread_deadlineusing the sameHERMES_TELEGRAM_INIT_TIMEOUT(default 30s). On timeout, fall back to the seed fallback IPs (_SEED_FALLBACK_IPS) instead of hanging indefinitely.This ensures the entire connect path — DoH discovery + initialize() — is bounded by wall-clock deadlines that don't depend on the event loop being responsive.
Fixes #74140