Skip to content

fix: bound DoH fallback-IP discovery with wall-clock deadline - #74260

Closed
rkfshakti wants to merge 1 commit into
NousResearch:mainfrom
rkfshakti:fix/telegram-gateway-hang-74140
Closed

fix: bound DoH fallback-IP discovery with wall-clock deadline#74260
rkfshakti wants to merge 1 commit into
NousResearch:mainfrom
rkfshakti:fix/telegram-gateway-hang-74140

Conversation

@rkfshakti

Copy link
Copy Markdown
Contributor

Problem

The Telegram gateway hangs indefinitely at Connecting to Telegram (attempt 1/8)… after Discovering Telegram API fallback IPs via DNS-over-HTTPS…. No timeout fires, no retry, no error — the process must be killed manually.

The fix from #64370 (_await_with_thread_deadline on initialize()) is confirmed present in the running image, but the issue still reproduces.

Root cause

The discover_fallback_ips() call in the connect path is not wrapped in _await_with_thread_deadline. While the DoH client has its own timeout, the system-DNS leg (asyncio.to_thread(socket.getaddrinfo)) can hang for minutes on broken VPN/DNS configs, blocking the event loop and preventing the "Connecting to Telegram" retry ladder from ever running.

The issue reporter sees the "Discovering Telegram API fallback IPs via DNS-over-HTTPS…" log, but the "Auto-discovered Telegram fallback IPs" log never appears — meaning discover_fallback_ips() itself is hanging, not initialize().

Fix

Wrap discover_fallback_ips() in _await_with_thread_deadline using the same HERMES_TELEGRAM_INIT_TIMEOUT (default 30s). On timeout, fall back to the seed fallback IPs (_SEED_FALLBACK_IPS) instead of hanging indefinitely.

This ensures the entire connect path — DoH discovery + initialize() — is bounded by wall-clock deadlines that don't depend on the event loop being responsive.

Fixes #74140

The discover_fallback_ips() call in the Telegram gateway connect path was
not wrapped in _await_with_thread_deadline. While the DoH client has its
own timeout, the system-DNS leg (asyncio.to_thread(socket.getaddrinfo))
can hang for minutes on broken VPN/DNS configs, blocking the event loop
and preventing the 'Connecting to Telegram' retry ladder from ever running.

The issue reporter confirmed the fix from NousResearch#64370 (_await_with_thread_deadline
on initialize()) is present, but the gateway still hangs at 'Connecting to
Telegram (attempt 1/8)' — because the hang starts earlier, in the DoH
discovery that precedes the retry loop.

Fix: wrap discover_fallback_ips() in _await_with_thread_deadline using the
same HERMES_TELEGRAM_INIT_TIMEOUT (default 30s). On timeout, fall back to
the seed fallback IPs instead of hanging indefinitely.

Fixes NousResearch#74140
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/plugins Plugin system and bundled plugins platform/telegram Telegram bot adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jul 29, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for tracing the fallback-IP discovery path.

Automated hermes-sweeper review found that current main already ships this fix:

  • plugins/platforms/telegram/telegram_network.py:242-258 starts system DNS in a worker thread, gathers DoH independently, and bounds the resolver wait with _DOH_TIMEOUT.
  • plugins/platforms/telegram/telegram_network.py:280-285 already falls back to _SEED_FALLBACK_IPS when discovery has no usable DoH result.
  • tests/gateway/test_telegram_network.py:467-490 covers a hung socket.getaddrinfo and verifies discovery returns without being gated on it.
  • This was implemented by e16743b0d5 (fix(telegram): diagnose blocked-loop init hangs, unbind DoH from system DNS) and shipped in v2026.7.20.

Closing as implemented on main.

@teknium1 teknium1 closed this Jul 30, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 30, 2026
@Pachnoc

Pachnoc commented Jul 30, 2026

Copy link
Copy Markdown

@teknium1 was this an automated sweep and not a your human review? Not trying to be rude however, I'm running v2026.7.20 — the exact version cited as containing the fix. The code referenced is confirmed present in my image (verified via grep). The hang still reproduces every time. The existing fix wraps initialize() but discover_fallback_ips() hangs before initialize() is ever reached — the 'Auto-discovered Telegram fallback IPs' log line never appears in failing runs, only in the intermittent successes. Requesting another review if possible?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P1 High — major feature broken, no workaround platform/telegram Telegram bot adapter sweeper:implemented-on-main Sweeper: behavior already present on current main sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Telegram gateway hangs at "Connecting to Telegram (attempt 1/8)" — fix #64370 confirmed present, still reproduces

4 participants