Skip to content

fix(telegram): bound fallback-pool connection budget by pool count - #82860

Open
chelsealong wants to merge 1 commit into
NousResearch:mainfrom
chelsealong:fix/telegram-fallback-pool-fd-budget-82678
Open

fix(telegram): bound fallback-pool connection budget by pool count#82860
chelsealong wants to merge 1 commit into
NousResearch:mainfrom
chelsealong:fix/telegram-fallback-pool-fd-budget-82678

Conversation

@chelsealong

Copy link
Copy Markdown
Contributor

What does this PR do?

Bounds the Telegram fallback-IP connection-pool budget so it no longer scales
multiplicatively with the number of fallback IPs.

Related Issue

Fixes #82678

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Root cause

TelegramAdapter.connect() builds two HTTPXRequest clients (general +
get_updates). On the fallback-IP path, each wraps a TelegramFallbackTransport
that owns one primary AsyncHTTPTransport plus one lazily-built transport per
fallback IP. The adapter passed the same httpx.Limits(max_connections=connection_pool_size)
(default 512, HERMES_TELEGRAM_HTTP_POOL_SIZE) straight into
TelegramFallbackTransport, which forwards that exact object to its primary
pool and to every fallback pool unchanged — that "caller limits win" behavior
is intentional and already covered by
test_caller_limits_win_over_pool_default / test_forwards_limits_to_inner_transports.

The result: the effective ceiling was 2 clients × (1 primary + N fallback) × 512, i.e. 1024 connections with zero fallback IPs and 2048 with one — far
beyond a typical 256 soft RLIMIT_NOFILE, before SQLite, logs, pipes, and
other sockets are counted. The fallback-IP list itself was also unbounded,
since DNS-over-HTTPS discovery returns every unique A record it sees.

#45507 lowers the same default from 512 to 64 but (as noted on the issue)
still doesn't bound the address count or the per-pool multiplication — with
one fallback IP it's still 2 × (1+1) × 64 = 256 before other descriptors are
counted, growing further with more addresses.

Changes Made

  • plugins/platforms/telegram/telegram_network.py: added cap_fallback_ips()
    (bounds the fallback-IP list to DEFAULT_MAX_FALLBACK_IPS = 3, logging when
    truncated) and safe_fallback_pool_limits() (derates a httpx.Limits by the
    total pool count — num_ptb_clients * (1 + fallback_ip_count) — so the
    total connection budget across every pool stays close to what was
    configured, rather than multiplying by pool count).
  • plugins/platforms/telegram/adapter.py: apply cap_fallback_ips() to the
    resolved fallback-IP list in connect(), and pass the derated limits (via
    safe_fallback_pool_limits()) into TelegramFallbackTransport instead of
    the raw per-client limits object. TelegramFallbackTransport itself is
    unchanged — its "caller limits win" contract stays intact.
  • tests/gateway/test_telegram_network.py: added TestCapFallbackIps and
    TestSafeFallbackPoolLimits covering the cap and the derating math
    (including the zero-fallback-IP and near-zero-budget edge cases).

How to Test

  1. bash scripts/run_tests.sh tests/gateway/test_telegram_network.py — new
    tests fail with AttributeError: module ... has no attribute 'cap_fallback_ips' / 'safe_fallback_pool_limits' against the pre-fix
    source (confirmed via git stash on just the two source files with the
    test file kept), and pass after the fix.

  2. bash scripts/run_tests.sh tests/gateway/test_telegram_network.py tests/gateway/test_telegram_fallback_pool_release_71593.py tests/gateway/test_telegram_closewait_limits_31599.py tests/gateway/test_telegram_connect.py tests/gateway/test_telegram_network_reconnect.py — all pass:

    === Summary: 5 files, 45 tests passed, 0 failed (100% complete) in 6.6s (8 workers) ===
    
  3. bash scripts/run_tests.sh tests/gateway/ (611 files):

    === Summary: 611 files, 5113 tests passed, 2 failed (100% complete) in 151.6s (8 workers) ===
    

    The 2 failures are both in tests/gateway/test_wecom_callback.py
    (test_build_event_extracts_text_message,
    test_poll_loop_dispatches_handle_message), unrelated to this change —
    they fail with the same AttributeError: 'NoneType' object has no attribute 'fromstring' in plugins/platforms/wecom/callback_adapter.py:400
    on HEAD~1 (before this commit) as well, confirmed by checking out the
    pre-fix source and rerunning that file in isolation.

  4. ruff check plugins/platforms/telegram/adapter.py plugins/platforms/telegram/telegram_network.py tests/gateway/test_telegram_network.pyAll checks passed!

Checklist

Code

Documentation & Housekeeping

  • N/A — no config keys added, no docs/architecture changes

Note on AI assistance

Drafted with AI assistance (an autonomous coding agent), with the diff,
tests, and RED→GREEN verification reviewed before pushing.

TelegramFallbackTransport forwards whatever `limits` it is given to its
primary pool and to every lazily-built fallback pool unchanged. The adapter
was passing the same per-pool `httpx.Limits(max_connections=connection_pool_size)`
used for a single client straight through, so the effective connection
ceiling scaled with 2 PTB clients * (1 + fallback IP count) instead of
staying at what the operator configured — with the default 512 pool size and
even one fallback IP, up to 2048 connections. The number of fallback IPs was
also unbounded, since DoH discovery returns every unique A record.

Cap the fallback-IP list and derate the per-pool limit by the number of
pools that will actually be constructed, so raising the fallback-IP count no
longer multiplies the total connection budget.

Fixes NousResearch#82678
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/plugins Plugin system and bundled plugins platform/telegram Telegram bot adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/plugins Plugin system and bundled plugins P3 Low — cosmetic, nice to have platform/telegram Telegram bot adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Telegram fallback path can exceed the process FD budget through uncapped 512-connection pools

2 participants