fix(telegram): wire keepalive limits into general request pool to fix CLOSE_WAIT fd leak (#31599) by kshitijk4poor · Pull Request #51541 · NousResearch/hermes-agent

kshitijk4poor · 2026-06-23T20:57:15Z

Summary

Closes #31599 — the Telegram adapter leaks httpx general-pool connections (CLOSE_WAIT fd leak), notably through an HTTP proxy.

The general request pool (_request[1], which routes bot.send_message / set_my_commands) is built from HTTPXRequest(...) with connection_pool_size / pool_timeout / timeouts but no httpx keepalive tuning, so httpx's default keepalive_expiry=5.0 lets dead sockets linger in CLOSE_WAIT. _drain_polling_connections() only resets _request[0] (the polling pool) and deliberately leaves _request[1] untouched — so the general pool has no recycling path. Telegram was the lone holdout of the #18451 CLOSE_WAIT class: wecom/dingtalk/signal/whatsapp/bluebubbles/qqbot already route through the shared platform_httpx_limits() helper; Telegram did not.

Verified still live on current main (plugins/platforms/telegram/adapter.py): no limits / keepalive_expiry / max_keepalive_connections on the general-pool construction; platform_httpx_limits() not wired in.

Fix

Wire the shared gateway/platforms/_http_client_limits.py::platform_httpx_limits() (bounded max_keepalive_connections + sub-default keepalive_expiry) into the general-pool HTTPXRequest construction across all three branches — fallback-transport, proxy (the reporter's actual path), and plain — via httpx_kwargs={"limits": ...}. PTB spreads httpx_kwargs last into its client kwargs, so this cleanly overrides PTB's default limits while preserving max_connections=connection_pool_size.

Why not PR #49930

The existing candidate #49930 was not salvageable:

it defines _TCP_KEEPIDLE/TCP_KEEPINTVL/TCP_KEEPCNT (Linux-only) at module top with no hasattr guard → crashes on import on macOS (the reporter's own OS);
it patches TelegramFallbackTransport, which the proxy repro never instantiates → doesn't fix the reported path.

This PR takes the proven keepalive-limits vector via the existing helper (no Linux-only socket constants → macOS-import-safe) and covers the proxy branch the reporter hit. Co-authored credit to @indigokarasu for the report + diagnosis.

Tests

tests/gateway/test_telegram_closewait_limits_31599.py — drives connect() across the proxy and plain branches with a recording HTTPXRequest, asserts each gets httpx_kwargs["limits"] = httpx.Limits with keepalive_expiry < 5.0, bounded max_keepalive_connections, and preserved max_connections. 56 pass (new tests + test_platform_http_client_limits.py + test_telegram_network.py); mutation-checked (dropping the limits wiring fails both branch tests). import plugins.platforms.telegram.adapter confirmed clean on macOS.

Fixes #31599

…est (#31599) PTB's HTTPXRequest builds its httpx.AsyncClient with `limits = httpx.Limits(max_connections=connection_pool_size)` and no keepalive tuning, so httpx's default keepalive_expiry=5.0 applies. Behind an HTTP proxy (Cloudflare Warp etc.) a peer-initiated FIN can sit in CLOSE_WAIT longer than that, leaking fds in the general request pool (_request[1], which routes bot.send_message/set_my_commands) — the pool _drain_polling_connections never resets. Telegram was the lone holdout adapter not using the shared #18451 CLOSE_WAIT helper. Wire gateway.platforms._http_client_limits.platform_httpx_limits() into the httpx client across ALL THREE request-construction branches — fallback-transport, proxy, and plain — via httpx_kwargs["limits"], which PTB spreads last into its client kwargs so our tuned limits win. PTB's connection_pool_size (max_connections) is preserved; only keepalive behaviour is tightened (max_keepalive_connections + keepalive_expiry<5.0). The fix is macOS-import-safe: no Linux-only socket TCP_KEEPIDLE/INTVL/CNT constants at module scope (unlike the broken candidate which crashed on import on the reporter's OS), and it patches the actual proxy path the repro hits rather than TelegramFallbackTransport, which the proxy repro never instantiates. Adds a mutation-survivable behavior-contract test asserting every HTTPXRequest built by connect() receives httpx_kwargs["limits"] with keepalive_expiry < httpx's 5.0 default, across both the proxy and plain branches. Reverting the limits wiring fails the test. Co-authored-by: indigokarasu <mx.indigo.karasu@gmail.com>

github-actions · 2026-06-23T20:58:19Z

🔎 Lint report: `salvage/31599-telegram-closewait` vs `origin/main`

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11124 on HEAD, 11121 on base (🆕 +3)

🆕 New issues (2):

Rule	Count
`unresolved-import`	2

First entries

tests/gateway/test_telegram_closewait_limits_31599.py:33: [unresolved-import] unresolved-import: Cannot resolve imported module `pytest`
tests/gateway/test_telegram_closewait_limits_31599.py:32: [unresolved-import] unresolved-import: Cannot resolve imported module `httpx`

✅ Fixed issues: none

Unchanged: 5863 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

… sockets When a Telegram long-poll TCP socket enters CLOSE-WAIT (remote sent FIN but httpx hasn't noticed), epoll still reports it readable so no exception is raised. PTB's error_callback never fires, the reconnect ladder never engages, and the gateway silently stops receiving messages while the process stays alive — until a manual systemctl restart. The existing recovery only covers two cases: error_callback-driven reconnects (which require an exception PTB never gets) and a one-shot _verify_polling_after_reconnect probe (which runs only right after an explicit reconnect). A socket that wedges during steady-state operation is never detected. Add _polling_heartbeat_loop: a background asyncio.Task started in connect() (polling mode only) that probes get_me() every 90s on the general request pool (not the getUpdates pool, so healthy long-polls are never interrupted). On asyncio.TimeoutError/OSError it hands off to the existing _handle_polling_network_error ladder; other errors are swallowed. disconnect() cancels and awaits the task. Worst-case detection window ~105s. Complementary to #51541 (general-pool keepalive limits / fd leak) — that recycles idle pooled connections; this detects a wedged active read. Fixes #48495 Co-authored-by: agt-user <267614622+agt-user@users.noreply.github.com>

… sockets When a Telegram long-poll TCP socket enters CLOSE-WAIT (remote sent FIN but httpx hasn't noticed), epoll still reports it readable so no exception is raised. PTB's error_callback never fires, the reconnect ladder never engages, and the gateway silently stops receiving messages while the process stays alive — until a manual systemctl restart. The existing recovery only covers two cases: error_callback-driven reconnects (which require an exception PTB never gets) and a one-shot _verify_polling_after_reconnect probe (which runs only right after an explicit reconnect). A socket that wedges during steady-state operation is never detected. Add _polling_heartbeat_loop: a background asyncio.Task started in connect() (polling mode only) that probes get_me() every 90s on the general request pool (not the getUpdates pool, so healthy long-polls are never interrupted). On asyncio.TimeoutError/OSError it hands off to the existing _handle_polling_network_error ladder; other errors are swallowed. disconnect() cancels and awaits the task. Worst-case detection window ~105s. Complementary to NousResearch#51541 (general-pool keepalive limits / fd leak) — that recycles idle pooled connections; this detects a wedged active read. Fixes NousResearch#48495 Co-authored-by: agt-user <267614622+agt-user@users.noreply.github.com>

…-telegram-closewait fix(telegram): wire keepalive limits into general request pool to fix CLOSE_WAIT fd leak (NousResearch#31599)