Skip to content

fix(telegram): drain general send pool on pool timeout before retry (#53524) - #54121

Merged
teknium1 merged 2 commits into
mainfrom
hermes/hermes-07a113e9
Jun 28, 2026
Merged

fix(telegram): drain general send pool on pool timeout before retry (#53524)#54121
teknium1 merged 2 commits into
mainfrom
hermes/hermes-07a113e9

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

Telegram replies stop reaching the user once the general send pool wedges: the send path treats a pool timeout as retryable but never resets _request[1], so retries keep hammering the dead httpx client and silently drop the reply. This drains and re-inits the general Bot API pool on a confirmed pool timeout before retrying.

Salvage of #53622 by @yungchentang. Fixes #53524.

Changes

  • plugins/platforms/telegram/adapter.py: add _drain_general_connections_after_pool_timeout() (shutdown + reinit of _request[1]), guarded by an asyncio.Lock so concurrent sends can't tear the pool down mid-request. The send retry loop calls it when _looks_like_pool_timeout() matches, before the next attempt. The polling-pool drain (_request[0]) is left exactly as-is.
  • tests/gateway/test_telegram_thread_fallback.py: regression test — a pool timeout drains _request[1] before retry and leaves _request[0] untouched.

Why this isn't a duplicate of #51541

Merged #51541 added keepalive limits to the general pool to prevent the slow CLOSE_WAIT fd bleed. It explicitly noted the general pool still had no acute-recovery path when it does wedge. This PR closes exactly that gap — complementary, not redundant. (The issue's force_close_tcp_sockets/launchd framing is a stale mental model: that symbol lives in run_agent.py for agent HTTP/TLS fd recycle, unrelated to the Telegram adapter.)

Validation

Before After
General pool on send pool-timeout left wedged, retries fail → silent drop shut down + re-init, retry hits fresh client
Polling pool (_request[0]) untouched untouched (asserted)
Concurrent drains n/a serialized by asyncio.Lock, no mid-send teardown
  • scripts/run_tests.sh tests/gateway/test_telegram_thread_fallback.py tests/gateway/test_telegram_closewait_limits_31599.py → 50 passed.
  • E2E (real imports): general pool drained on pool timeout, polling pool 0/0, lock serializes concurrent drains, no-bot path crash-safe, pool-timeout detection discriminates from connect-timeout.
  • ruff clean, py_compile clean.

Infographic

Telegram send-pool recovery

@github-actions

Copy link
Copy Markdown
Contributor

🔎 Lint report: hermes/hermes-07a113e9 vs origin/main

ruff

Total: 0 on HEAD, 0 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 0 pre-existing issues carried over.

ty (type checker)

Total: 11624 on HEAD, 11624 on base (➖ 0)

🆕 New issues: none

✅ Fixed issues: none

Unchanged: 6104 pre-existing issues carried over.

Diagnostics are surfaced as warnings — this check never fails the build.

@alt-glitch alt-glitch added type/bug Something isn't working platform/telegram Telegram bot adapter comp/gateway Gateway runner, session dispatch, delivery sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages P1 High — major feature broken, no workaround labels Jun 28, 2026
@teknium1
teknium1 merged commit dddaea0 into main Jun 28, 2026
54 of 57 checks passed
@teknium1
teknium1 deleted the hermes/hermes-07a113e9 branch June 28, 2026 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P1 High — major feature broken, no workaround platform/telegram Telegram bot adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(gateway): Telegram HTTP connection pool exhaustion causes silent reply failure

3 participants