fix(telegram): drain send connection pool on sustained pool timeouts - #38812
fix(telegram): drain send connection pool on sustained pool timeouts#38812E-R-Butch wants to merge 2 commits into
Conversation
The httpx connection pool used for send_message / edit_message (_request[1], the general pool) had no recovery path. When proxy-level connection leaks filled all 512 pool slots, every send/edit failed with "Pool timeout: all connections occupied" and retrying was useless. Add _drain_send_connections() that cycles _request[1] (shutdown → initialize), mirroring _drain_polling_connections() which already exists for the polling pool (_request[0]). Wire it into the send() retry loop: after 3 consecutive pool timeout errors, drain the pool before the next retry attempt. Reset the counter on successful sends. The polling pool is intentionally left untouched by this drain — same separation guarantee the existing polling drain makes for sends.
|
I found one issue worth fixing before merge.
async def test_send_pool_drain_resets_counter():
adapter._send_pool_timeout_count = 3
await adapter._drain_send_connections()
adapter._send_pool_timeout_count = 0 # ← manual reset
assert adapter._send_pool_timeout_count == 0 # ← always passesThe manual Suggested fix: either remove the manual reset and assert directly (if the intent is to test that the caller resets), or rename the test to reflect what it actually verifies (e.g., async def test_send_pool_drain_resets_counter():
adapter._send_pool_timeout_count = 3
await adapter._drain_send_connections()
# _drain_send_connections itself doesn't reset the counter;
# the send() caller does. This test verifies drain completes cleanly.
assert adapter._send_pool_timeout_count == 3 # unchanged by drainThis is a minor test quality issue — the production code is correct (the |
- test_send_pool_drain_resets_counter: manual =0 before assert made it a no-op. Renamed to test_drain_send_does_not_reset_counter, correctly asserts counter stays at 3 after drain (reset is caller's responsibility in send()). - Replaced inconsistent hasattr/getattr patterns with uniform getattr default for defensive read; use direct assignment for writes where counter is guaranteed to exist. - Added test_send_pool_timeout_counter_full_lifecycle for counter lifecycle. Co-authored-by: liuhao1024 (code review)
|
Thanks for catching this — the manual Fixed in the latest push:
|
|
Automated hermes-sweeper review: this Telegram send-pool recovery fix is now implemented on main. Evidence:
Thanks to @E-R-Butch for the original fix and to @liuhao1024 for catching the dead-test issue in review; the same behavioral guarantee is now covered on main. |
Problem
The httpx connection pool used for send_message / edit_message (
_request[1], the general pool) had no recovery mechanism. When proxy-level connection leaks (half-closed TCP connections through Clash, etc.) filled all 512 pool slots, every subsequent send/edit failed with:Retrying doesn't help — stuck connections must be torn down. The only recovery was a full gateway restart.
The polling pool (
_request[0]) already had_drain_polling_connections(), but the send pool had no equivalent.Fix
_drain_send_connections()— mirrors_drain_polling_connections()but cycles_request[1](general request) instead of_request[0](polling)_send_pool_timeout_countcounter to track consecutive pool timeoutssend()retry loop: after 3 consecutive pool timeout errors, drain the send pool before the next retry attemptEach pool is drained independently — this fix does not touch the polling pool, same as the existing polling drain does not touch the send pool.
Tests
7 new tests in
test_telegram_network_reconnect.py:_request[1], not_request[0]_appor_app.botis None