fix(telegram): probe polling liveness after reconnect to detect wedged Updater

[jslizar]

Closes #18086

Detect wedged Telegram polling after reconnect via heartbeat probe

Summary

_handle_polling_network_error currently treats Updater.start_polling() returning successfully as proof that polling has resumed. In practice, the underlying long-poll task can be left wedged on a stale httpx connection — Updater.running is True but no getUpdates calls actually progress, no error callback fires, and the reconnect ladder sits at attempt 1 forever. The fatal-error path is never reached, so the gateway runs indefinitely with no working Telegram polling.

This PR adds a deferred heartbeat probe scheduled after each successful start_polling() in the reconnect path. The probe verifies that the bot endpoint is reachable through the same client a healthy long-poll would use; on failure it re-enters the existing reconnect ladder so the proven escalation path (eventually _set_fatal_error(retryable=True) after MAX_NETWORK_RETRIES) can fire.

Why

See the linked issue for the full repro and log evidence: a single transient Bad Gateway from getUpdates triggered the customer-side wedge — Updater.stop() raised a TimedOut from _get_updates_cleanup (logged as "Suppressing error to ensure graceful shutdown"), the surrounding try / except Exception: pass swallowed it, then start_polling() returned without raising but polling never actually consumed any further updates. No "polling resumed" log, no "reconnecting in 10s, attempt 2/10", no fatal-error path — just silence for 11 hours until the container was manually restarted.

The first reconnect attempt looks like recovery in the logs; only the absence of subsequent activity reveals the wedge. The existing safeguards (MAX_NETWORK_RETRIES, fatal-error retryable=True with supervisor restart) are sound but unreachable when polling silently wedges.

Approach

Two-part fix, both inside the existing reconnect abstraction (no PTB-internal coupling, no Application rebuild):

1. After a successful start_polling() in _handle_polling_network_error, schedule _verify_polling_after_reconnect() as a background task.

2. The probe waits HEARTBEAT_PROBE_DELAY (60s, comfortably above one healthy long-poll cycle), then verifies Updater.running is still True and Bot.get_me() returns within PROBE_TIMEOUT (10s) via asyncio.wait_for. Either failure feeds back into _handle_polling_network_error so the reconnect ladder advances.

This is a minimal additive layer — no behavior change on the happy path, and on the wedged path the existing MAX_NETWORK_RETRIES ladder eventually escalates to fatal-error so external supervisors (systemd Restart=on-failure) can do their job.

Considered alternatives

• Full Application rebuild on every reconnect. Heavier, more places to break, requires re-binding handlers; this PR is the conservative additive change.
• Don't swallow Updater.stop() exceptions. Worth doing on its own, but doesn't help in the case where stop() cleans up successfully and start_polling() is the silent failure point.
• Watermark counter incremented per dispatched update. Doesn't differentiate "wedged" from "legitimately idle" — a healthy bot with no inbound DMs has the same signature as a wedged one.

Bot.get_me() was chosen as the probe because it shares the bot's httpx client (so a wedged pool fails the probe) and doesn't conflict with Updater.getUpdates() (no 409).

Test plan

tests/platforms/test_polling_heartbeat.py (added in this PR) covers:

• Healthy reconnect path: Updater is running, get_me returns → probe is a no-op.
• Updater non-running after delay: probe re-enters the reconnect ladder with a synthetic RuntimeError.
• get_me times out: probe re-enters the reconnect ladder with the timeout exception.
• get_me raises (NetworkError / ConnectionError): probe re-enters the reconnect ladder with the original exception.
• Adapter already fatal-error'd: probe bails without further action.

Tests use unittest.mock.AsyncMock for the PTB Application/Updater/Bot and patch asyncio.sleep/asyncio.wait_for so the suite runs in milliseconds.

End-to-end repro (operator-side, optional): boot the gateway against a transparent proxy that returns 502 Bad Gateway for getUpdates for 30s then 200; observe the heartbeat probe firing and the reconnect ladder progressing past attempt 1 instead of silencing.

Files changed

• gateway/platforms/telegram.py — schedules _verify_polling_after_reconnect() after a successful reconnect; defines the new method.
• tests/platforms/test_polling_heartbeat.py — unit tests for the probe.

Open questions for maintainers

• Constants HEARTBEAT_PROBE_DELAY and PROBE_TIMEOUT are currently hardcoded; happy to move them next to MAX_NETWORK_RETRIES / BASE_DELAY as module-level constants or env knobs if there's a preference.
• Whether to also remove the bare except Exception: pass around Updater.stop() in the reconnect path. That's a behavioral change worth its own discussion, kept out of this PR for scope.
• Whether the probe should be reused outside the reconnect path (e.g. periodic liveness) — that would close the loop on wedges that occur without an antecedent network error.

[teknium1]

Merged via #18751 — your commit cherry-picked onto current main with authorship preserved via rebase-merge. One test-file merge conflict resolved (HEAD had drain-connection tests from a prior salvage; kept both those and your heartbeat-probe tests). Thanks for the detailed issue write-up and the thoughtful defense-in-depth fix!

#18751