Skip to content

fix(telegram): recover when polling updater stops while process stays alive (#55769) - #55789

Closed
PRATHAMESH75 wants to merge 1 commit into
NousResearch:mainfrom
PRATHAMESH75:fix/telegram-stopped-updater-watchdog
Closed

fix(telegram): recover when polling updater stops while process stays alive (#55769)#55789
PRATHAMESH75 wants to merge 1 commit into
NousResearch:mainfrom
PRATHAMESH75:fix/telegram-stopped-updater-watchdog

Conversation

@PRATHAMESH75

Copy link
Copy Markdown
Contributor

What does this PR do?

After a turn that triggers a background-review, the Telegram gateway can silently stop receiving messages while the process stays alive (systemd green, threads healthy) — no logs, no errors — until manually restarted (#55769).

The adapter already has a polling heartbeat (_polling_heartbeat_loop_probe_pending_updates) designed to catch a wedged long-poll consumer. But it only handled the running-but-wedged case. When PTB's Updater stops entirely (running == False) with no reconnect in flight — the long-poll task is simply gone — the probe treated it as "someone else's job", reset its counter, and returned. get_me() on the general request path stays healthy, so neither PTB's error_callback nor the connectivity heartbeat ever fires. Result: the watchdog has a blind spot exactly matching the report — process alive, send path fine, polling dead, indefinitely.

This fixes the gap at the existing boundary: _probe_pending_updates now detects the stopped-updater case directly and feeds it into the same _handle_polling_network_error recovery ladder the wedged-consumer and PTB error paths already use. No new restart machinery, no new config keys, no new env vars.

Detection is debounced over two consecutive probes (and skipped while a reconnect is already in flight) so a just-starting updater or the brief stop()start_polling() window of an in-flight reconnect never trips a false recovery.

Related Issue

Fixes #55769

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • plugins/platforms/telegram/adapter.py_probe_pending_updates: detect updater.running == False (no reconnect in flight) as a dead poller and escalate through _handle_polling_network_error after two consecutive probes; added _polling_not_running_count debounce counter; moved the in-flight-reconnect guard ahead of the updater check so its transient stop/start window can't be misread.
  • tests/gateway/test_telegram_pending_update_probe.py — replaced the test that encoded the old (buggy) "stopped updater = no-op" assumption with coverage for: single stopped probe does not escalate, two consecutive stopped probes trigger recovery, a recovered (running) updater resets the counter, and an in-flight reconnect suppresses escalation.

How to Test

scripts/run_tests.sh tests/gateway/test_telegram_pending_update_probe.py

Result: 9 tests passed, 0 failed.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run the affected tests via scripts/run_tests.sh and they pass
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)
  • I've tested on my platform: macOS (Darwin 25.5.0)

Documentation & Housekeeping

Screenshots / Logs

On recovery the heartbeat now logs (previously: total silence):

[telegram] Telegram polling heartbeat: updater stopped while in polling mode (stuck probe 1/2)
[telegram] Telegram polling heartbeat: updater stopped while in polling mode (stuck probe 2/2)
[telegram] Telegram updater is not running (long-poll task gone); triggering polling restart
[telegram] Telegram polling resumed after network error (attempt 1)

… alive

The polling heartbeat's pending-update probe treated a stopped updater
(running=False) as "someone else's job" and silently reset its counter,
so a long-poll task that disappears with no reconnect in flight was never
recovered. get_me() on the general request path stays healthy, so neither
PTB's error_callback nor the connectivity probe ever fires — the gateway
keeps running but stops receiving messages indefinitely (NousResearch#55769).

Detect the stopped-updater case directly in _probe_pending_updates and feed
it into the existing _handle_polling_network_error ladder, debounced over two
consecutive probes so a just-starting updater or the brief stop()->start_polling()
window of an in-flight reconnect never trips it.
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery platform/telegram Telegram bot adapter P3 Low — cosmetic, nice to have sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jun 30, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Merged via PR #55905 (commit e55e9fa on main) — your commit was cherry-picked onto current main with your authorship preserved in git log. Clean, surgical fix that closes the watchdog blind spot exactly at the existing boundary. Thanks @PRATHAMESH75!

@teknium1

Copy link
Copy Markdown
Contributor

Merged via PR #55921 — your commit was cherry-picked onto current main with your authorship preserved in git log. Thanks for the clean, well-tested fix! #55921

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have platform/telegram Telegram bot adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Gateway Telegram polling silently dies after bg-review thread hits LLM connection errors + memory tool failures

3 participants