Skip to content

fix(telegram): recover from post-update polling conflict without entering limbo - #25630

Closed
CryptoByz wants to merge 1 commit into
NousResearch:mainfrom
CryptoByz:fix/telegram-polling-conflict-limbo
Closed

fix(telegram): recover from post-update polling conflict without entering limbo#25630
CryptoByz wants to merge 1 commit into
NousResearch:mainfrom
CryptoByz:fix/telegram-polling-conflict-limbo

Conversation

@CryptoByz

Copy link
Copy Markdown
Contributor

fix(telegram): recover from post-update polling conflict without entering limbo
Fixes #23783
Problem
Running hermes update while the gateway is active leaves Telegram broken
in two distinct ways.

Stage 1, 409 Conflict on startup.
When the old gateway process exits, Telegram keeps its getUpdates
long-poll session open on its servers for up to ~30 s. The new gateway
starts immediately (the watcher loop has no built-in delay) and receives
a 409 Conflict from Telegram's API before that server-side session has
expired.

Stage 2, Silent limbo after a failed retry (the worse bug).
_handle_polling_conflict() attempts to recover by sleeping 10 s and
calling start_polling() again. If that second start_polling() call
raises (because the server-side session is still alive), the method
returns silently. At this point:

updater.running is False
No fatal error is set
The gateway process is alive and /status reports "connected"
No messages are received or sent

The adapter is in an undetectable limbo. The only recovery was to
manually kill the process and restart the gateway.
Root causes

_handle_polling_conflict after a failed retry, the method
returns without scheduling another attempt or setting a fatal error.
The adapter silently stops processing messages.
hermes update (manual gateways) after SIGTERM / SIGUSR1, the
watcher process respawns the new gateway with no delay, maximising
the chance of hitting the Telegram server-side 409 window.

Fix
gateway/platforms/telegram.py _handle_polling_conflict

Retry count: 3 → 5, giving up to ~175 s of total back-off time
(well beyond Telegram's ~30 s server-side expiry window).
Back-off delay now increases with each attempt (10 + attempt * 10
s: 15 s, 25 s, 35 s, 45 s, 55 s). A flat 10 s was not always enough
to clear the first 409; an increasing delay avoids hammering the API
while still recovering quickly on the common case.
Failed retries now schedule the next attempt via loop.create_task()
instead of returning silently. This closes the limbo window: the
adapter is always either recovering, running, or in a declared fatal
state.
Clearer log messages and fatal error text the warning now explains
why the conflict happened (server-side session still open) and the
fatal message gives the correct recovery command (hermes gateway restart, not the stale hermes start).

hermes_cli/main.py _cmd_update_impl (manual gateway path)

After killing the old gateway (via SIGUSR1 drain or SIGTERM fallback),
wait up to 5 s for the OS-level process exit before the watcher
loop can spawn the new gateway. This reduces but does not guarantee
elimination of the Telegram 409 window, because Telegram's server-side
session expiry is independent of the local process lifecycle. The
Telegram adapter's retry logic handles any remaining 409s.

What this does NOT change

The systemd / launchd restart path is unaffected; those services have
their own RestartSec cooldown and the --replace handoff already
waits for the old process to exit.
drop_pending_updates remains False; messages queued while the
gateway was restarting are delivered once polling resumes.
No behaviour change when there is a genuine concurrent Hermes / OpenClaw
instance using the same token that still exhausts retries and sets a
fatal error, as before.

Testing
Manually reproduced on a Raspberry Pi 5 running Hermes with a manual
(non-systemd) gateway and Telegram enabled:

Start hermes gateway run in the background.
Run hermes update while a conversation is active.
Before: gateway reports connected but stops responding; logs show
a single 409 then silence.
After: gateway logs show retry attempts with increasing delays,
resumes polling on retry 1–2 in most cases, and is fully responsive
within ~30 s of the update completing.

@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/gateway Gateway runner, session dispatch, delivery platform/telegram Telegram bot adapter labels May 14, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Related: This PR fixes #23783 and also addresses the failure mode described in #25221 (gateway stays alive with dead polling after conflict retry failure). Also competes with open PR #23806 (longer backoff) and #17713 (startup race conditions including 409 conflict).

@teknium1

Copy link
Copy Markdown
Contributor

Merged — cherry-picked your commit onto current main with authorship preserved (rebase-merge). 6/6 conflict tests passing after a stale test-assertion update (your PR bumped MAX_CONFLICT_RETRIES from 3→5; the test had hardcoded the old constant). Thanks for the careful two-stage diagnosis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P1 High — major feature broken, no workaround platform/telegram Telegram bot adapter type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Telegram bot token already in use after hermes update — gateway fails to start, Telegram broken afterwards

3 participants