fix(telegram): recover from post-update polling conflict without entering limbo - #25630
Closed
CryptoByz wants to merge 1 commit into
Closed
fix(telegram): recover from post-update polling conflict without entering limbo#25630CryptoByz wants to merge 1 commit into
CryptoByz wants to merge 1 commit into
Conversation
Collaborator
Closed
3 tasks
This was referenced May 19, 2026
Contributor
|
Merged — cherry-picked your commit onto current main with authorship preserved (rebase-merge). 6/6 conflict tests passing after a stale test-assertion update (your PR bumped MAX_CONFLICT_RETRIES from 3→5; the test had hardcoded the old constant). Thanks for the careful two-stage diagnosis. |
This was referenced Aug 3, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(telegram): recover from post-update polling conflict without entering limbo
Fixes #23783
Problem
Running hermes update while the gateway is active leaves Telegram broken
in two distinct ways.
Stage 1, 409 Conflict on startup.
When the old gateway process exits, Telegram keeps its getUpdates
long-poll session open on its servers for up to ~30 s. The new gateway
starts immediately (the watcher loop has no built-in delay) and receives
a 409 Conflict from Telegram's API before that server-side session has
expired.
Stage 2, Silent limbo after a failed retry (the worse bug).
_handle_polling_conflict() attempts to recover by sleeping 10 s and
calling start_polling() again. If that second start_polling() call
raises (because the server-side session is still alive), the method
returns silently. At this point:
updater.running is False
No fatal error is set
The gateway process is alive and /status reports "connected"
No messages are received or sent
The adapter is in an undetectable limbo. The only recovery was to
manually kill the process and restart the gateway.
Root causes
_handle_polling_conflict after a failed retry, the method
returns without scheduling another attempt or setting a fatal error.
The adapter silently stops processing messages.
hermes update (manual gateways) after SIGTERM / SIGUSR1, the
watcher process respawns the new gateway with no delay, maximising
the chance of hitting the Telegram server-side 409 window.
Fix
gateway/platforms/telegram.py _handle_polling_conflict
Retry count: 3 → 5, giving up to ~175 s of total back-off time
(well beyond Telegram's ~30 s server-side expiry window).
Back-off delay now increases with each attempt (10 + attempt * 10
s: 15 s, 25 s, 35 s, 45 s, 55 s). A flat 10 s was not always enough
to clear the first 409; an increasing delay avoids hammering the API
while still recovering quickly on the common case.
Failed retries now schedule the next attempt via loop.create_task()
instead of returning silently. This closes the limbo window: the
adapter is always either recovering, running, or in a declared fatal
state.
Clearer log messages and fatal error text the warning now explains
why the conflict happened (server-side session still open) and the
fatal message gives the correct recovery command (hermes gateway restart, not the stale hermes start).
hermes_cli/main.py _cmd_update_impl (manual gateway path)
After killing the old gateway (via SIGUSR1 drain or SIGTERM fallback),
wait up to 5 s for the OS-level process exit before the watcher
loop can spawn the new gateway. This reduces but does not guarantee
elimination of the Telegram 409 window, because Telegram's server-side
session expiry is independent of the local process lifecycle. The
Telegram adapter's retry logic handles any remaining 409s.
What this does NOT change
The systemd / launchd restart path is unaffected; those services have
their own RestartSec cooldown and the --replace handoff already
waits for the old process to exit.
drop_pending_updates remains False; messages queued while the
gateway was restarting are delivered once polling resumes.
No behaviour change when there is a genuine concurrent Hermes / OpenClaw
instance using the same token that still exhausts retries and sets a
fatal error, as before.
Testing
Manually reproduced on a Raspberry Pi 5 running Hermes with a manual
(non-systemd) gateway and Telegram enabled:
Start hermes gateway run in the background.
Run hermes update while a conversation is active.
Before: gateway reports connected but stops responding; logs show
a single 409 then silence.
After: gateway logs show retry attempts with increasing delays,
resumes polling on retry 1–2 in most cases, and is fully responsive
within ~30 s of the update completing.