Skip to content

fix(gateway): retry Telegram 409 polling conflicts before giving up - #2312

Merged
teknium1 merged 1 commit into
mainfrom
hermes/hermes-31d7db3b
Mar 21, 2026
Merged

fix(gateway): retry Telegram 409 polling conflicts before giving up#2312
teknium1 merged 1 commit into
mainfrom
hermes/hermes-31d7db3b

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

Cherry-pick of #2297 by @robbyczgw-cla. Closes #2296.

A single Telegram 409 Conflict error was permanently killing Telegram polling with no recovery. Transient 409s are expected during hermes gateway restart or systemctl restart — the old long-poll session may linger on Telegram's servers for a few seconds.

Changes

  • Retry up to 3 times with 10s delay before marking as fatal
  • On each retry: stop the updater, sleep, then explicitly restart polling (clean reconnection)
  • Reset counter on successful recovery so future transient 409s also get retries
  • Store error callback ref so retries can re-register it with start_polling()
  • After 3 failed retries, mark fatal as before (genuine dual-instance conflict)

Tests

  • test_polling_conflict_retries_before_fatal — single 409 triggers retry, not fatal
  • test_polling_conflict_becomes_fatal_after_retries — exhausted retries → fatal

All 1272 gateway tests pass (5 conflict tests pass).

A single Telegram 409 Conflict from getUpdates permanently killed
Telegram polling with no recovery possible (retryable=False on
first occurrence).  This is too aggressive for production use with
process supervisors.

Transient 409s are expected during:
- --replace handoffs where the old long-poll session lingers on
  Telegram servers for a few seconds after SIGTERM
- systemd Restart=on-failure respawns that overlap with the dying
  instance cleanup

Now _handle_polling_conflict() retries up to 3 times with a
10-second delay between attempts.  The 30-second total retry window
lets stale server-side sessions expire.  If all retries fail, the
error is still marked as permanently fatal — preserving the original
protection against genuine dual-instance conflicts.

Tests updated: split the single conflict test into two — one verifying
retry on transient conflict, one verifying fatal after exhausted
retries.

Closes #2296
@teknium1
teknium1 merged commit 453f4c5 into main Mar 21, 2026
1 check passed
angelburgosrosado pushed a commit to angelburgosrosado/hermes-agent that referenced this pull request Apr 27, 2026
…31d7db3b

fix(gateway): retry Telegram 409 polling conflicts before giving up
02356abc pushed a commit to 02356abc/hermes-agent that referenced this pull request May 14, 2026
…31d7db3b

fix(gateway): retry Telegram 409 polling conflicts before giving up
gweeteve pushed a commit to gweeteve/hermes-agent that referenced this pull request Jun 2, 2026
…31d7db3b

fix(gateway): retry Telegram 409 polling conflicts before giving up
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Telegram 409 polling conflict treated as permanently fatal — no retry

1 participant