Skip to content

fix: guard against stale /restart redelivery on gateway startup - #18528

Closed
dontcallmejames wants to merge 1 commit into
NousResearch:mainfrom
dontcallmejames:fix/startup-restart-guard
Closed

dontcallmejames wants to merge 1 commit into
NousResearch:mainfrom
dontcallmejames:fix/startup-restart-guard

Conversation

@dontcallmejames

Copy link
Copy Markdown
Contributor

Problem

When the .restart_last_processed.json dedup marker is missing (e.g. cleaned up manually or lost due to a filesystem issue), a stale /restart command re-delivered by Telegram's polling can trigger an infinite restart loop: each fresh gateway instance processes the re-delivered /restart and immediately restarts itself again. No Telegram messages are processed during this loop because the polling connection never stays alive long enough.

Hit this in production today — gateway had been restarting every ~2 minutes all day, zero messages processed since April 14.

Fix

Adds a startup-time guard inside _is_stale_restart_redelivery():

  • When the dedup marker file is missing AND the gateway process is less than 60 seconds old, treat the /restart as stale and return True
  • A legitimate /restart cannot arrive before the gateway has finished its boot sequence plus adapter handshake; anything arriving this early is a re-delivery from the previous gateway cycle

Changes

  1. Added self._startup_time: float = time.time() in GatewayRunner.__init__
  2. Added early-return guard: when marker missing + uptime < 60s → return True

The existing marker-file dedup mechanism is the right primary defense; this is belt-and-suspenders for when that file goes missing.

When the .restart_last_processed.json dedup marker is missing (e.g.
cleaned up manually or lost due to a filesystem issue), a stale
/restart command re-delivered by Telegram's polling can trigger an
infinite restart loop: each fresh gateway instance immediately
restarts itself again.

Add a startup-time guard: when the marker is missing and the gateway
process is less than 60 seconds old, treat the /restart as stale.
A legitimate /restart cannot arrive before the gateway has finished
its boot sequence plus adapter handshake; anything arriving this
early is a re-delivery from the previous gateway cycle.
@dontcallmejames
dontcallmejames force-pushed the fix/startup-restart-guard branch from 2f59d1a to 2b5042e Compare May 1, 2026 19:33
@alt-glitch alt-glitch added type/bug Something isn't working P1 High — major feature broken, no workaround comp/gateway Gateway runner, session dispatch, delivery platform/telegram Telegram bot adapter labels May 1, 2026
@krkln

krkln commented Jun 10, 2026

Copy link
Copy Markdown

We hit this exact bug in production (June 2026 — gateway restart loop, zero messages processed). Applied the same fix with a 10s cooldown instead of 60s — works. Our PR #43472 was marked as duplicate of this one. Thanks for the fix, hope it gets merged soon!

@teknium1 teknium1 added sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jun 29, 2026
teknium1 added a commit that referenced this pull request Jul 1, 2026
…ssing (#56107)

When .restart_last_processed.json goes missing, a redelivered /restart from
Telegram polling can no longer be caught by the update_id comparison, so it
re-restarts the gateway forever (issue #18528, reported by @dontcallmejames
who hit it in production — gateway restarting every ~2min, zero messages
processed).

Fallback: on marker-missing, suppress the /restart only when we can confirm
we just came out of a restart cycle (_booted_from_restart, captured at startup
from .restart_notify.json before it is unlinked) AND the process is still
within a 60s post-boot window. Consumed one-shot. This closes the loop without
swallowing a genuine first /restart on a fresh boot — the flaw in the original
bare-uptime approach.

Credit to @dontcallmejames for the diagnosis and original patch.
@teknium1

teknium1 commented Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Merged via #56107 (commit cdd5539 on main). Thanks for the diagnosis and the original patch — you correctly identified the root cause (missing dedup marker → redelivered /restart bypasses the update_id check → infinite restart loop) and hitting it in production made the failure mode concrete.

The guard was reworked in the salvage: instead of a bare "uptime < 60s → stale" window, it now suppresses the /restart only when it can independently confirm the process just came out of a restart cycle (_booted_from_restart, captured at startup from .restart_notify.json before it's unlinked) and is still within the 60s post-boot window, consumed one-shot. This closes the loop without the side effect the bare-uptime version had — the marker is missing on every fresh install / first-ever restart, so a blanket timer would silently swallow a legitimate first /restart a user sends right after boot.

Your name is in the merge commit and PR body. Appreciate the fix.

@teknium1 teknium1 closed this Jul 1, 2026
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
…ssing (NousResearch#56107)

When .restart_last_processed.json goes missing, a redelivered /restart from
Telegram polling can no longer be caught by the update_id comparison, so it
re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames
who hit it in production — gateway restarting every ~2min, zero messages
processed).

Fallback: on marker-missing, suppress the /restart only when we can confirm
we just came out of a restart cycle (_booted_from_restart, captured at startup
from .restart_notify.json before it is unlinked) AND the process is still
within a 60s post-boot window. Consumed one-shot. This closes the loop without
swallowing a genuine first /restart on a fresh boot — the flaw in the original
bare-uptime approach.

Credit to @dontcallmejames for the diagnosis and original patch.
Jasper6439 pushed a commit to Jasper6439/hermes-agent that referenced this pull request Jul 5, 2026
…ssing (NousResearch#56107)

When .restart_last_processed.json goes missing, a redelivered /restart from
Telegram polling can no longer be caught by the update_id comparison, so it
re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames
who hit it in production — gateway restarting every ~2min, zero messages
processed).

Fallback: on marker-missing, suppress the /restart only when we can confirm
we just came out of a restart cycle (_booted_from_restart, captured at startup
from .restart_notify.json before it is unlinked) AND the process is still
within a 60s post-boot window. Consumed one-shot. This closes the loop without
swallowing a genuine first /restart on a fresh boot — the flaw in the original
bare-uptime approach.

Credit to @dontcallmejames for the diagnosis and original patch.
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
…ssing (NousResearch#56107)

When .restart_last_processed.json goes missing, a redelivered /restart from
Telegram polling can no longer be caught by the update_id comparison, so it
re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames
who hit it in production — gateway restarting every ~2min, zero messages
processed).

Fallback: on marker-missing, suppress the /restart only when we can confirm
we just came out of a restart cycle (_booted_from_restart, captured at startup
from .restart_notify.json before it is unlinked) AND the process is still
within a 60s post-boot window. Consumed one-shot. This closes the loop without
swallowing a genuine first /restart on a fresh boot — the flaw in the original
bare-uptime approach.

Credit to @dontcallmejames for the diagnosis and original patch.
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
…ssing (NousResearch#56107)

When .restart_last_processed.json goes missing, a redelivered /restart from
Telegram polling can no longer be caught by the update_id comparison, so it
re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames
who hit it in production — gateway restarting every ~2min, zero messages
processed).

Fallback: on marker-missing, suppress the /restart only when we can confirm
we just came out of a restart cycle (_booted_from_restart, captured at startup
from .restart_notify.json before it is unlinked) AND the process is still
within a 60s post-boot window. Consumed one-shot. This closes the loop without
swallowing a genuine first /restart on a fresh boot — the flaw in the original
bare-uptime approach.

Credit to @dontcallmejames for the diagnosis and original patch.
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
…ssing (NousResearch#56107)

When .restart_last_processed.json goes missing, a redelivered /restart from
Telegram polling can no longer be caught by the update_id comparison, so it
re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames
who hit it in production — gateway restarting every ~2min, zero messages
processed).

Fallback: on marker-missing, suppress the /restart only when we can confirm
we just came out of a restart cycle (_booted_from_restart, captured at startup
from .restart_notify.json before it is unlinked) AND the process is still
within a 60s post-boot window. Consumed one-shot. This closes the loop without
swallowing a genuine first /restart on a fresh boot — the flaw in the original
bare-uptime approach.

Credit to @dontcallmejames for the diagnosis and original patch.
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
…ssing (NousResearch#56107)

When .restart_last_processed.json goes missing, a redelivered /restart from
Telegram polling can no longer be caught by the update_id comparison, so it
re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames
who hit it in production — gateway restarting every ~2min, zero messages
processed).

Fallback: on marker-missing, suppress the /restart only when we can confirm
we just came out of a restart cycle (_booted_from_restart, captured at startup
from .restart_notify.json before it is unlinked) AND the process is still
within a 60s post-boot window. Consumed one-shot. This closes the loop without
swallowing a genuine first /restart on a fresh boot — the flaw in the original
bare-uptime approach.

Credit to @dontcallmejames for the diagnosis and original patch.
melon-xf added a commit to melon-xf/hermes-agent that referenced this pull request Sep 3, 2026
…ssing (NousResearch#56107)

When .restart_last_processed.json goes missing, a redelivered /restart from
Telegram polling can no longer be caught by the update_id comparison, so it
re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames
who hit it in production — gateway restarting every ~2min, zero messages
processed).

Fallback: on marker-missing, suppress the /restart only when we can confirm
we just came out of a restart cycle (_booted_from_restart, captured at startup
from .restart_notify.json before it is unlinked) AND the process is still
within a 60s post-boot window. Consumed one-shot. This closes the loop without
swallowing a genuine first /restart on a fresh boot — the flaw in the original
bare-uptime approach.

Credit to @dontcallmejames for the diagnosis and original patch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P1 High — major feature broken, no workaround platform/telegram Telegram bot adapter sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants