fix: guard against stale /restart redelivery on gateway startup - #18528
dontcallmejames wants to merge 1 commit into
Conversation
When the .restart_last_processed.json dedup marker is missing (e.g. cleaned up manually or lost due to a filesystem issue), a stale /restart command re-delivered by Telegram's polling can trigger an infinite restart loop: each fresh gateway instance immediately restarts itself again. Add a startup-time guard: when the marker is missing and the gateway process is less than 60 seconds old, treat the /restart as stale. A legitimate /restart cannot arrive before the gateway has finished its boot sequence plus adapter handshake; anything arriving this early is a re-delivery from the previous gateway cycle.
2f59d1a to
2b5042e
Compare
|
We hit this exact bug in production (June 2026 — gateway restart loop, zero messages processed). Applied the same fix with a 10s cooldown instead of 60s — works. Our PR #43472 was marked as duplicate of this one. Thanks for the fix, hope it gets merged soon! |
…ssing (#56107) When .restart_last_processed.json goes missing, a redelivered /restart from Telegram polling can no longer be caught by the update_id comparison, so it re-restarts the gateway forever (issue #18528, reported by @dontcallmejames who hit it in production — gateway restarting every ~2min, zero messages processed). Fallback: on marker-missing, suppress the /restart only when we can confirm we just came out of a restart cycle (_booted_from_restart, captured at startup from .restart_notify.json before it is unlinked) AND the process is still within a 60s post-boot window. Consumed one-shot. This closes the loop without swallowing a genuine first /restart on a fresh boot — the flaw in the original bare-uptime approach. Credit to @dontcallmejames for the diagnosis and original patch.
|
Merged via #56107 (commit cdd5539 on main). Thanks for the diagnosis and the original patch — you correctly identified the root cause (missing dedup marker → redelivered The guard was reworked in the salvage: instead of a bare "uptime < 60s → stale" window, it now suppresses the Your name is in the merge commit and PR body. Appreciate the fix. |
…ssing (NousResearch#56107) When .restart_last_processed.json goes missing, a redelivered /restart from Telegram polling can no longer be caught by the update_id comparison, so it re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames who hit it in production — gateway restarting every ~2min, zero messages processed). Fallback: on marker-missing, suppress the /restart only when we can confirm we just came out of a restart cycle (_booted_from_restart, captured at startup from .restart_notify.json before it is unlinked) AND the process is still within a 60s post-boot window. Consumed one-shot. This closes the loop without swallowing a genuine first /restart on a fresh boot — the flaw in the original bare-uptime approach. Credit to @dontcallmejames for the diagnosis and original patch.
…ssing (NousResearch#56107) When .restart_last_processed.json goes missing, a redelivered /restart from Telegram polling can no longer be caught by the update_id comparison, so it re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames who hit it in production — gateway restarting every ~2min, zero messages processed). Fallback: on marker-missing, suppress the /restart only when we can confirm we just came out of a restart cycle (_booted_from_restart, captured at startup from .restart_notify.json before it is unlinked) AND the process is still within a 60s post-boot window. Consumed one-shot. This closes the loop without swallowing a genuine first /restart on a fresh boot — the flaw in the original bare-uptime approach. Credit to @dontcallmejames for the diagnosis and original patch.
…ssing (NousResearch#56107) When .restart_last_processed.json goes missing, a redelivered /restart from Telegram polling can no longer be caught by the update_id comparison, so it re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames who hit it in production — gateway restarting every ~2min, zero messages processed). Fallback: on marker-missing, suppress the /restart only when we can confirm we just came out of a restart cycle (_booted_from_restart, captured at startup from .restart_notify.json before it is unlinked) AND the process is still within a 60s post-boot window. Consumed one-shot. This closes the loop without swallowing a genuine first /restart on a fresh boot — the flaw in the original bare-uptime approach. Credit to @dontcallmejames for the diagnosis and original patch.
…ssing (NousResearch#56107) When .restart_last_processed.json goes missing, a redelivered /restart from Telegram polling can no longer be caught by the update_id comparison, so it re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames who hit it in production — gateway restarting every ~2min, zero messages processed). Fallback: on marker-missing, suppress the /restart only when we can confirm we just came out of a restart cycle (_booted_from_restart, captured at startup from .restart_notify.json before it is unlinked) AND the process is still within a 60s post-boot window. Consumed one-shot. This closes the loop without swallowing a genuine first /restart on a fresh boot — the flaw in the original bare-uptime approach. Credit to @dontcallmejames for the diagnosis and original patch.
…ssing (NousResearch#56107) When .restart_last_processed.json goes missing, a redelivered /restart from Telegram polling can no longer be caught by the update_id comparison, so it re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames who hit it in production — gateway restarting every ~2min, zero messages processed). Fallback: on marker-missing, suppress the /restart only when we can confirm we just came out of a restart cycle (_booted_from_restart, captured at startup from .restart_notify.json before it is unlinked) AND the process is still within a 60s post-boot window. Consumed one-shot. This closes the loop without swallowing a genuine first /restart on a fresh boot — the flaw in the original bare-uptime approach. Credit to @dontcallmejames for the diagnosis and original patch.
…ssing (NousResearch#56107) When .restart_last_processed.json goes missing, a redelivered /restart from Telegram polling can no longer be caught by the update_id comparison, so it re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames who hit it in production — gateway restarting every ~2min, zero messages processed). Fallback: on marker-missing, suppress the /restart only when we can confirm we just came out of a restart cycle (_booted_from_restart, captured at startup from .restart_notify.json before it is unlinked) AND the process is still within a 60s post-boot window. Consumed one-shot. This closes the loop without swallowing a genuine first /restart on a fresh boot — the flaw in the original bare-uptime approach. Credit to @dontcallmejames for the diagnosis and original patch.
…ssing (NousResearch#56107) When .restart_last_processed.json goes missing, a redelivered /restart from Telegram polling can no longer be caught by the update_id comparison, so it re-restarts the gateway forever (issue NousResearch#18528, reported by @dontcallmejames who hit it in production — gateway restarting every ~2min, zero messages processed). Fallback: on marker-missing, suppress the /restart only when we can confirm we just came out of a restart cycle (_booted_from_restart, captured at startup from .restart_notify.json before it is unlinked) AND the process is still within a 60s post-boot window. Consumed one-shot. This closes the loop without swallowing a genuine first /restart on a fresh boot — the flaw in the original bare-uptime approach. Credit to @dontcallmejames for the diagnosis and original patch.
Problem
When the
.restart_last_processed.jsondedup marker is missing (e.g. cleaned up manually or lost due to a filesystem issue), a stale/restartcommand re-delivered by Telegram's polling can trigger an infinite restart loop: each fresh gateway instance processes the re-delivered/restartand immediately restarts itself again. No Telegram messages are processed during this loop because the polling connection never stays alive long enough.Hit this in production today — gateway had been restarting every ~2 minutes all day, zero messages processed since April 14.
Fix
Adds a startup-time guard inside
_is_stale_restart_redelivery():/restartas stale and returnTrue/restartcannot arrive before the gateway has finished its boot sequence plus adapter handshake; anything arriving this early is a re-delivery from the previous gateway cycleChanges
self._startup_time: float = time.time()inGatewayRunner.__init__TrueThe existing marker-file dedup mechanism is the right primary defense; this is belt-and-suspenders for when that file goes missing.