fix(gateway): prevent infinite restart loop on session resume - #49243
fix(gateway): prevent infinite restart loop on session resume#49243JoaoMarcos44 wants to merge 2 commits into
Conversation
When a session resumes after a gateway restart or timeout, we inject a prompt warning to prevent the model from re-executing the command that triggered the restart. When the user message is empty (auto-resume startup turn), we instruct the model to report successful recovery and ask for further instructions, breaking the infinite restart loop.
|
Fix PR for #49201 (resume-after-restart reboot loop). Related: #46963 (freshness gate on resume_pending — different mechanism), #46934 (stale resume_pending bypasses idle reset), #29728 (skip shutdown-timeout startup auto-resume), #43191 (docs FAQ for the restart loop). This PR breaks the loop at the prompt level: the resume-pending system note now tells the model that a restart/stop/shutdown command in the history already executed successfully (gateway is back online) and must not be re-run or verified, plus dedicated empty-message (startup-turn) recovery guidance. Verified on main: the new wording is absent, so the old "A new message has arrived..." note is still live and the bug is unfixed — this is a genuine, non-no-op fix. |
When a tool call itself restarts the gateway (docker restart, systemctl restart, and similar), the process is terminated mid-call — before the tool result is persisted and before the orderly drain rewind can run. The transcript tail is left as an assistant(tool_calls) with no matching tool answer. On resume the model re-issues the unanswered call, taking the gateway down again — an infinite loop (#49201). Source fix: _build_gateway_agent_history now strips a trailing assistant(tool_calls) block that has no tool answers (_strip_dangling_tool_call_tail), so there is nothing for the model to re-execute. This complements _strip_interrupted_tool_tails, which only handles the case where a tool result row exists with an interrupt marker. Cognitive backstop: the resume-pending system note now states that any restart command in the history already ran and must not be re-executed or verified, and the empty-message auto-resume startup turn reports recovery and asks for instructions instead of the nonsensical "address the user's NEW message" (there is no new message on that turn). Reimplements the intent of #49243 by @JoaoMarcos44 at the replay layer. Fixes #49201
|
Thanks @JoaoMarcos44 — merged via #49321 (commit 75ed07a), with your authorship preserved in git history. We took your fix one layer deeper. Your PR addressed the loop at the prompt level (tell the model not to re-run the restart). #49321 keeps that as a cognitive backstop — including your empty-message startup-turn recovery wording — but adds a source-level fix: on resume, Closing this in favor of the merged PR. Appreciate the clear repro and the right instinct on the fix. |
When a tool call itself restarts the gateway (docker restart, systemctl restart, and similar), the process is terminated mid-call — before the tool result is persisted and before the orderly drain rewind can run. The transcript tail is left as an assistant(tool_calls) with no matching tool answer. On resume the model re-issues the unanswered call, taking the gateway down again — an infinite loop (NousResearch#49201). Source fix: _build_gateway_agent_history now strips a trailing assistant(tool_calls) block that has no tool answers (_strip_dangling_tool_call_tail), so there is nothing for the model to re-execute. This complements _strip_interrupted_tool_tails, which only handles the case where a tool result row exists with an interrupt marker. Cognitive backstop: the resume-pending system note now states that any restart command in the history already ran and must not be re-executed or verified, and the empty-message auto-resume startup turn reports recovery and asks for instructions instead of the nonsensical "address the user's NEW message" (there is no new message on that turn). Reimplements the intent of NousResearch#49243 by @JoaoMarcos44 at the replay layer. Fixes NousResearch#49201
When a tool call itself restarts the gateway (docker restart, systemctl restart, and similar), the process is terminated mid-call — before the tool result is persisted and before the orderly drain rewind can run. The transcript tail is left as an assistant(tool_calls) with no matching tool answer. On resume the model re-issues the unanswered call, taking the gateway down again — an infinite loop (NousResearch#49201). Source fix: _build_gateway_agent_history now strips a trailing assistant(tool_calls) block that has no tool answers (_strip_dangling_tool_call_tail), so there is nothing for the model to re-execute. This complements _strip_interrupted_tool_tails, which only handles the case where a tool result row exists with an interrupt marker. Cognitive backstop: the resume-pending system note now states that any restart command in the history already ran and must not be re-executed or verified, and the empty-message auto-resume startup turn reports recovery and asks for instructions instead of the nonsensical "address the user's NEW message" (there is no new message on that turn). Reimplements the intent of NousResearch#49243 by @JoaoMarcos44 at the replay layer. Fixes NousResearch#49201
When a tool call itself restarts the gateway (docker restart, systemctl restart, and similar), the process is terminated mid-call — before the tool result is persisted and before the orderly drain rewind can run. The transcript tail is left as an assistant(tool_calls) with no matching tool answer. On resume the model re-issues the unanswered call, taking the gateway down again — an infinite loop (NousResearch#49201). Source fix: _build_gateway_agent_history now strips a trailing assistant(tool_calls) block that has no tool answers (_strip_dangling_tool_call_tail), so there is nothing for the model to re-execute. This complements _strip_interrupted_tool_tails, which only handles the case where a tool result row exists with an interrupt marker. Cognitive backstop: the resume-pending system note now states that any restart command in the history already ran and must not be re-executed or verified, and the empty-message auto-resume startup turn reports recovery and asks for instructions instead of the nonsensical "address the user's NEW message" (there is no new message on that turn). Reimplements the intent of NousResearch#49243 by @JoaoMarcos44 at the replay layer. Fixes NousResearch#49201
When a tool call itself restarts the gateway (docker restart, systemctl restart, and similar), the process is terminated mid-call — before the tool result is persisted and before the orderly drain rewind can run. The transcript tail is left as an assistant(tool_calls) with no matching tool answer. On resume the model re-issues the unanswered call, taking the gateway down again — an infinite loop (NousResearch#49201). Source fix: _build_gateway_agent_history now strips a trailing assistant(tool_calls) block that has no tool answers (_strip_dangling_tool_call_tail), so there is nothing for the model to re-execute. This complements _strip_interrupted_tool_tails, which only handles the case where a tool result row exists with an interrupt marker. Cognitive backstop: the resume-pending system note now states that any restart command in the history already ran and must not be re-executed or verified, and the empty-message auto-resume startup turn reports recovery and asks for instructions instead of the nonsensical "address the user's NEW message" (there is no new message on that turn). Reimplements the intent of NousResearch#49243 by @JoaoMarcos44 at the replay layer. Fixes NousResearch#49201
When a tool call itself restarts the gateway (docker restart, systemctl restart, and similar), the process is terminated mid-call — before the tool result is persisted and before the orderly drain rewind can run. The transcript tail is left as an assistant(tool_calls) with no matching tool answer. On resume the model re-issues the unanswered call, taking the gateway down again — an infinite loop (NousResearch#49201). Source fix: _build_gateway_agent_history now strips a trailing assistant(tool_calls) block that has no tool answers (_strip_dangling_tool_call_tail), so there is nothing for the model to re-execute. This complements _strip_interrupted_tool_tails, which only handles the case where a tool result row exists with an interrupt marker. Cognitive backstop: the resume-pending system note now states that any restart command in the history already ran and must not be re-executed or verified, and the empty-message auto-resume startup turn reports recovery and asks for instructions instead of the nonsensical "address the user's NEW message" (there is no new message on that turn). Reimplements the intent of NousResearch#49243 by @JoaoMarcos44 at the replay layer. Fixes NousResearch#49201
Description
Fixes #49201
This PR resolves an issue where the gateway falls into an infinite boot-loop when resuming a session that was interrupted by a gateway restart or shutdown command.
Cause
When the gateway container is abruptly restarted, the process is killed. Upon reboot, the gateway auto-resumes the pending session. Because the transcript ends with a terminal command (e.g.,
ssh docker restart ...orreboot) with a[Result unavailable...]stub, the LLM assumes the command failed or got lost and re-runs it, triggering another restart/shutdown.Solution
We modified the system note injection logic for resume-pending turns in
gateway/run.py(_handle_message_with_agent):This cleanly breaks the loop at the cognitive level of the model without modifying global database schemas or core CLI configurations.
Verification & Tests
test_resume_pending_empty_message_guidancetotests/gateway/test_restart_resume_pending.pyto verify prompt injection and instructions on empty message recovery turns.tests/gateway/test_restart_resume_pending.pypass successfully.tests/gateway/test_session.py(79/79 passed)tests/gateway/test_restart_drain.py,test_restart_notification.py,test_restart_redelivery_dedup.py(55/55 passed)tests/gateway/test_gateway_shutdown.py(17/17 passed)tests/gateway/test_auto_continue.py(10/10 passed)