Skip to content

fix(gateway): prevent infinite restart loop on session resume - #49243

Closed
JoaoMarcos44 wants to merge 2 commits into
NousResearch:mainfrom
JoaoMarcos44:fix/gateway-restart-loop
Closed

fix(gateway): prevent infinite restart loop on session resume#49243
JoaoMarcos44 wants to merge 2 commits into
NousResearch:mainfrom
JoaoMarcos44:fix/gateway-restart-loop

Conversation

@JoaoMarcos44

@JoaoMarcos44 JoaoMarcos44 commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Description

Fixes #49201

This PR resolves an issue where the gateway falls into an infinite boot-loop when resuming a session that was interrupted by a gateway restart or shutdown command.

Cause

When the gateway container is abruptly restarted, the process is killed. Upon reboot, the gateway auto-resumes the pending session. Because the transcript ends with a terminal command (e.g., ssh docker restart ... or reboot) with a [Result unavailable...] stub, the LLM assumes the command failed or got lost and re-runs it, triggering another restart/shutdown.

Solution

We modified the system note injection logic for resume-pending turns in gateway/run.py (_handle_message_with_agent):

  1. Empty User Message (Startup Turn): If the startup turn has an empty message, we instruct the model to report successful recovery to the user and ask for further instructions, rather than attempting to resume the last command.
  2. Restart Loop Prevention Warning: For all resume-pending cases, we inject a clear instruction:

    If the last action in the conversation history was a command to restart, stop, or shutdown the gateway or container, it has already executed successfully and the gateway is now back online; do NOT re-execute it and do NOT run any command to verify it. Do NOT re-execute old tool calls — skip any unfinished work from the conversation history.

This cleanly breaks the loop at the cognitive level of the model without modifying global database schemas or core CLI configurations.

Verification & Tests

  • Added a new unit test case test_resume_pending_empty_message_guidance to tests/gateway/test_restart_resume_pending.py to verify prompt injection and instructions on empty message recovery turns.
  • Verified that all 74 tests in tests/gateway/test_restart_resume_pending.py pass successfully.
  • Verified that related component test suites pass without regression:
    • tests/gateway/test_session.py (79/79 passed)
    • tests/gateway/test_restart_drain.py, test_restart_notification.py, test_restart_redelivery_dedup.py (55/55 passed)
    • tests/gateway/test_gateway_shutdown.py (17/17 passed)
    • tests/gateway/test_auto_continue.py (10/10 passed)

When a session resumes after a gateway restart or timeout, we inject a prompt warning to prevent the model from re-executing the command that triggered the restart. When the user message is empty (auto-resume startup turn), we instruct the model to report successful recovery and ask for further instructions, breaking the infinite restart loop.
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P1 High — major feature broken, no workaround labels Jun 19, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

Fix PR for #49201 (resume-after-restart reboot loop).

Related: #46963 (freshness gate on resume_pending — different mechanism), #46934 (stale resume_pending bypasses idle reset), #29728 (skip shutdown-timeout startup auto-resume), #43191 (docs FAQ for the restart loop).

This PR breaks the loop at the prompt level: the resume-pending system note now tells the model that a restart/stop/shutdown command in the history already executed successfully (gateway is back online) and must not be re-run or verified, plus dedicated empty-message (startup-turn) recovery guidance. Verified on main: the new wording is absent, so the old "A new message has arrived..." note is still live and the bug is unfixed — this is a genuine, non-no-op fix.

teknium1 pushed a commit that referenced this pull request Jun 20, 2026
When a tool call itself restarts the gateway (docker restart, systemctl
restart, and similar), the process is terminated mid-call — before the
tool result is persisted and before the orderly drain rewind can run. The
transcript tail is left as an assistant(tool_calls) with no matching tool
answer. On resume the model re-issues the unanswered call, taking the
gateway down again — an infinite loop (#49201).

Source fix: _build_gateway_agent_history now strips a trailing
assistant(tool_calls) block that has no tool answers
(_strip_dangling_tool_call_tail), so there is nothing for the model to
re-execute. This complements _strip_interrupted_tool_tails, which only
handles the case where a tool result row exists with an interrupt marker.

Cognitive backstop: the resume-pending system note now states that any
restart command in the history already ran and must not be re-executed or
verified, and the empty-message auto-resume startup turn reports recovery
and asks for instructions instead of the nonsensical "address the user's
NEW message" (there is no new message on that turn).

Reimplements the intent of #49243 by @JoaoMarcos44 at the replay layer.

Fixes #49201
@teknium1

Copy link
Copy Markdown
Contributor

Thanks @JoaoMarcos44 — merged via #49321 (commit 75ed07a), with your authorship preserved in git history.

We took your fix one layer deeper. Your PR addressed the loop at the prompt level (tell the model not to re-run the restart). #49321 keeps that as a cognitive backstop — including your empty-message startup-turn recovery wording — but adds a source-level fix: on resume, _build_gateway_agent_history now strips a trailing assistant(tool_calls) block that has no matching tool answer (the SIGKILL-mid-call signature). That removes the thing the model was reacting to entirely, so the loop can't form regardless of which restart command ran, and it doesn't leave an unanswered tool call on the wire for strict providers to reject.

Closing this in favor of the merged PR. Appreciate the clear repro and the right instinct on the fix.

@teknium1 teknium1 closed this Jun 20, 2026
waefrebeorn pushed a commit to waefrebeorn/slermes that referenced this pull request Jul 2, 2026
When a tool call itself restarts the gateway (docker restart, systemctl
restart, and similar), the process is terminated mid-call — before the
tool result is persisted and before the orderly drain rewind can run. The
transcript tail is left as an assistant(tool_calls) with no matching tool
answer. On resume the model re-issues the unanswered call, taking the
gateway down again — an infinite loop (NousResearch#49201).

Source fix: _build_gateway_agent_history now strips a trailing
assistant(tool_calls) block that has no tool answers
(_strip_dangling_tool_call_tail), so there is nothing for the model to
re-execute. This complements _strip_interrupted_tool_tails, which only
handles the case where a tool result row exists with an interrupt marker.

Cognitive backstop: the resume-pending system note now states that any
restart command in the history already ran and must not be re-executed or
verified, and the empty-message auto-resume startup turn reports recovery
and asks for instructions instead of the nonsensical "address the user's
NEW message" (there is no new message on that turn).

Reimplements the intent of NousResearch#49243 by @JoaoMarcos44 at the replay layer.

Fixes NousResearch#49201
habarmc1223-sudo pushed a commit to habarmc1223-sudo/hermes-agent-fluxmem that referenced this pull request Jul 8, 2026
When a tool call itself restarts the gateway (docker restart, systemctl
restart, and similar), the process is terminated mid-call — before the
tool result is persisted and before the orderly drain rewind can run. The
transcript tail is left as an assistant(tool_calls) with no matching tool
answer. On resume the model re-issues the unanswered call, taking the
gateway down again — an infinite loop (NousResearch#49201).

Source fix: _build_gateway_agent_history now strips a trailing
assistant(tool_calls) block that has no tool answers
(_strip_dangling_tool_call_tail), so there is nothing for the model to
re-execute. This complements _strip_interrupted_tool_tails, which only
handles the case where a tool result row exists with an interrupt marker.

Cognitive backstop: the resume-pending system note now states that any
restart command in the history already ran and must not be re-executed or
verified, and the empty-message auto-resume startup turn reports recovery
and asks for instructions instead of the nonsensical "address the user's
NEW message" (there is no new message on that turn).

Reimplements the intent of NousResearch#49243 by @JoaoMarcos44 at the replay layer.

Fixes NousResearch#49201
santhreal pushed a commit to santhreal/hermes-agent that referenced this pull request Jul 13, 2026
When a tool call itself restarts the gateway (docker restart, systemctl
restart, and similar), the process is terminated mid-call — before the
tool result is persisted and before the orderly drain rewind can run. The
transcript tail is left as an assistant(tool_calls) with no matching tool
answer. On resume the model re-issues the unanswered call, taking the
gateway down again — an infinite loop (NousResearch#49201).

Source fix: _build_gateway_agent_history now strips a trailing
assistant(tool_calls) block that has no tool answers
(_strip_dangling_tool_call_tail), so there is nothing for the model to
re-execute. This complements _strip_interrupted_tool_tails, which only
handles the case where a tool result row exists with an interrupt marker.

Cognitive backstop: the resume-pending system note now states that any
restart command in the history already ran and must not be re-executed or
verified, and the empty-message auto-resume startup turn reports recovery
and asks for instructions instead of the nonsensical "address the user's
NEW message" (there is no new message on that turn).

Reimplements the intent of NousResearch#49243 by @JoaoMarcos44 at the replay layer.

Fixes NousResearch#49201
Gravezzz pushed a commit to Gravezzz/hermes-agent that referenced this pull request Jul 21, 2026
When a tool call itself restarts the gateway (docker restart, systemctl
restart, and similar), the process is terminated mid-call — before the
tool result is persisted and before the orderly drain rewind can run. The
transcript tail is left as an assistant(tool_calls) with no matching tool
answer. On resume the model re-issues the unanswered call, taking the
gateway down again — an infinite loop (NousResearch#49201).

Source fix: _build_gateway_agent_history now strips a trailing
assistant(tool_calls) block that has no tool answers
(_strip_dangling_tool_call_tail), so there is nothing for the model to
re-execute. This complements _strip_interrupted_tool_tails, which only
handles the case where a tool result row exists with an interrupt marker.

Cognitive backstop: the resume-pending system note now states that any
restart command in the history already ran and must not be re-executed or
verified, and the empty-message auto-resume startup turn reports recovery
and asks for instructions instead of the nonsensical "address the user's
NEW message" (there is no new message on that turn).

Reimplements the intent of NousResearch#49243 by @JoaoMarcos44 at the replay layer.

Fixes NousResearch#49201
leewenjie pushed a commit to leewenjie/hermes-agent that referenced this pull request Aug 7, 2026
When a tool call itself restarts the gateway (docker restart, systemctl
restart, and similar), the process is terminated mid-call — before the
tool result is persisted and before the orderly drain rewind can run. The
transcript tail is left as an assistant(tool_calls) with no matching tool
answer. On resume the model re-issues the unanswered call, taking the
gateway down again — an infinite loop (NousResearch#49201).

Source fix: _build_gateway_agent_history now strips a trailing
assistant(tool_calls) block that has no tool answers
(_strip_dangling_tool_call_tail), so there is nothing for the model to
re-execute. This complements _strip_interrupted_tool_tails, which only
handles the case where a tool result row exists with an interrupt marker.

Cognitive backstop: the resume-pending system note now states that any
restart command in the history already ran and must not be re-executed or
verified, and the empty-message auto-resume startup turn reports recovery
and asks for instructions instead of the nonsensical "address the user's
NEW message" (there is no new message on that turn).

Reimplements the intent of NousResearch#49243 by @JoaoMarcos44 at the replay layer.

Fixes NousResearch#49201
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/gateway Gateway runner, session dispatch, delivery P1 High — major feature broken, no workaround type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Session resume after gateway restart causes reboot loop — interrupted tasks re-execute blindly

3 participants