Skip to content

docs: add gateway restart loop troubleshooting to FAQ - #43191

Closed
otakai wants to merge 1 commit into
NousResearch:mainfrom
otakai:docs/gateway-restart-loop
Closed

docs: add gateway restart loop troubleshooting to FAQ#43191
otakai wants to merge 1 commit into
NousResearch:mainfrom
otakai:docs/gateway-restart-loop

Conversation

@otakai

@otakai otakai commented Jun 10, 2026

Copy link
Copy Markdown

Summary

Adds a troubleshooting section to the FAQ documenting the gateway infinite restart loop caused by auto-resume of sessions that executed systemctl --user stop/restart hermes-gateway from inside the gateway process.

What is documented

  • Root cause: Sessions with resume_pending=true re-execute systemctl stop hermes-gateway on every restart, creating an infinite loop.
  • Fix: Clear resume_pending in sessions.json + reset systemd.
  • Prevention: A ~/.local/bin/systemctl wrapper that blocks stop/restart hermes-gateway when _HERMES_GATEWAY=1 is set (the gateway sets this env var).
  • Related config: agent.restart_drain_timeout recommendation.

Note that hermes gateway restart and hermes gateway stop already detect _HERMES_GATEWAY=1 and refuse to run from inside the gateway process, but systemctl --user stop/restart hermes-gateway bypasses this guard. The documented wrapper fills that gap.

Documents the infinite restart loop caused by auto-resume of sessions
that executed 'systemctl --user stop/restart hermes-gateway' from inside
the gateway process. Includes root cause analysis, step-by-step fix,
wrapper script to block systemctl inside the gateway, and restart_drain_timeout recommendation.
@alt-glitch alt-glitch added type/docs Documentation improvements P3 Low — cosmetic, nice to have comp/gateway Gateway runner, session dispatch, delivery labels Jun 10, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for documenting a real failure mode.

Automated hermes-sweeper review found that current main fixes this at the source and closes the documented bypass:

  • 75ed07ace (fix(gateway): break the restart loop at the source on session resume) sanitizes an unanswered trailing tool call before replay (gateway/run.py:900), with regression coverage in tests/gateway/test_auto_continue.py:174.
  • tools/terminal_tool.py:2250 now blocks gateway lifecycle commands, including raw systemctl forms, whenever _HERMES_GATEWAY=1; the wrapper proposed here is no longer needed.
  • The source fix shipped in v2026.7.1.

Closing as implemented on main.

@teknium1 teknium1 closed this Jul 14, 2026
@teknium1 teknium1 added the sweeper:implemented-on-main Sweeper: behavior already present on current main label Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:implemented-on-main Sweeper: behavior already present on current main type/docs Documentation improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants