Skip to content

fix(gateway): stop self-restart and resume replay loops - #47725

Closed
alanwilhelm wants to merge 3 commits into
NousResearch:mainfrom
alanwilhelm:codex/gateway-self-restart-guard
Closed

fix(gateway): stop self-restart and resume replay loops#47725
alanwilhelm wants to merge 3 commits into
NousResearch:mainfrom
alanwilhelm:codex/gateway-self-restart-guard

Conversation

@alanwilhelm

@alanwilhelm alanwilhelm commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes two gateway stability failures from the same local incident:

  1. A gateway-run agent could bypass the existing hermes gateway restart protection by shelling directly to the macOS launchd service:

    launchctl kickstart -k gui/501/ai.hermes.gateway

    That sent SIGTERM to the running gateway from inside an active turn.

  2. After the interrupted turn, the affected Discord channel stayed broken because the session remained resume_pending=True while replaying an oversized transcript. The local incident had ~329k recorded prompt tokens on a configured 400k context model, below the old 85% gateway hygiene threshold but above the agent's 50% compression threshold. Incomplete turns did not clear resume_pending, so every real user message re-entered the same poisoned context.

This PR now blocks gateway self-restart commands from gateway/API sessions before /yolo or approvals.mode=off, and adds a pre-agent restart-recovery reset gate: oversized resume_pending sessions are rotated to a fresh active session before any model call. The old transcript is preserved in session history; it is not replayed into the current turn.

Related overlap: existing open PRs #7817, #33084, and #33562 also cover launchd gateway lifecycle commands in DANGEROUS_PATTERNS. This PR is stricter in gateway context because it hard-blocks before yolo/off approvals and also covers execute_code snippets.

Related Issue

No public issue for the local incident. Related to the same launchd bypass class discussed in #7817, #33084, and #33562.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • ✨ New feature (non-breaking change that adds functionality)
  • 🔒 Security fix
  • 📝 Documentation update
  • ✅ Tests (adding or improving test coverage)
  • ♻️ Refactor (no behavior change)
  • 🎯 New skill (bundled or hub)

Changes Made

  • tools/approval.py — adds _check_gateway_self_restart_guard() and blocks Hermes gateway lifecycle commands from gateway/API sessions before yolo or disabled approvals can bypass the approval layer.
  • tools/approval.py — adds launchd ai.hermes.gateway lifecycle commands to DANGEROUS_PATTERNS for ordinary dangerous-command detection outside the hard-blocked gateway context.
  • tools/approval.py — applies the same gateway self-restart guard to check_execute_code_guard() so Python snippets containing launchd gateway lifecycle calls are denied from gateway sessions before whole-script approval bypasses.
  • gateway/run.py — adds a restart-recovery safety gate for resume_pending sessions whose transcript is already beyond the agent compression threshold, capped at 50% context. Those sessions are reset before agent startup, with session-boundary transient state cleared and Telegram topic bindings re-pointed when applicable.
  • tests/tools/test_approval.py — covers launchd Hermes gateway lifecycle command detection and isolates the find approval-key regression test from host-local permanent allowlists.
  • tests/tools/test_command_guards.py — covers terminal and execute_code gateway-session hard blocks, including the approvals.mode=off path.
  • tests/gateway/test_restart_resume_pending.py — covers the exact 329k prompt / 400k context incident shape and verifies the oversized resume_pending session rotates before agent startup.

How to Test

  1. From a gateway-context guard path, verify launchctl kickstart -k gui/501/ai.hermes.gateway returns approved: False before approval mode/yolo handling.

  2. Verify an execute_code snippet containing subprocess.run(['launchctl', 'kickstart', '-k', 'gui/501/ai.hermes.gateway']) is also blocked from a gateway session before whole-script approval bypasses.

  3. Verify a resume_pending session with last_prompt_tokens=329_227 and context_length=400_000 resets before agent startup instead of replaying the transcript.

  4. Run focused CI-parity wrapper tests:

    scripts/run_tests.sh tests/gateway/test_restart_resume_pending.py
    scripts/run_tests.sh tests/gateway/test_session_hygiene.py tests/tools/test_approval.py tests/tools/test_command_guards.py tests/hermes_cli/test_gateway_restart_loop.py
  5. Run syntax/bytecode validation:

    venv/bin/python -m compileall -q gateway/run.py tools/approval.py

Checklist

Code

Documentation & Housekeeping

  • I've updated relevant documentation (README, docs/, docstrings) — N/A; no user-facing docs changed
  • I've updated cli-config.yaml.example if I added/changed config keys — N/A
  • I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A
  • I've considered cross-platform impact (Windows, macOS) per the compatibility guide; launchd detection is macOS-specific, existing systemd detection remains unchanged, and the restart-recovery reset gate is platform-neutral
  • I've updated tool descriptions/schemas if I changed tool behavior — N/A; no model-facing tool schema changed

For New Skills

N/A.

Screenshots / Logs

Focused wrapper test results:

=== Summary: 1 files, 77 tests passed, 0 failed (100% complete) in 18.0s (24 workers) ===
=== Summary: 4 files, 306 tests passed, 0 failed (100% complete) in 5.9s (24 workers) ===

Compile checks:

venv/bin/python -m compileall -q gateway/run.py tools/approval.py

@alt-glitch alt-glitch added type/security Security vulnerability or hardening comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists labels Jun 17, 2026
@alanwilhelm
alanwilhelm force-pushed the codex/gateway-self-restart-guard branch from e6e8be8 to fb0c29f Compare June 17, 2026 08:02
@alanwilhelm alanwilhelm changed the title [codex] fix approval guard for gateway self-restarts fix(tools): block gateway self-restarts from gateway sessions Jun 17, 2026
@alanwilhelm
alanwilhelm force-pushed the codex/gateway-self-restart-guard branch from fb0c29f to 18ffb3b Compare June 17, 2026 08:45
@alanwilhelm alanwilhelm changed the title fix(tools): block gateway self-restarts from gateway sessions fix(gateway): stop self-restart and resume replay loops Jun 17, 2026
@alanwilhelm

Copy link
Copy Markdown
Contributor Author

Closing stale PR per cleanup (2026-07-07).

@alanwilhelm alanwilhelm closed this Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/tools Tool registry, model_tools, toolsets P2 Medium — degraded but workaround exists type/security Security vulnerability or hardening

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants