Skip to content

fix(gateway): skip home-channel shutdown broadcast when idle - #57164

Open
Screddyice wants to merge 3 commits into
NousResearch:mainfrom
Screddyice:bug-fix/idle-shutdown-home-notification
Open

fix(gateway): skip home-channel shutdown broadcast when idle#57164
Screddyice wants to merge 3 commits into
NousResearch:mainfrom
Screddyice:bug-fix/idle-shutdown-home-notification

Conversation

@Screddyice

@Screddyice Screddyice commented Jul 2, 2026

Copy link
Copy Markdown

What does this PR do?

Skips the home-channel "⚠️ Gateway shutting down — Your current task will be interrupted." broadcast when an external stop or restart (systemd, cron auto-update) hits a gateway with no running agents.

_notify_active_sessions_of_shutdown notifies every configured home channel on every external shutdown, even when nothing is running. The warning is false in that case: there is no task to interrupt. Deployments that restart the gateway on a schedule get the ping on every cycle. #20103 and #29846 both report this; my two production gateways logged ten of these pings since June 16, all while idle, doubled because both gateways share one home chat.

The config opt-out that closed #20103 (gateway_restart_notification: false) silences the idle pings, but it also silences the interruption warning for chats with active work, the one message that carries real information. Users pick between spam and losing real warnings. This change drops only the idle external broadcast:

  • Idle external shutdown (systemd/cron): skipped. This is the new behavior.
  • Active sessions, any shutdown: affected chats and home channel still warned.
  • Gateway-initiated restarts (_restart_requested), even idle: still announced, so the home channel hears the "I'll be back" resume hint. test_restart_notifies_home_channel_even_without_active_sessions pins this and stays green.
  • In-chat /restart: unchanged, already suppressed (test_idle_in_chat_restart_does_not_send_interruption_warning).
  • Force-interrupt drains: unchanged, the drain-marker gate from feat(gateway): suppress home-channel shutdown broadcast on flagged drains #54824 covers them.

Prior art: #20126 proposed this same idle skip and its author closed it after it went stale unreviewed, inviting a fresh attempt. #20943 (open) goes further and stops all home-channel interruption broadcasts; this PR is narrower and keeps the broadcast whenever sessions are interrupted or the gateway restarts itself.

Related Issue

Refs #20103, #29846 (both closed via the config opt-out; this fixes the default behavior). Supersedes stale #20126.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • gateway/run.py: return early from the home-channel broadcast in _notify_active_sessions_of_shutdown when no agents are running and no restart was requested, after the in-chat-restart skip and before the drain-marker gate. Logs the skip at INFO.
  • tests/gateway/test_gateway_shutdown.py: add test_idle_external_shutdown_skips_home_channel_notification. It fails on main (the home channel receives the warning) and passes with this change.

How to Test

  1. scripts/run_tests.sh tests/gateway/test_gateway_shutdown.py tests/gateway/test_restart_notification.py tests/gateway/test_restart_resume_pending.py → 128 passed.
  2. Revert gateway/run.py only and rerun the new test: it fails with the home-chat send captured.
  3. Live on a systemd deployment: systemctl --user restart hermes-gateway while idle. The gateway logs "Skipping home-channel shutdown notification: no active sessions to interrupt" and sends nothing. Restart during a running task and the affected chat still gets the warning. Verified on two production gateways.

Full-suite note: scripts/run_tests.sh (per-file isolation, 36 workers, macOS) reports 45 failures in 14 files on this machine, in credential/platform-dependent areas (anthropic adapter, bedrock, WSL, service manager, live-system guard) plus one flaky concurrency test. I ran the same files against pristine upstream/main in the same environment: identical failures. Every file this PR touches or affects passes.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
  • I searched for existing PRs to make sure this isn't a duplicate
  • My PR contains only changes related to this fix/feature (no unrelated commits)
  • I've run the test suite via scripts/run_tests.sh (canonical per-file runner); all touched suites pass, remaining failures reproduce identically on pristine main (env-dependent, detailed above)
  • I've added tests for my changes (required for bug fixes, strongly encouraged for features)

_notify_active_sessions_of_shutdown pinged every configured home
channel with "Gateway shutting down — Your current task will be
interrupted." on every external stop or restart, even with zero
running agents. Scheduled systemd/cron restarts of an idle gateway
hit this on every cycle (NousResearch#20103, NousResearch#29846), and the warning is false
in that case: there is no task to interrupt.

Return early from the home-channel section when no agents are
running. Chats with active work keep their interruption warning,
in-chat /restart behavior is unchanged, and the drain-marker gate
from NousResearch#54824 still covers force-interrupt drains, which reach this
section with active sessions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages P3 Low — cosmetic, nice to have labels Jul 2, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Related to the home-channel shutdown-broadcast cluster: this PR explicitly supersedes the stale, self-closed #20126 (same idle-skip) and is narrower than the open #20943 (which drops all home-channel interruption broadcasts). It keeps the per-active-session interrupt ping and only skips the idle broadcast. Refs #20103/#29846 (closed via the gateway_restart_notification opt-out) — this fixes the default behavior instead. Maintainer picks between this narrow fix and #20943.

Scope the new idle skip to external shutdowns (restart_requested
False). Gateway-initiated restarts still announce to home channels
even when idle — test_restart_notifies_home_channel_even_without_
active_sessions pins that behavior so the home channel hears the
"I'll be back" resume hint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Screddyice
Screddyice marked this pull request as ready for review July 2, 2026 17:41
@teknium1 teknium1 added the sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform label Jul 15, 2026
@teknium1

Copy link
Copy Markdown
Contributor

Thanks for the narrowly scoped fix. The premise remains present on current main: gateway/run.py:5822 captures an empty active-agent snapshot, and absent an in-chat restart the method proceeds to the configured-home-channel send loop at gateway/run.py:5963-5990. The proposed not active and not self._restart_requested guard fixes that external idle-shutdown case.

The scope also preserves established behavior: tests/gateway/test_restart_resume_pending.py:1550-1564 requires an idle gateway-initiated restart to notify the home channel, and tests/gateway/test_restart_drain.py:579-606 requires unsuppressed drains with active work to retain both the active-chat and distinct-home notifications. The PR's guard does not alter either path.

Automated hermes-sweeper review.

Resolve the shutdown test insertion conflict and document idle gateway restart behavior.

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P3 Low — cosmetic, nice to have sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature Request: Add config option to suppress shutdown notifications when no active sessions exist

3 participants