Skip to content

fix(gateway): Windows post-update respawn must mark itself detached (silent gateway death after in-app update) - #53128

Closed
iso2kx wants to merge 1 commit into
NousResearch:mainfrom
iso2kx:fix/windows-gateway-post-update-respawn-console-guard
Closed

fix(gateway): Windows post-update respawn must mark itself detached (silent gateway death after in-app update)#53128
iso2kx wants to merge 1 commit into
NousResearch:mainfrom
iso2kx:fix/windows-gateway-post-update-respawn-console-guard

Conversation

@iso2kx

@iso2kx iso2kx commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Summary

On Windows, the post-update gateway respawn watcher launched the new gateway without marking it as a detached service, so the respawned gateway would die silently the moment the spawning console went away — leaving the user with no messaging gateway after an in-app update until a manual restart.

Root cause

_spawn_gateway_restart_watcher (in hermes_cli/gateway.py) respawns the gateway after the old PID exits. On Windows it set the right detach creation flags, but it did not:

  • set HERMES_GATEWAY_DETACHED=1 in the child env, nor
  • redirect the child's stdin to DEVNULL.

_windows_gateway_should_absorb_console_controls() decides whether the gateway installs the console-control guard (SetConsoleCtrlHandler(NULL, TRUE) + SIGINT/SIGBREAK ignore). It keys off HERMES_GATEWAY_DETACHED, falling back to sys.stdin.isatty(). Because the respawn carried neither signal, the new gateway inherited the spawning console, classed itself interactive, and skipped the guard.

It then died the instant Windows broadcast CTRL_CLOSE_EVENT / CTRL_LOGOFF_EVENT when the parent console (the post-update desktop shell) exited — no shutdown log, no traceback.

The gateway exit diagnostics captured the fingerprint exactly: the dead post-update gateway came up with stdin_is_tty=true, absorb_windows_console_controls=false, whereas every healthy detached gateway (autostart .cmd / _spawn_detached) shows true.

Fix

Mirror the canonical detached launch path (gateway_windows._spawn_detached) inside the respawn watcher:

  • set HERMES_GATEWAY_DETACHED=1 in the child env, and
  • redirect child stdin to DEVNULL (belt-and-braces: even if the env marker were ever dropped, the severed stdin keeps the isatty() fallback from misclassifying it).

This covers both respawn paths, since profile gateways (launch_detached_profile_gateway_restart) and unmapped / Scheduled-Task gateways (launch_detached_gateway_restart_by_cmdline) share this watcher.

Tests

Adds TestGatewayRestartWatcher:

  • test_watcher_marks_respawn_detached_and_severs_stdin — asserts the watcher template sets the detached marker, severs stdin, and passes the env overlay to Popen (and that the template is valid Python).
  • test_by_cmdline_spawns_watcher — asserts the by-cmdline entry point spawns the watcher with the expected argv shape.
  • test_rejects_empty_inputs — guards the early-return validation.
pytest tests/hermes_cli/test_gateway.py -k "RestartWatcher or by_cmdline or detached_absorbs"
4 passed

No new failures introduced (the pre-existing Linux/Windows-only failures in this file — signal.SIGKILL, systemd-linger, Docker-root-guard — are unrelated and fail identically on a clean baseline).

The post-update gateway respawn watcher (_spawn_gateway_restart_watcher)
launched the new gateway without HERMES_GATEWAY_DETACHED=1 and without
severing stdin. On Windows, _windows_gateway_should_absorb_console_controls()
keys off that env marker (falling back to stdin.isatty()), so the respawned
gateway inherited the spawning console, classed itself interactive, and
SKIPPED SetConsoleCtrlHandler(NULL, TRUE).

It then died silently the instant Windows broadcast CTRL_CLOSE_EVENT /
CTRL_LOGOFF_EVENT when the parent console (the post-update desktop shell)
went away -- no shutdown log, no traceback. The exit diagnostics confirmed
the dead gateway came up with stdin_is_tty=true, absorb_windows_console_controls=false,
while every healthy detached gateway shows true.

Fix: mirror gateway_windows._spawn_detached in the respawn watcher -- set
HERMES_GATEWAY_DETACHED=1 in the child env and redirect stdin to DEVNULL so
the respawn installs the console-control guard and survives the desktop
console teardown. Covers both respawn paths (profile and unmapped/scheduled
-task gateways share this watcher).

Adds regression tests asserting the watcher template carries the detached
marker and severed stdin.
@iso2kx

iso2kx commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Closing as a duplicate of #52239, which already sets HERMES_GATEWAY_DETACHED=1 in the post-update respawn watcher's env overlay — that's the core fix for the silent gateway death (the missing detached marker meant the respawn skipped the SetConsoleCtrlHandler guard and died on CTRL_CLOSE/LOGOFF). #52239 is broader (also fixes the persistent windowless-console issue via pythonw.exe) and predates this PR.

The only additive bit here was redirecting the respawn's stdin to DEVNULL as belt-and-braces, in case the env marker is ever dropped. Not worth a separate PR; deferring to #52239.

@iso2kx iso2kx closed this Jun 26, 2026
@iso2kx
iso2kx deleted the fix/windows-gateway-post-update-respawn-console-guard branch June 26, 2026 15:28
@alt-glitch alt-glitch added type/bug Something isn't working comp/gateway Gateway runner, session dispatch, delivery platform/windows Native Windows-specific behavior or breakage P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/windows Native Windows-specific behavior or breakage sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants