Skip to content

fix(gateway): retry detached restart watcher without breakaway (salvage #42993) - #70262

Merged
teknium1 merged 1 commit into
mainfrom
salvage/42993-watcher-breakaway-retry
Jul 24, 2026
Merged

fix(gateway): retry detached restart watcher without breakaway (salvage #42993)#70262
teknium1 merged 1 commit into
mainfrom
salvage/42993-watcher-breakaway-retry

Conversation

@teknium1

Copy link
Copy Markdown
Contributor

Summary

Salvages PR #42993 by @VerbalChainsaw: the Windows /restart watcher spawn now retries once without CREATE_BREAKAWAY_FROM_JOB when a restrictive parent job object rejects the breakaway bit with ERROR_ACCESS_DENIED — previously the OSError killed the watcher and the gateway restart silently never happened (#42116).

Changes

  • gateway/run.py: try/except OSError around the outer watcher Popen, single retry with windows_detach_flags_without_breakaway(), path-safe warning on dual failure (interpreter basename + numeric error only). Mirrors the canonical fallback in gateway_windows._spawn_detached.
  • tests/tools/test_windows_native_support.py: 4 behavioral tests (retry-once, argv/env preservation, no-leak logging, happy-path single spawn).
  • contributors/emails/phixxation@gmail.com: contributor mapping.

Validation

Result
test_windows_native_support.py + test_restart_drain.py 95 passed, 0 failed
A/B (fix reverted) retry + dual-failure tests fail as expected
Base drift 0 behind origin/main

Authorship preserved via cherry-pick; rebase-merge. Duplicate #42242 (same root cause, first diagnosis by @worlldz) was closed with credit — its branch was stale/mixed and hard-coded the now-banned DETACHED_PROCESS.

Closes #42993. Fixes #42116.

Infographic

watcher-breakaway-retry

@github-actions

github-actions Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

ran on 1255932

ℹ️ Info

Desktop E2E visual evidence · View test artifacts · View job

1 visual diff.

inline evidence upload failed.

Failed to upload diff-665a0833239e-onboarding-overlay-diff.png with gh image (exit code 1): Error uploading /home/runner/work/_temp/e2e-evidence/diff-665a0833239e-onboarding-overlay-diff.png: step 0 (get upload token): uploadToken not found on repo page — do you have write access to NousResearch/hermes-agent? (or, if NousResearch enforces SAML SSO, authorize at https://github.com/orgs/NousResearch/sso)

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery platform/windows Native Windows-specific behavior or breakage sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows labels Jul 23, 2026
The Windows /restart watcher's outer Popen spawns the watcher with
windows_detach_popen_kwargs() (which carries CREATE_BREAKAWAY_FROM_JOB),
but a restrictive parent job object can reject that bit with OSError and
the current call has no retry. Preserve the current watcher
implementation and add a focused breakaway-denied fallback.

Preserved from current main: watcher_python / pythonw.exe selection, the
str(restart_after_s) deadline, the scrubbed watcher_env, the intentional
no-breakaway inline respawn, and the entire POSIX setsid/bash path.

- primary keeps **windows_detach_popen_kwargs()
- on OSError, retry the same argv/env with
  creationflags=windows_detach_flags_without_breakaway()
- on dual failure, log a definitive, path-safe warning (interpreter
  basename + numeric winerror/errno only) and return without crashing

Replace the superseded breakaway-first inline design and its AST tests
with focused behavioral coverage that drives the real coroutine with a
mocked subprocess.Popen (retry, argv/env/DEVNULL preservation, POSIX
single-session kwarg, no-breakaway inline respawn, secret-safe logging).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@teknium1
teknium1 force-pushed the salvage/42993-watcher-breakaway-retry branch from 64d3c60 to 1255932 Compare July 23, 2026 23:24
@teknium1
teknium1 merged commit 30bb555 into main Jul 24, 2026
37 checks passed
@teknium1
teknium1 deleted the salvage/42993-watcher-breakaway-retry branch July 24, 2026 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists platform/windows Native Windows-specific behavior or breakage sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(gateway/run): Windows detached gateway /restart fails in job objects (WinError 5)

3 participants