fix(gateway): retry detached restart watcher without breakaway

[VerbalChainsaw]

What does this PR do?

Repairs the Windows detached /restart watcher against current main.

The outer watcher spawn uses windows_detach_popen_kwargs(), which includes CREATE_BREAKAWAY_FROM_JOB. Restrictive parent job objects can reject that bit with OSError, but the current call has no retry. This PR preserves the current watcher implementation and adds a focused retry without the breakaway bit.

Current-main behavior preserved

• watcher_python / pythonw.exe selection
• str(restart_after_s) deadline argument
• scrubbed watcher_env, including removal of _HERMES_GATEWAY
• current inline respawn using windows_detach_flags_without_breakaway()
• the complete POSIX setsid / bash path

Change

1. Primary outer spawn remains **windows_detach_popen_kwargs().
2. On OSError, retry the same argv and environment with creationflags=windows_detach_flags_without_breakaway().
3. If both attempts fail, log a definitive path-safe warning containing only the interpreter basename and numeric winerror/errno, then return without crashing the CLI.

Tests

Added focused behavioral coverage that drives the real coroutine with mocked subprocess.Popen and verifies:

• first-call OSError triggers exactly one no-breakaway retry
• argv, deadline, interpreter selection, DEVNULL streams, and scrubbed environment are preserved
• the primary helper kwargs remain present on POSIX CI as well as Windows
• the inline respawn remains intentionally no-breakaway
• the happy path spawns once
• dual failure warns without leaking argv, watcher source, environment contents, exception text, or a deterministic secret sentinel

Verification

Native Windows, Python 3.11.9, pytest 9.0.3:

• 4 focused tests passed
• focused + existing watcher + adjacent restart suites: 98 passed
• full test_windows_native_support.py: 68 passed, 2 pre-existing Windows-host failures reproduced on pristine origin/main
• git apply --check against current origin/main: passed

[liuhao1024]

Code Review — Verification

Reviewed: Clean ✅

This PR adds to the inlined restart watcher's call in , with a proper fallback that retries without the breakaway bit. The pattern mirrors the canonical fallback in (PR #40909).

Strengths:

• The outer watcher Popen is also wrapped in with a nested fallback — catches the case where the parent job object itself refuses breakaway
• The final fallback logs a so the user has a signal (no silent failure)
• AST-based regression test () is exceptionally thorough — verifies the bit is set at runtime via with mocked , not just static text matching
• Tests cover 12+ stealth attack patterns (const-after-use, const-in-comment, lambda wrappers, etc.)

No issues found. The implementation is well-structured and the test coverage is comprehensive.

[teknium1]

Thanks for tracing the Windows job-object failure mode. The outer watcher fallback is still needed on current main: gateway/run.py:6278-6284 calls windows_detach_popen_kwargs() without an OSError retry, while hermes_cli/_subprocess_compat.py:156-183 provides the no-breakaway fallback.

Problems

• The gateway/run.py hunk predates the current watcher refactor. Current main now passes a restart deadline through the watcher (gateway/run.py:6207-6210), preserves a scrubbed environment (gateway/run.py:6251-6277), and may use pythonw.exe (gateway/run.py:6257-6269). The proposed argv at PR gateway/run.py:4285 would need a focused port rather than a direct cherry-pick.
• Current main intentionally uses windows_detach_flags_without_breakaway() for the inline respawn at gateway/run.py:6243-6247 (introduced by e7d2f0b93). The new AST tests requiring the former breakaway-first inline shape should not be carried forward.

Suggested changes

• Port only the outer Popen retry around current gateway/run.py:6278, preserving its argv and environment, then retry with windows_detach_flags_without_breakaway().
• Add a focused regression test for a first-call OSError on the current outer watcher spawn rather than source-shape assertions for the superseded inline implementation.

Automated hermes-sweeper review.

[teknium1]

Current main now passes watcher_python, restart_after_s, and the scrubbed watcher_env here (gateway/run.py:6251-6283). Port the retry wrapper around that current call; do not restore this older argv shape.

[teknium1]

This test hard-requires a breakaway-first inline respawn, but current main intentionally uses windows_detach_flags_without_breakaway() at gateway/run.py:6243-6247. Replace it with a focused behavioral test for the outer watcher retry on current main.

[VerbalChainsaw]

Thanks for the detailed steer — reconstructed against current main (46e87b14f) accordingly.

• Dropped the stale inline rewrite and old argv; the inline respawn keeps windows_detach_flags_without_breakaway() (e7d2f0b93) untouched.
• Ported only the outer watcher Popen OSError fallback around the current call, preserving watcher_python, str(restart_after_s), the scrubbed watcher_env, and pythonw.exe selection. Primary keeps **windows_detach_popen_kwargs(); the retry uses creationflags=windows_detach_flags_without_breakaway(). Mirrors _spawn_detached.
• On dual failure it doesn't claim success: a logger.warning stating the watcher was not started, with only the interpreter basename + numeric winerror/errno (no argv/env/secrets); the CLI still exits cleanly.
• Replaced the AST suite with a focused behavioral class (mocked Popen): first-call-OSError retry, argv/env/DEVNULL preservation, the breakaway split, the still-no-breakaway inline respawn, and secret-safe logging.
• Force-pushed a rebase onto current main since the branch predated the watcher refactor. Happy to adjust.

[teknium1]

Merged via PR #70262 — your commit was cherry-picked onto current main with authorship preserved in git log (rebase-merge). The focused retry, argv/env preservation, and behavioral tests all landed as you wrote them. Thanks for the clean fix and for correctly scoping it to the one missing retry site.