[IzzyWizzy666]

Summary

Windows hosts that wrap each spawned process in a Job Object without
JOB_OBJECT_LIMIT_BREAKAWAY_OK make CreateProcess fail outright with
ERROR_ACCESS_DENIED (WinError 5) when CREATE_BREAKAWAY_FROM_JOB is
requested — the flag is not merely ignored.

Windows Task Scheduler does exactly this for every task it starts, and a
Task Scheduler task is how a gateway normally comes back after an unattended
reboot. So three detached spawns could never start on the auto-start path, even
though the identical spawn works fine from a plain shell or the Desktop app.

Site	Failure
plugins/platforms/whatsapp/adapter.py — bridge	gateway starts and cron executes, but the bridge never connects; port 3000 never listens
gateway/slash_commands.py — /update	the gateway-side updater process dies
hermes_cli/web_server_gateway.py — dashboard actions	dashboard-spawned actions die

The asymmetry is the tell: the same code path works when the gateway is started
by hand, and fails only when it is started by the mechanism that exists to keep
it running.

Repro

Register a throwaway scheduled task with the same principal and have it attempt
both variants from inside a real Task Scheduler job:

WITH_BREAKAWAY    -> PermissionError: [WinError 5] Access is denied
WITHOUT_BREAKAWAY -> OK

Bridge symptom on a real install:

ERROR ... [Whatsapp] Failed to start bridge: [WinError 5] Access is denied
  File ".../plugins/platforms/whatsapp/adapter.py", line 477, in connect
    self._bridge_process = subprocess.Popen(
PermissionError: [WinError 5] Access is denied

retried every 60–300s, identical each time; curl 127.0.0.1:3000/health never
answers.

Fix

Add popen_detached_with_breakaway_fallback() to hermes_cli/_subprocess_compat.py
and route all three sites through it, mirroring the fallback that already exists
in gateway_windows._spawn_detached: retry once with
windows_detach_flags_without_breakaway(), which drops only that bit.

POSIX is unaffected — there is no breakaway concept there, so the original error
propagates unmasked.

hermes_cli/gateway.py::_spawn_detached_gateway is deliberately left alone: it
is only reachable from the macOS-26 launchd fallback, where
windows_detach_popen_kwargs() returns start_new_session and the breakaway
bit is irrelevant. Fixing an unreachable-on-Windows path would be noise.

Test Plan

• pytest across test_windows_native_support.py, test_gateway_windows.py,
  test_windows_gateway_job_teardown_48820.py, test_whatsapp_connect.py,
  test_75349_whatsapp_multiplex_secret_scope.py, test_whatsapp_bridge_pidfile.py,
  test_whatsapp_bridge_dir_resolution.py, test_whatsapp_from_owner.py
  → 93 passed, 5 skipped, 1 pre-existing failure

• New tests fail against the unfixed helper and pass against the fixed one
  (RED → GREEN, verified by reverting the retry branch)

• ruff check clean on all five changed files

• Live verification on a Windows install: a gateway restarted by its own
  scheduled task now logs

  WARNING ... Bridge spawn failed ([WinError 5] Access is denied); retrying without CREATE_BREAKAWAY_FROM_JOB
  INFO    ... ✓ whatsapp connected
  INFO    ... Gateway running with 1 platform(s)
  
  and `curl 127.0.0.1:3000/health` returns `{"status":"connected"}`
  

The single failure (TestWindowlessGatewayRestartSpec::test_noop_on_non_windows)
is unrelated and pre-existing on Windows: it asserts a POSIX default
(cwd == "") and receives a Windows temp directory. It fails identically with
these changes stashed, so it is not a regression from this PR.

[alt-glitch]

This was generated by AI during triage.

Related: #87777 covers the same WinError 5 breakaway retry for the WhatsApp bridge only; this PR generalizes it (shared helper in hermes_cli/_subprocess_compat.py) and also fixes the /update and dashboard spawn sites. Merged #40956 applied the same retry to the restart watcher. Maintainers may prefer this broader one and close #87777.

[denohk]

Confirming this still reproduces on current main (c6e0f2498e, v0.21.5, Windows 11 build 26200), and that the diagnosis in the PR body is exactly right.

Live evidence from a self-managed Windows install (today)

The gateway comes back through a Windows Scheduled Task (wscript //B //Nologo …\gateway-service\Hermes_Gateway.vbs → python.exe -m hermes_cli.main gateway run).

1. The Task-Scheduler-started child really is inside a job object:

   • gateway pid → IsProcessInJob = 1
   • a process started from an interactive shell on the same machine → 0

2. Reproduced the job semantics in isolation (own job object created with default limit flags → JOB_OBJECT_LIMIT_BREAKAWAY_OK unset, child assigned with AssignProcessToJobObject, then the child spawns node.exe):

   with_breakaway    (0x01000000 | 0x08000000 | 0x00000200) → PermissionError [WinError 5]
   without_breakaway (             0x08000000 | 0x00000200) → OK rc=0
   

   Same command line, same interpreter, same user — only the breakaway bit differs.

3. User-visible symptom on the auto-start path only, once per reconnect attempt (34 occurrences; the same install was fine while the gateway had been started from an interactive shell):

   ERROR hermes_plugins.platforms__whatsapp.adapter: [Whatsapp] Failed to start bridge: [WinError 5] 存取被拒。
     File ".../plugins/platforms/whatsapp/adapter.py", line 537, in connect
       self._bridge_process = subprocess.Popen(
         [node, str(bridge_path), "--port", ...], ..., **windows_detach_popen_kwargs())
   

4. Applying the same fallback locally — retry once with windows_detach_flags_without_breakaway() in plugins/platforms/whatsapp/adapter.py (connect) and gateway/slash_commands.py (the detached update spawn) — made the bridge start on the first attempt after a restart:

   ✓ telegram connected
   ✓ whatsapp connected
   ✓ weixin connected
   

The shared helper proposed here (popen_detached_with_breakaway_fallback) is the better shape than the two ad-hoc retries I used, and it also covers hermes_cli/web_server_gateway.py, which I had not hit yet.

Is anything blocking this from merging? Happy to test a build from the branch on the same Task-Scheduler-started setup (Windows 11, gateway auto-start, WhatsApp bridge + /update path) if that would help move it along.

[ericscho]

Independent Windows reproduction + a note on shape (shared helper instead of per-site try/except)

Environment: hermes-agent 0.21.5 (6590f13), Windows 11, Hermes-managed Node under
%LOCALAPPDATA%\hermes\tools\node-26.7.0-win32-x64, gateway started by the Hermes_Gateway
scheduled-task lineage (VBS -> .cmd -> python -m hermes_cli.main gateway run).

Observed: WhatsApp permanently "disconnected"; every reconnect attempt (~every 5 min) logs

hermes_plugins.platforms__whatsapp.adapter: [Whatsapp] Failed to start bridge: [WinError 5] Access is denied
... plugins/platforms/whatsapp/adapter.py:633, in connect
    subprocess.Popen([node, str(bridge_path), ...], stdout=bridge_log_fh, stderr=bridge_log_fh,
                     env=self._bridge_env(), **windows_detach_popen_kwargs())

Same failure class in the same lineage: hermes_cli.web_server: Failed to spawn backup at
hermes_cli/web_server_gateway.py:489 (_spawn_hermes_action, identical detach flags), plus
[Errno 13] Permission denied: '...\\.local\\state\\hermes\\gateway-locks\\host-serve.lock'.

Ruled out before blaming the job object:

• node resolution: find_node_executable() returns the PM-installed binary and never consults PATH;
• the daemon's cwd (read from its PEB): C:\Users\erics\AppData\Local\hermes - valid;
• elevation: parent and child both non-elevated; no integrity-level difference;
• env/handles: the identical call shape (bundled node + flags 0x09000200 + a file for
  stdout/stderr + a custom env) returns rc=0 when run from an interactive session on the same box,
  and hermes gateway restart from that session makes WhatsApp connect immediately.

That last point matches this patch's diagnosis: it is the inherited Job Object, not the call.
Also useful for anyone reproducing: _HERMES_GATEWAY_BREAKAWAY (written only by
gateway_windows._spawn_detached) was absent from the failing gateway's environment, i.e. that
gateway was started by a different path and inherited the restricting job. Do not rely on
IsProcessInJob(pid, NULL, ...) to detect this cross-process: it reports True very widely.

On shape - instead of each site growing its own try/except (gateway_windows._spawn_detached,
main_desktop._launch, hermes_cli/gateway.py:1176, run_shutdown.py:1480 all have one already), a
single helper next to windows_detach_popen_kwargs() can carry the retry and make every call site a
one-liner:

def popen_detached(argv: Sequence[str], **kwargs) -> subprocess.Popen:
    try:
        return subprocess.Popen(argv, **kwargs, **windows_detach_popen_kwargs())
    except OSError as exc:
        winerror = getattr(exc, "winerror", None)
        if winerror is None:
            winerror = exc.errno
        if not IS_WINDOWS or winerror != 5:
            raise
        logger.warning("Detached spawn denied by a Job Object that forbids breakaway; "
                       "retrying without CREATE_BREAKAWAY_FROM_JOB")
        return subprocess.Popen(argv, **kwargs,
                                creationflags=windows_detach_flags_without_breakaway())

Verified locally against a stubbed Popen that raises PermissionError(winerror=5) when the flags
contain 0x01000000: attempts 0x9000200 -> 0x8000200 (breakaway bit dropped, same pid returned);
non-5 errors re-raised after one attempt; happy path unchanged at 0x9000200; and a real spawn
through the helper returns rc=0.

Sites I could not find a fallback in today, same class: gateway/slash_commands.py:140 (the
in-gateway hermes update --gateway spawn - the gateway cannot update itself once it is inside
such a job) and hermes_cli/bundled_app.py:177.

Duplicates worth consolidating: #132463, #125117, #125531 (whatsapp), #129325 (dashboard),
#68128 (report).