Skip to content

fix(gateway): route the post-update restart handoff through a job-surviving owner - #127240

Closed
JoaoMarcos44 wants to merge 1 commit into
NousResearch:mainfrom
JoaoMarcos44:fix/107002-job-object-handoff
Closed

JoaoMarcos44 wants to merge 1 commit into
NousResearch:mainfrom
JoaoMarcos44:fix/107002-job-object-handoff

Conversation

@JoaoMarcos44

@JoaoMarcos44 JoaoMarcos44 commented Sep 29, 2026 •

Copy link
Copy Markdown

Root cause

_spawn_gateway_restart_watcher (hermes_cli/gateway.py) respawned the gateway with
CREATE_BREAKAWAY_FROM_JOB and, when the parent Job Object refused breakaway, retried with a plain
Popen inside the same job. When that job tears down — the updater's non-breakaway job closing is
the reported case — the respawned gateway dies with it, and hermes update reports failure after a
successful code swap. #121635 fixed the watcher identity matcher and the verification window but
its own body states the breakaway-denied teardown "is not fixed here". No open PR examined covers
the post-update handoff (audit ownership analysis: #108720/#125117 cover other spawns, #115163 does
not integrate with the updater resume path).

Fix

Per the issue's gap, the handoff needs an owner that survives job teardown — not another Popen
retry with different flags. On Windows, when the watcher is inside a job (process_is_in_job()),
the respawn is delegated to a transient Task Scheduler task (_spawn_gateway_via_transient_task),
whose service outlives the updater's job. Outside a job, the existing breakaway spawn and its
no-breakaway retry are preserved unchanged.

Also extends tests/home_io_guard.py with sysconfig.get_path("stdlib") in the allowed interpreter
prefixes: on stable-ABI Python installs the stdlib source lives under a directory that
sys.base_prefix does not cover, and the guard's false positive blocked the new test's interpreter
file I/O (no weakening of the guard's scope for user files).

Verification

  • New regression test tests/hermes_cli/test_gateway_restart_job_handoff.py runs the production
    respawn path in a real Windows Job Object:
    • before the fix: ImportError: cannot import name 'process_is_in_job' (and the audit repro
      tests/audit/test_issue_107002.py shows the behavioural failure on unfixed code:
      "respawned gateway died when the updater's non-breakaway Job Object closed");
    • after: 1 passed, 0 failed (canonical runner scripts/run_tests.sh).
  • Sabotage check: production files stashed (tests kept) reproduced the failure; restoring them
    returned 1 passed, 0 failed.

References: #107002 (original report), #127089 (audit tracking), #127076 (audit umbrella),
#121635 (partial fix that documents this gap).

Tracking

Closes #127089 (premature-closure tracking)
Refs #107002 (original report), #127076 (audit umbrella)

…viving owner

When the updater's watcher sits inside a Job Object that refuses (or later
tears down around) CREATE_BREAKAWAY_FROM_JOB, the respawned gateway dies with
the job. Delegate the handoff to a transient Task Scheduler task, whose
service outlives the updater's job, whenever the watcher is inside a job;
keep the existing breakaway spawn (and its no-breakaway retry) elsewhere.

Also teach tests/home_io_guard about the stable-ABI stdlib directory so the
new job-handoff test is not tripped by interpreter-source file I/O.

@arkheioncorp arkheioncorp left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: fix(gateway): route the post-update restart handoff through a job-surviving owner

Verdict: APPROVE

On Windows, when the watcher process sits inside a Job Object (Electron/Tauri parent), respawning the gateway with CREATE_BREAKAWAY_FROM_JOB can fail with ERROR_ACCESS_DENIED if the job refuses breakaway. This PR detects process_is_in_job() and, when inside a job, delegates the respawn to a transient Task Scheduler task (_spawn_gateway_via_transient_task) whose launcher runs outside the updater job. The temp task + VBS launcher are cleaned up in a finally block.

Findings

  • No security concerns. The VBS launcher receives only non-secret runtime env vars (no credentials). Credentials remain profile-scoped in HERMES_HOME. The temp directory is created with a predictable prefix (hermes-gateway-handoff-) under the system temp — low risk, but worth noting. Cleanup in finally is correct; if schtasks /Delete fails, the warning is logged, not swallowed silently.
  • No logic errors. The job-detection path and the non-job CREATE_BREAKAWAY fallback are both preserved. Exception handling in the job path writes to _stdio_fh and re-raises — consistent with the existing watcher contract.
  • Suggestions (non-blocking):
    • The temp dir prefix is predictable. On a crowded/shared temp dir, a 12-hex-uuid suffix makes collisions very unlikely, but the prefix itself is not secret. This is not a practical risk for a desktop product, but if the threat model ever includes a malicious local actor racing temp-file creation, consider whether the prefix should be randomized too.
    • _spawn_gateway_via_transient_task has a 30s timeout for the launcher to mark started.marker. If WScript fails to launch (e.g., blocked by policy), the task is deleted and the temp dir removed — good. Confirm the 30s bound is acceptable for the slowest supported Windows configuration; if not, the error message could include the timeout value so operators can tune it.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/cli CLI entry point, hermes_cli/, setup wizard platform/windows area/install-update Installer, updater, packaging, wheels, doctor sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Sep 29, 2026
@kshitijk4poor

Copy link
Copy Markdown

Thanks — the premise holds up. #121635 (which closed #107002) says in its own body that the breakaway-denied teardown is "not fixed here", and Microsoft's nested-jobs docs support the claim that a breakaway can succeed while an outer job still contains the process.

I reviewed this on macOS: tests, a simulated run of the generated watcher with the job check / spawn / schtasks faked, and a static read. This PR has no CI runs and the new test is Windows-only, so none of the Windows behaviour has been proven yet. Blocking issues:

  1. No restart when the handoff fails. If schtasks /Create or /Run is denied, there's no interactive logon (the task uses InteractiveToken — SSH sessions, services, locked-down machines), IsProcessInJob errors, or the 30 s marker wait expires, the watcher logs and re-raises and the gateway is never restarted. main always at least attempted the direct Popen. Please fall back to the existing breakaway → non-breakaway Popen chain when the handoff fails.
  2. Loses fix(gateway): survive Windows Job-Object teardown across gateway restarts (#48820) #100594's diagnostics on this path. _WINDOWS_GATEWAY_BREAKAWAY_ENV is set to "" (the reader treats that as absent), and gateway stdio no longer reaches logs/gateway-stdio.log. test_respawn_stamps_breakaway_and_leaves_stdio_trace expects both.
  3. Orphaned hidden task. Cleanup lives only in the watcher's finally, but this path runs precisely because the watcher's job may be torn down. A teardown during /Run or the marker wait leaves a hidden, trigger-less Hermes_Update_Handoff_* task plus the %TEMP% launcher folder behind permanently. Having the launcher delete its own task/files would close that.
  4. Cleanup:
    • _quote_vbs_command_line is unused, and would raise on any newline because _quote_vbs_string refuses them.
    • The /Create retry loop duplicates _install_scheduled_task's.
    • On the not-in-job branch the except OSError retry without breakaway can't trigger, and the new comment describes a path that can't happen.
    • The tests/home_io_guard.py change is unrelated and should be its own PR.
    • The new test file re-implements helpers that tests/hermes_cli/test_gateway_job_teardown_live.py (TestWatcherRespawnLive) already has.

Two other open PRs target the same gap: #84409 and #115495. It would help to settle one design across the three. Whichever lands, it needs a receipt from a real Windows run — the wine2e/** lane (windows-venv-e2e.yml) exists for exactly this.

Copy link
Copy Markdown
Author

Thanks for the careful Windows-path analysis. The premise is real, but this head leaves the gateway down when Task Scheduler handoff fails, loses the existing breakaway diagnostic/stdio trace, and can orphan a hidden task if the watcher dies before its finally. Its Windows-only integration test has no hosted run on this head. I am closing this implementation while the job-surviving owner and cleanup are designed together with #84409/#115495; I cannot claim the restart contract is fixed by this PR.

@alt-glitch alt-glitch added os/windows Native Windows behaviour or breakage: PowerShell install, Windows paths, cmd.exe, WSL-vs-native diff and removed platform/windows labels Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/install-update Installer, updater, packaging, wheels, doctor comp/cli CLI entry point, hermes_cli/, setup wizard os/windows Native Windows behaviour or breakage: PowerShell install, Windows paths, cmd.exe, WSL-vs-native diff P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants