fix(update): Windows desktop update stall watchdog + fleet-check false failure

[teknium1]

Windows desktop updates stop hanging forever after the build finishes

A Windows desktop user who clicks Update today can end up with an app parked on "Updating Hermes" indefinitely — no exit-code line, no relaunch, a zombie updater tree — or with a successful update reported as failed (exit 1) after ~14 minutes of silence. This PR fixes both, on one branch, because both live in the same update pipeline.

Credit: the watchdog architecture is @fangliquanflq's work from #95625, cherry-picked with authorship preserved. This PR salvages it, amends its one blocking defect, and fixes the second root cause the original didn't touch.

Two root causes, named honestly

Issue #95589's desktop-handoff hang is not #95086's scoped-kill problem. Adjudication traced two independent mechanisms:

Root cause A — the drain loop never arms its abandon clock for a live-and-silent child. scripts/desktop-update/windows.ps1's step drain only started its abandon countdown after $proc.HasExited. A child that finishes its visible work but stays alive and silent in finalization hangs the hand-off forever: no .hermes-update-result.json, the .hermes-update-in-progress marker never clears, the Desktop is never relaunched.

Root cause B — the post-update fleet verification counts the Windows pause/resume token as a runtime (#93406 residual). _fleet_probe_expected_runtimes treated the _windows_gateway_resume token (profiles/unmapped entries) as "fleet rows expected". But the token is pause/resume bookkeeping, not a runtime inventory: unmapped Scheduled-Task gateways never publish gateway_state.json rows at all, and a resumed profile gateway relaunches detached and may not republish within the probe window. So collect_fleet_versions() legitimately returns zero rows for a healthy update, the verification loop silently waits out its window, prints "Fleet version check returned no rows", and exits 1 — a false failure on a successful update.

What ships

1. Salvage of #95625 (3 commits, @fangliquanflq's authorship preserved)

The right watchdog architecture for root cause A: a C# HermesUpdateJob job object per step, CREATE_SUSPENDED → assign-to-job → resume (no updater instruction runs before the cancellation boundary exists), a no-output ceiling on live-and-silent children, whole-tree TerminateAndWait with code-124 retry, fail-closed finalization when quiescence can't be proven, and a -SelfTestPipeDrain stall arm as executable proof.

2. Ours-authored amendment: logs/update.log growth is progress (the blocking defect)

#95625's 5-minute no-output ceiling would have killed every healthy large update: real updates are 40+ minutes stdout-silent by design, because hermes update captures the Electron/vite build into logs/update.log, not the child's stdout (hermes_cli/update_cmd.py's update-log tee). The amendment fingerprints logs/update.log (size + mtime) and, when the idle ceiling is otherwise reached, treats growth as progress — reset the clock instead of terminating the tree. The stat runs only once the ceiling fires, so the hot drain path never touches the filesystem. HERMES_UPDATE_STEP_IDLE_SECONDS remains the override; HERMES_UPDATE_PROGRESS_LOG points the self-test at its own file.

TDD + sabotage proof: -SelfTestPipeDrain gains a fourth arm, logstall — a step silent on its pipes but appending to the progress log every second, which must reach its natural exit 3, never 124. Linux CI pins the same contract at source level (TestIdleWatchdogCountsUpdateLogGrowth, written red-first). Sabotage-verified: making the log-growth consult inert ($currentLogStamp = $progressLogStamp) fails test_stall_branch_consults_log_growth_before_terminating; restoring it goes green.

3. Our fix for root cause B (#93406)

_fleet_probe_expected_runtimes no longer counts the resume token; expected-runtimes keys only on row-capable signals — restart-phase bookkeeping, the pre-restart PID snapshot, and the pre-update plan inventory — which already cover any genuinely live pre-update Windows gateway.

Counterfactual proof: tests/hermes_cli/test_update_fleet_probe_resume_token.py was run against the pre-fix predicate first and fails there (assert True is False — token-only returned True); it passes with the fix. The row-capable signals are pinned unchanged, and test_update_fleet_check_fail_closed.py is updated to the corrected semantics.

Verification

• tests/test_desktop_update_windows_pipe_drain.py — 5 passed (source contracts), 1 skipped (windows_only executable fixture).
• tests/hermes_cli/test_update_fleet_probe_resume_token.py + test_update_fleet_check_fail_closed.py — 16 passed.
• ruff check clean on all touched Python files; windows.ps1 additions are pure ASCII, CRLF preserved, brace-balanced.
• Note: 6 pre-existing failures in tests/hermes_cli/test_cmd_update.py reproduce identically with this branch's changes stashed (they are environmental/pre-existing on the base, untouched by this PR).

Windows harness leg design (for the record; running it is a follow-up)

From the adjudication, the CI leg that would make this class of regression impossible to reintroduce — designed here, executable on a windows-latest runner whenever the maintainer orders it at merge time:

• Tier 1 — self-test leg (~2 min): run scripts/desktop-update/windows.ps1 -SelfTestPipeDrain on windows-latest with HERMES_UPDATE_PIPE_DRAIN_SECONDS=3, HERMES_UPDATE_STEP_IDLE_SECONDS=3, HERMES_SELFTEST_HOLD_SECONDS=45 (i.e. un-skip test_update_step_survives_pipe_leak_flood_and_live_child_stall there). Covers all four arms — leak, flood, stall, logstall — through the real Invoke-HermesStep, real job objects, real process table.
• Tier 2 — stall-injection leg (contract-job extension): in the existing desktop-windows-e2e.yml contract job, add a leg that stages an update whose final step is wrapped to enter silent finalization after its visible work (the Windows desktop: hermes update hangs after build — never relaunches desktop (repeatable 2/2, zombie process) #95589 shape, injected via a HERMES_UPDATE_STEP_WRAP test hook or a staged commit whose post-build step execs a sleeper). Assert: watchdog fires within the configured ceiling, exit-code line is written, .hermes-update-in-progress is cleared, result JSON exists, and the Desktop relaunch step runs — i.e. the hand-off's terminal obligations survive a stalled child end-to-end. A companion healthy-slow leg (step silent on stdout while appending to logs/update.log beyond the ceiling) must complete WITHOUT a 124.

Tier 1 is cheap enough for every push; Tier 2 rides the existing push-to-main/nightly triggers (never pull_request, per the harness's 2x-cost-runner policy).

Infographic

Windows harness A/B — Tier 1 EXECUTED (windows-latest, PR head 951d927)

Result: -SelfTestPipeDrain PASS — exit 0 in ~104s with all four arms asserted:

• stalled child → killed at code 124, TreeQuiesced, child + descendant dead, started after job assignment (the watchdog works)
• silent-but-logging child → NOT killed, natural exit 3 — logs/update.log growth counted as progress (the amendment works; healthy 40-min builds survive)
• leak arm → abandoned at the 20s grace with output preserved (Windows: Desktop update hand-off hangs forever when gateway stays resident (windows.ps1 waits on ReadToEndAsync that never sees EOF) #90455 shape)
• flood arm → unmetered, all 8 MB captured

Proof run: https://github.com/NousResearch/hermes-agent/actions/runs/33026462955 (exit=0 line in the log; driven via a python pipe with a hard 420s timeout). The job's own conclusion shows cancelled because the Actions log pipeline wedges echoing the flood arm's 8 MB / 128KB-per-line output — attempts 1–2 (https://github.com/NousResearch/hermes-agent/actions/runs/33023126499, https://github.com/NousResearch/hermes-agent/actions/runs/33024889974) established that wedge signature: identical silence at the flood arm's first 128KB line, step-level timeout unable to advance, always() step never emitted. Runner-infra artifact, not a watchdog defect.

Counterfactual note: no disarm knob exists for a runner-side Leg B; the log-growth sabotage proof runs in pytest form (tests/test_desktop_update_windows_pipe_drain.py, windows_only, green in this PR's CI).

Future re-runs: cap HERMES_SELFTEST_FLOOD_KB (e.g. 256) or avoid echoing raw fixture stdout to the step console.

Proof branch proof/95897-tier1 deleted; nothing from it merges.

[andrexibiza]

Reviewed exact head 8b09e062b8882e79967b9e23aefe0f2839575665 on live main@2f8d6b55523f52f70a9946035c08333cbf2889fe. This is a direct current-main child (5 commits, 5 files, +660/-79), and there were no existing submitted reviews when I started.

There is strong work here. The CREATE_SUSPENDED → AssignProcessToJobObject → ResumeThread sequence closes the startup-before-cancellation-boundary race cleanly; timeout handling terminates the whole job rather than just the direct process; TerminateAndWait() makes retry contingent on proving zero active processes; and the finalizer refuses to remove the marker/relaunch Desktop if quiescence cannot be proven. That is a much stronger recovery contract than “kill pid and hope.” Preserving fangliquanflq’s #95625 commits/credit while tightening the failure boundary is also the right salvage shape.

I found two P1 blockers on the exact submitted object.

P1 — the watchdog’s update.log progress signal does not exist during the real Desktop build

The amendment correctly recognizes that pipe silence is not equivalent to update inactivity, but the implementation relies on a false premise about how the build log is written.

At this head, hermes_cli/update_cmd.py::_run_logged_subprocess() does:

result = subprocess.run(
    cmd,
    ...,
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    text=True,
    ...,
)
_log_only_write(result.stdout or "")
return result

The Desktop build then calls that helper with desktop_build_cmd.

subprocess.run(... stdout=PIPE ...) waits for the child to finish and buffers its output in memory. _log_only_write(...) runs after the build process returns. So during the long Electron/vite build, the actual logs/update.log is not growing from that build output at all. The outer Python updater is blocked inside subprocess.run, and the PowerShell handoff sees exactly the condition this branch calls a stall: no stdout/stderr movement and an unchanged update.log.

That means the new default 300-second watchdog can terminate a healthy build at five minutes, return 124, retry once, and terminate the retry the same way. This is the exact healthy-large-update cliff the amendment says it removed.

The new logstall self-test does not exercise the production shape: its fixture writes the progress file directly with Add-Content once per second. The Linux source-contract tests only prove that the PowerShell code consults the file; they do not prove the real updater grows it while the build is in flight.

Required repair: make the progress source genuinely live. The cleanest shape is to change the logged-subprocess path to stream/tee combined build output into update.log incrementally while still suppressing it from the terminal, or introduce another progress receipt that is actually updated during the build. Then add a regression through the real logged-subprocess mechanism: a pipe-silent build lasting beyond HERMES_UPDATE_STEP_IDLE_SECONDS must keep producing the chosen progress receipt and must reach its natural exit rather than 124. Please keep the existing silent-no-progress stall arm as the negative control.

P1 — removing the Windows resume token reopens the merged #93406 fail-open class

hermes_cli/update_cmd.py::_fleet_probe_expected_runtimes() now explicitly discards windows_resume_token, and the tests change token-only profiles / unmapped cases from expected True to False.

That reverses the reason #93654 was merged. The Windows token is not merely presentation bookkeeping: _pause_windows_gateways_for_update() creates it from gateway processes that were actually discovered and stopped. Existing regression coverage shows a real shape like:

{
    "resume_needed": True,
    "profiles": {"work": 101},
    "unmapped_pids": [202],
    "unmapped": [{"pid": 202, "argv": [...]}],
}

#93654 deliberately counted the token because the Windows pause/resume path can populate neither restarted_services nor killed_pids, and later PID discovery can be empty precisely because the updater already paused those runtimes. #93406’s live report demonstrated the consequential failure: a gateway was known live pre-update, collect_fleet_versions() returned [] at t+0/+5/+15/+60 and even minutes later while the gateway was serving Telegram, and the old success path silently treated that as verified enough.

The current caller already has the right bounded shape: it polls for up to 30 seconds only when _fleet_rows_expected is true, and then marks a persistently empty snapshot incomplete. By making a token-only paused profile return false, this PR skips both the bounded poll and the zero-row incomplete guard. A known pre-update runtime can therefore fall back to implicit success again.

The rationale in this PR identifies a real distinction, but it collapses two different token classes:

• unmapped Scheduled-Task entries may indeed have no fleet-row representation;
• profiles are still proof that an actual profile gateway was paused and a replacement is expected.

Those should not share one “ignore the token” verdict.

Required repair: preserve the fail-closed evidence while fixing the false failure. Split the token semantics: a paused profile remains a positive pre-update liveness receipt and must converge to a verified replacement (or an explicit incomplete/partial outcome); an unmapped runtime needs a separate reconciliation outcome rather than an impossible fleet-row requirement. The closed #94580 explored the correct direction—bounded wait while preserving #93406’s fail-closed contract—though the current tree already contains the bounded polling mechanism, so this does not require reviving that PR.

Please add a vertical regression through the actual lifecycle rather than only calling the predicate in isolation:

1. _pause_windows_gateways_for_update() yields a profile token; later PID snapshot is empty; zero fleet rows must not become green.
2. The same profile publishes a current row during the bounded window; verification converges to success.
3. An unmapped-only token does not demand a row the collector can never produce, but its runtime outcome is still accounted for explicitly rather than erased.

Interlocks / ownership

• #95625 / fangliquanflq is the source watchdog carrier being salvaged here. #95897 is the better consolidation point once the live-progress issue above is repaired; preserve the existing commit authorship and retire the duplicate carrier explicitly rather than leaving two watchdog owners.
• Merged #93654 is the canonical #93406 fail-closed fleet-verification contract, carrying the #93410 / RelaxJonh salvage lineage and Halldrix’s live Windows evidence. #95897 should compose with it, not reverse its token-only safety case.
• #95896 is complementary recycled-PID identity hardening: it fixes false STALE rows, not empty-snapshot authority.
• #95898 is complementary Windows runtime-outcome reconciliation. Its accounting surface is the right place to keep “unmapped but intentionally relaunched” distinct from “no runtime existed.”
• #95589 remains the live handoff-stall incident this watchdog is meant to close. The job-object/quiescence work is the right spine for that closure.

At review time there is no substantive exact-head hosted acceptance receipt yet. The head has only skipped Label rerun check-runs; no CI/Docker/Nix product matrix has executed on 8b09e062…. The local/source tests are useful development evidence, but the native Windows job-object path is exactly the part that needs an exact-object host witness before landing.

The process-tree safety work is genuinely substantial and worth preserving. Once the watchdog watches a progress signal the real build actually emits, and the fleet fix stops false failures without deleting pre-update liveness proof, this becomes a much stronger Windows update transaction. 🚀

[andrexibiza]

Landing-edge topology update after the review above: main advanced two commits to b0563d5ad3fd48f04cbe1d5a92e277e8408c461e, so this unchanged head is now 5 ahead / 2 behind merge base 2f8d6b55523f52f70a9946035c08333cbf2889fe. I read the main-only file list: the drift is confined to memory setup/Supermemory and FAL video-generation paths, with no collision against this PR’s five files. The two P1 findings therefore stand unchanged; no restack-only semantic finding is added.

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 5c90ba1 — test: invert the resume-token fleet-probe pin to the new con}

⚠️ Warnings

OSV vulnerability scan · View job

6 known vulnerabilities found in pinned dependencies.

• CVE-2026-67213 in website/package-lock.json
• CVE-2026-71554 in uv.lock
• CVE-2026-70608 in package-lock.json
• CVE-2026-56876 in package-lock.json
• CVE-2026-70606 in package-lock.json
• CVE-2026-71554 in uv.lock

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.

---

debug info

CI timings

CI timings · View report · View job

Wall time 4m9s vs 3m55s (+6.0%). 9 job(s) slower, 2 faster, 1 unchanged.

• OS-specific tests / Windows-only tests: +99.0s
• Python tests / Run tests: +8.0s
• OSV scan / Scan lockfiles / osv-scan: -6.0s
• Python tests / e2e: +4.0s
• OS-specific tests / macOS-only tests: +4.0s