feat(update): complete Phase 4 fleet UX and canary rollouts (#91277)

[andrexibiza]

Phase 4: fleet UX, canary rollouts, and safe cross-runtime updates

Implements the still-unique rollout/canary remainder of #91277. This PR intentionally does not close #91277.

Current source of truth — 2026-08-27 14:22 CDT

• Live PR head: 48c11b0511c2bb18788b01a2c94d345b6daffe86.
• State: open and draft.
• The historical carrier spans 166 changed paths and is not the intended landing object.
• Exact-head proof on historical objects is provenance only; no receipt transfers to a current-main recomposition.
• Review-thread scan has no unresolved inline thread.

What current main already owns

This carrier is no longer the owner of the full Phase-4 shape.

• feat(desktop): managed SSH remote updates — the Desktop updates a remote instance end to end (#91277 Phase 4, extracted from #93042) #95942 merged the managed SSH remote-update engine extracted from this work, preserving lineage while intentionally deferring canary/rollout semantics.
• feat(desktop): Managed updates section — per-connection SSH updates with live receipts (#91277 Phase 4 UI) #95982 merged the per-connection Managed Updates UI and correlated receipts, with rollout/canary scope deliberately stripped.
• fix(update): Windows fleet check no longer fails a healthy gateway resume (#95589, salvage #94580) #95859, fix(update): Windows desktop update stall watchdog + fleet-check false failure #95897, and fix(update): crashed updates no longer strand stopped gateways — a fresh process settles the restart debt (#92145, salvage #94392) #95930 established newer updater settlement/recovery behavior.
• fix(serve): write machine sentinels to real fd 1 so the desktop handshake survives the stdout redirect (#96282) #96311 merged the canonical Desktop source-announcement repair for READY / BACKEND_PORT_IN_USE.
• fix(update): gitignored user files also block the destructive ZIP overlay #96440 now protects gitignored/user files during ZIP-fallback update; rollback/update composition must preserve that non-destructive destination contract.
• fix(state): snapshot/update restores no longer corrupt a live state.db (#90950 restore class; salvage #65960, #72085, #78465) #95531 now owns holder-safe snapshot/update state.db restoration: destructive replacement fails closed under foreign holders and current restore semantics must not be overwritten by an older rollout rollback path.

#96280 remains a distinct Windows realization defect: a uv launcher can own Electron's process/pipe while the serving interpreter is a grandchild. #96315 independently shows the observer side: a backend can be alive while a generation-bound waiter still loses. Neither permits rollout promotion based on elapsed time, process existence, or attachment to one stdout owner.

Surviving unique scope: the canary/rollout transaction over already-shipped update, UI/receipt, settlement, restart, restore, destination-preservation, and readiness contracts. The 166-file historical carrier is not the merge candidate.

Rollout transaction law

A fleet rollout is one settlement graph:

checkpoint → quiesce → apply → realize → verify → promote | rollback → reconcile

Each edge is bound to the exact connection/profile/install target, transaction generation, and current realization. A parent may not promote if a child is failed, timed out, stale, receipt-mismatched, or indeterminate.

The destination/rollback side is now equally explicit:

• a failed operation may clean only transaction-owned staging artifacts;
• it may not destroy current user/package/state data before rollback authority is established;
• a rollback must preserve newer fix(update): gitignored user files also block the destructive ZIP overlay #96440 user-file protection and fix(state): snapshot/update restores no longer corrupt a live state.db (#90950 restore class; salvage #65960, #72085, #78465) #95531 holder-safe DB restoration semantics;
• unknown realization or unknown mutation ownership requires reconciliation, not optimistic retry.

The newly reported #96597 Desktop download data-loss class is not rollout implementation scope, but it is the other side of this exact invariant: opening the durable destination before settlement and unlinking it on failure converts an indeterminate transfer into destructive mutation. Rollout code must not reproduce that shape at install/state/package sinks.

Landing contract

The next valid product object must:

1. rematerialize only the still-unique canary/rollout composition on then-current main, with the smallest collision-free FILE-LIST;
2. preserve shipped settlement/recovery semantics from fix(update): Windows fleet check no longer fails a healthy gateway resume (#95589, salvage #94580) #95859/fix(update): Windows desktop update stall watchdog + fleet-check false failure #95897/fix(update): crashed updates no longer strand stopped gateways — a fresh process settles the restart debt (#92145, salvage #94392) #95930 and managed-update ownership from feat(desktop): managed SSH remote updates — the Desktop updates a remote instance end to end (#91277 Phase 4, extracted from #93042) #95942/feat(desktop): Managed updates section — per-connection SSH updates with live receipts (#91277 Phase 4 UI) #95982;
3. preserve fix(update): gitignored user files also block the destructive ZIP overlay #96440 destination/user-file protection and fix(state): snapshot/update restores no longer corrupt a live state.db (#90950 restore class; salvage #65960, #72085, #78465) #95531 holder-safe restore semantics;
4. preserve contributor lineage from this carrier and its extracted children;
5. prove checkpoint → quiesce → apply → realize → verify → promote/rollback against observable generation-bound postconditions;
6. prove canary-to-wave promotion cannot occur when any child settlement is failed, timeout, unknown, stale_generation, or receipt_mismatch;
7. inject failure at pre-apply, post-apply/pre-realize, verify, and rollback boundaries and prove the prior durable state remains recoverable;
8. obtain fresh exact-final-head CI, Docker, Nix, rollout/platform-specific witnesses, and any required independent CI-sensitive acceptance;
9. perform a final current-main collision/subtraction read immediately before ready-for-review.

Until those gates exist on one current-main product object, this PR remains draft.

Phase-4 invariants retained from the historical carrier

• Separate Desktop-client updates from backend deployment updates so a current shell cannot hide stale local, SSH, or remote gateways.
• Use one install-scoped update mutex across Python, Bash, PowerShell, Electron, and Rust.
• Bind update ownership and rollback to immutable correlation/generation/process-incarnation evidence.
• Preserve concurrent/local Git and user state; refuse rollback that would overwrite a concurrently advanced or still-owned durable target.
• Route messaging-platform /update through the same authoritative transaction with identity-bound confirmation and exact-correlation terminal receipts.
• Treat checkpoint/apply/quiesce/restart/realize/verify/promote/rollback/reconcile as one settlement graph; partial or indeterminate children cannot be promoted to parent success.

Provenance and coordination

• fix(update): don't reset --hard when local carries unpushed commits, not a rewrite #93499 / @aaramos — local-commit preservation direction, source 69ed873961900b99b241bf85b3cb80d6d31219ec.
• fix(desktop): keep Windows startup off an occupied shared backend port #93624 / @fangliquanflq — isolated Desktop backend direction, source 1425a8ba8c863b5177bd5dba6a3261f599369a4c.
• [WIP] fix(update): bind fleet proof to process incarnation #91559 — verifier lineage where still relevant.
• feat(desktop): managed SSH remote updates — the Desktop updates a remote instance end to end (#91277 Phase 4, extracted from #93042) #95942 / feat(desktop): Managed updates section — per-connection SSH updates with live receipts (#91277 Phase 4 UI) #95982 — extracted current-main owners of managed SSH update and Managed Updates UI/receipts.
• fix(update): Windows fleet check no longer fails a healthy gateway resume (#95589, salvage #94580) #95859 / fix(update): Windows desktop update stall watchdog + fleet-check false failure #95897 / fix(update): crashed updates no longer strand stopped gateways — a fresh process settles the restart debt (#92145, salvage #94392) #95930 — shipped settlement/recovery contracts.
• fix(serve): write machine sentinels to real fd 1 so the desktop handshake survives the stdout redirect (#96282) #96311 / Windows: desktop boot always fails with "Timed out waiting for Hermes backend port announcement (90000ms)" although backend is healthy — uv venv launcher grandchild stdout never reaches the spawn pipe; readyFile channel fixes it #96280 / Desktop app: backend readiness race causes reproducible 90s boot timeout #96315 — source, realization, and observer boundaries for backend readiness.
• fix(update): gitignored user files also block the destructive ZIP overlay #96440 — shipped non-destructive ZIP-fallback/user-file preservation.
• fix(state): snapshot/update restores no longer corrupt a live state.db (#90950 restore class; salvage #65960, #72085, #78465) #95531 — shipped holder-safe snapshot/update DB restoration.
• Desktop: failed gateway download truncates/deletes a pre-existing file in the downloads directory #96597 — adjacent destructive-destination failure class; informs mutation ownership without expanding this PR's FILE-LIST.
• Complementary work remains in fix(desktop): kill all Hermes backends before updating on Windows #92879 and fix(desktop): keep isolated SSH backend on session-token auth #93828; Graceful gateway restart blocked indefinitely by a stuck cron agent turn (drain never completes) #93711 remains residual and is not claimed solved here.

Historical exact-object receipts for earlier publication objects remain provenance only and are not evidence for 48c11b… or a future recomposition.

Topology rule

Do not open another fleet/update carrier to obtain evidence or salvage already-shipped seams. Current-main product ownership is the source of truth: subtract what main owns, compose the unique rollout/canary remainder once, and certify that exact final object.

Disposition: #93042 is the provenance owner of Phase-4 rollout/canary semantics, but its 166-file head is intentionally draft. The next valid object is a narrow current-main rollout transaction that consumes, rather than replaces, shipped update/settlement/readiness/restore/destination-safety contracts.

[Enough1122]

AI code review — automated review for reference, author can ignore or act on any point.

Diff too large for automated batch review (122 changed files, +30802/-1558) — recommending manual human review.

[andrexibiza]

Exact-current acceptance receipt — 2026-08-23 14:27 CDT

Live PR head is now c34b2ecefb1e93c314dd55555fce87a72175a71e (style(desktop): naturally sort update exports). The PR body’s 08b8903d6eca3378418ee17101a9c418ac72d98e acceptance block is therefore historical evidence only; it does not transfer to this head.

Current exact-head workflow truth:

• CI 32661313316: failure only at the ci-reviewed label gate. Python main/e2e, JS/TS, Windows, macOS, Linux + Windows Rust, Ruff/ty, Windows-footgun, OSV, supply-chain, lockfile, installer, docs and attribution jobs all completed successfully. Review label gate / Review label gate failed at Fail on missing label, which then makes All required checks pass fail.
• Nix 32661312787: success.
• Docker 32661312733: success.
• Submitted review list and inline review threads: none. The automated top-level reviewer explicitly declined batch review because this is 122 changed files / +30,802 / -1,558 and requested manual review.

So the current blocker is independent review authority only, not an observed code/test regression. Please review by the architecture boundaries rather than treating 122 files as 122 unrelated units: deployment classification/refusal; install-scoped lock + correlation identity; rollout checkpoint/apply/quiesce/restart/verify/rollback; runtime ownership/restart settlement; Desktop↔gateway route/profile publication; and native-platform witnesses.

No new parallel updater/fleet PR should be opened for findings inside those boundaries; route each finding back to this owner or the existing Phase 3 companion #92545. I will not claim this head merge-ready until independent review authority is present.

[andrexibiza]

Full wave analysis in progress

[andrexibiza]

Current-head acceptance gate — 2026-08-23 21:53 CDT

Live repository truth has moved again. The PR is open/non-draft at exact head 3baf2332373217d501c89a5ca5e295b1a0ce7483; current upstream main is c584d15cdc31e1ebf3989c426ed05fb2ddb0c9fc. GitHub compare reports the branch diverged: 2 ahead / 149 behind, merge base 4a3e5c4094d96c2e184dead99bccb4f4aabb2e1b. The PR metadata currently reports mergeable=false.

The body still carries execution receipts and 5×2×3 adjudication for older immutable heads. Those remain historical evidence only. On current head 3baf233..., GitHub exposes no combined status contexts and no PR-triggered workflow runs at this read-back. Do not transfer the old green matrix to this SHA.

I re-read the Region-4 executable-path blocker against the live head. It is still present:

• _capture_rollout_active_optional_dependencies() resolves project_root, selects venv / .venv with candidate.is_dir(), computes the interpreter path, checks only interpreter.is_file(), and then executes that interpreter with subprocess.run(...).
• The canonical rollout path validator in update_rollout.py rejects symlink/reparse-point venv roots later. These two paths therefore still have split provenance authority: the optional-dependency preflight can execute through a symlink/reparse-point venv before the canonical rollout validator gets a chance to refuse it.

Required invariant: no candidate interpreter may execute until the selected venv root and executable path have passed the same no-symlink/no-reparse provenance validation used by the rollout authority. The fix should reuse one canonical validator rather than invent a second partial check. Add adversarial witnesses for a POSIX symlinked venv and a Windows junction/reparse-point venv, and prove the probe is never invoked on rejection.

After that repair, rebuild/rebase against current main, run fresh exact-head CI/Nix/Docker, and re-adjudicate the surviving R1/R2/R3/R5 findings against the new SHA rather than inheriting either the old rejection or the old green execution receipts. This remains the highest-leverage Phase-4 closure gate.

[andrexibiza]

Draft transition — semantic gates, not CI color, are authoritative

I have returned this PR to draft.

The attached 5×2×3 contracts on the prior exact head c34b2ecefb1e93c314dd55555fce87a72175a71e are explicit: R1/R2/R3/R5 = REQUEST_CHANGES, R4 = REJECT. Green execution lanes do not supersede those adjudications, and the ci-reviewed gate must not be treated as the remaining substantive blocker.

The closure target stays this existing Phase 4 object rather than spawning parallel PRs. The blocking classes are:

• canonical install/update authority before any executable preflight, including symlink/reparse rejection;
• true cross-process ownership serialization/CAS and fail-closed process discovery;
• marker→spawn and PID-generation authority through lifecycle effects;
• actor-bound confirmations and complete drain accounting for Kanban/API/cron/auto-decompose/in-flight claims;
• (connectionId, profile, generation) preservation through batch dedup/restart/re-home;
• receipt/process-ledger publication that never falls back unlocked or loses evidence on persistence failure;
• Windows/CRLF and interop witnesses that exercise the actual production boundary;
• non-destructive source-history handling: a normal upstream advance plus local-only commits must never be classified as an upstream rewrite merely because merge --ff-only failed. fix(update): don't reset --hard when local carries unpushed commits, not a rewrite #93499 records a live 2026-08-23 incident where that ambiguity fell through to git reset --hard origin/<branch> and silently deleted a local-only commit. fix(update): don't reset --hard when local carries unpushed commits, not a rewrite #93499 was then self-closed unmerged by its author, so no shipped repository fix can be inferred from that proposal. Phase 4 must absorb and regression-prove this invariant instead of opening another overlapping update PR;
• verification must prove the post-update object exists and is healthy: [Bug]: post-update fleet version check produces zero rows on a healthy resumed gateway — success path lacks the fail-closed check the restart-failure path already has #93406 records an update path that could obtain a zero-row fleet-verification set and still return success. An empty/unknown proof set is not success; it must be represented as an explicit non-success terminal state.

Implementation receipt — R1 Windows profile-root authority

The branch has now advanced to exact head 1719384152cb01188441494a8a7b217f8ec33768 with DCO-signed commit fix(update): preserve install root across profile path casing.

That commit closes the specific R1 profile-casing defect by making the Rust install-root classifier accept ASCII case variants of the profiles path component, matching Windows path semantics and Electron's normalized comparison. Regression coverage now exercises profiles, Profiles, and PROFILES for multiple named profiles. Read-back confirms the committed source at that exact SHA.

This is not a re-certification of Region 1 or of the PR. There are currently no hosted workflow runs attached to 1719384152cb01188441494a8a7b217f8ec33768, and every other adjudicated R1–R5 finding remains live until repaired or explicitly disproven at a newer immutable head. The old c34b2ec… CI/5×2×3 receipts apply only to that old head.

Do not mark ready for review until the remaining exact-head findings are repaired or explicitly disproven, followed by fresh exact-head CI/Docker/Nix and a new independent 5×2×3 adjudication. Phase 3 #92545 remains a separate upstream refusal contract; Phase 4 must compose with it rather than bypass or absorb it.

[andrexibiza]

Current-head topology correction (2026-08-26 14:57 CDT): the live delivery head is now d9542adb6ed8a15e7f18c0e0d3f1baa8a6f0c239, while temporary witness #95778 was pinned to obsolete source bd6964d725d9673e5399335772a09ce189bb669f. I closed #95778 rather than allowing proof machinery for an older object to remain in the active topology. GitHub currently registers zero workflow runs for exact live head d9542adb…; prior cc9f235… / bd6964d… receipts therefore do not certify the current merge object. #93042 remains the sole delivery owner. The next acceptance object is the unchanged current head with executable exact-SHA CI/Docker/Nix (or a new product head only if a real code change is required), followed by the landing-edge collision read. No additional publication carrier should be treated as product truth.

[andrexibiza]

Landing-edge topology guard — 2026-08-26 17:53 CDT

The live publication object is now d9542adb6ed8a15e7f18c0e0d3f1baa8a6f0c239, while current main has advanced to db127f75021fe74ee4c8f32433a3db12bd3c10dc. The live comparison is diverged: 364 commits ahead / 48 behind from merge base bee489d5638d98517942f9618b9da94abd581a8d.

There is now a concrete landing-edge semantic collision, not merely branch age. After this PR's current head was minted, #95859 merged on main as b3e477f304e43b7ff6427c7e186bfd9e524bbff2 and changed hermes_cli/update_cmd.py, which is also part of this PR's 165-file surface. #95859 makes post-update fleet recovery wait on the actual resumed-gateway identity predicate with a bounded 2s poll / 30s deadline instead of declaring a healthy detached Windows resume failed after a fixed 2 seconds. That current-main behavior is part of update settlement truth and cannot be overwritten or silently dropped by Phase 4 composition.

I have moved this PR back to draft to make that topology explicit. No CI/Docker/Nix receipt from cc9f235…, bd6964d…, or any earlier object transfers to d9542adb…, and no receipt on d9542adb… can certify a future current-main composition.

The next valid publication object is one semantic composition onto then-current main that preserves #95859's bounded resume-settlement behavior plus this PR's Phase 4 ownership/rollback/receipt invariants, followed by fresh exact-final-head CI, Docker, Nix, and any required CI-sensitive acceptance gate. Do not create another carrier PR for this.

[andrexibiza]

Current-main topology correction after the latest extraction pass:

• live delivery head is 48c11b0511c2bb18788b01a2c94d345b6daffe86, not the older cc9f235… object described in the body;
• GitHub currently reports this live object not mergeable and it spans 166 changed paths;
• there are zero registered GitHub Actions runs on 48c11b…, so none of the historical exact-head receipts in the body certify the current object;
• feat(desktop): managed SSH remote updates — the Desktop updates a remote instance end to end (#91277 Phase 4, extracted from #93042) #95942 has now merged the managed SSH remote-update engine that was extracted from this carrier, preserving the Phase 4 lineage while intentionally deferring canary/rollout semantics;
• feat(desktop): Managed updates section — per-connection SSH updates with live receipts (#91277 Phase 4 UI) #95982 is the live UI-side extraction for per-connection managed updates/receipts and deliberately strips the rollout/canary layer;
• current main also contains newer settlement/recovery semantics from fix(update): Windows fleet check no longer fails a healthy gateway resume (#95589, salvage #94580) #95859, fix(update): Windows desktop update stall watchdog + fleet-check false failure #95897, fix(update): crashed updates no longer strand stopped gateways — a fresh process settles the restart debt (#92145, salvage #94392) #95930, and related updater work that this carrier must compose rather than overwrite.

I have moved this PR back to draft because the merge object no longer matches the architectural ownership claimed by the old publication body. The surviving unique scope is now the canary/rollout composition over the shipped update engine + current UI/settlement contracts, not the 166-file historical carrier.

Landing condition: materialize that remaining scope onto current main with the smallest collision-free FILE-LIST, preserve extracted contributor credit, then obtain fresh CI/Docker/Nix and rollout-specific evidence on the exact final head. Historical green and extracted child receipts do not transfer to 48c11b….

[andrexibiza]

Implemented the topology cleanup before adding another receipt: the PR body is now reconciled in place to live head 48c11b0511c2bb18788b01a2c94d345b6daffe86, draft/nonmergeable state, zero exact-head workflow registrations, and the current ownership boundary after #95942/#95982/#95859/#95897/#95930. Historical cc9f235… green is retained only as non-transferable provenance.

The merge target is now stated narrowly: one current-main canary/rollout composition over the shipped managed-update engine/UI/settlement contracts, smallest collision-free FILE-LIST, preserved contributor lineage, then fresh exact-final-head CI/Docker/Nix + rollout/platform witnesses. No second fleet/update carrier.

[andrexibiza]

Current-main update interlock, 2026-08-27 09:55 CDT:

Two fresh reports land on seams that this historical Phase-4 carrier used to own, but they should not widen this PR back into a general updater/remote-lifecycle carrier.

• Desktop SSH fails on remote host with /bin/sh=dash: 'hermes-update-mutex: 1: Bad substitution' #96367 reproduces the extracted managed-SSH path failing first connect on Debian/Ubuntu /bin/sh=dash because the remote mutex payload is executed with sh -c while containing bash-only ${var//...} substitution.
• Windows Desktop update aborts on unrelated SCM service in STOP_PENDING #96360 reproduces Windows hermes update --gateway failing closed on an unrelated SCM service in STOP_PENDING because indeterminate service state is rejected before candidate ownership is qualified against validated Hermes gateway ancestry.

These are current-product defects in the already-extracted update/lifecycle surface. The remaining #93042 object should consume their eventual current-main resolutions; it should not reimplement either fix inside the 166-file carrier.

The shared architectural rule is narrower and useful for the canary/rollout remainder: qualify the candidate owner before applying fail-closed authority semantics, and bind shell/runtime assumptions to the actual execution environment. A rollout must not promote on a path that is only valid under bash or that treats unrelated host state as update authority.

Landing boundary remains unchanged: rematerialize only canary/rollout composition on then-current main, preserve whatever exact fixes settle #96367/#96360, and certify the final composed object. Historical Phase-4 receipts do not certify those newer runtime surfaces.

[andrexibiza]

Topology reconciliation implemented in the PR body at live head 48c11b0511c2bb18788b01a2c94d345b6daffe86.

The important correction is that #96282 is no longer an open readiness dependency: #96311 shipped the source-announcement half. #96280 remains the distinct launcher/grandchild realization boundary, and #96315 is independent incident evidence that source announcement is not equivalent to generation settlement.

That narrows this carrier further. The only defensible surviving product scope is canary/rollout composition over current-main update, receipt, recovery, restart, and readiness contracts. The 166-file carrier should not land by conflict resolution; the next valid object is a current-main, collision-free rematerialization of that unique remainder with exact-final-head proof.

[andrexibiza]

Reconciled the rollout carrier against newer current-main mutation contracts. #96440’s user-file preservation and #95531’s holder-safe DB restore now sit inside the rollout settlement boundary; the adjacent #96597 data-loss class makes the other side explicit: failure cleanup may delete only operation-owned staging, never a pre-existing durable destination. The 166-file carrier remains provenance. The only valid landing shape is the narrow canary/rollout graph consuming shipped updater/UI/readiness/restore semantics.