fix(update): verify updater-owning gateway restart out of cgroup

[joelshu1]

What does this PR do?

Fixes the post-update finalization path when hermes update is running inside one of the systemd gateway units it must restart.

The updater now completes dashboard/backend cleanup and non-owning fleet work first, then delegates the updater-owning gateway's terminal restart to a transient verifier outside that gateway's cgroup. The verifier publishes success only after systemd reports a healthy replacement owner and the replacement gateway acknowledges readiness.

Root cause and supported reproduction

A gateway-triggered updater inherits the gateway's systemd cgroup. setsid / start_new_session=True changes the process session, but does not move the updater out of that cgroup. The old order restarted the gateway fleet before _finish_dashboard_update_cleanup(). When the owning gateway exited, systemd tore down the cgroup and killed the updater before the dashboard/backend finalizer was reachable. The early result marker added by #8288 could therefore expose success while required finalization had not run.

The first stripped-environment attempt was not a clean reproducer and is not used as proof. The supported reproduction is the second, journal-backed attempt on Linux user systemd: non-owning gateways moved to new PIDs, the updater reached the owning gateway, the journal recorded the updater worker/shell/process being killed during the owning unit's cgroup teardown, and the managed dashboard remained active on its unchanged pre-update PID. Its health endpoint still returned 200, which proved availability but not code freshness.

Ownership-aware finalization design

• Detect the updater's exact owning systemd (scope, unit) instead of relying on a service-name substring.
• Finish dashboard/backend cleanup and verify managed dashboard recovery before the terminal owner action.
• Reconcile non-owning gateway units before the owning unit.
• Persist a nonce-bound restart request containing the original active/sub states, MainPID, start generation, restart policy, deadline, and deferred terminal result.
• Launch a transient systemd-run verifier in a separate cgroup; only that verifier signals the old owner.
• Publish exit 0 only after all of these are true:
  • ActiveState=active
  • SubState=running
  • MainPID changed
  • the systemd start generation changed
  • the replacement gateway wrote the matching readiness acknowledgement after startup
• Gate system-scope systemd-run and systemctl work behind prompt-free --no-ask-password plus root or sudo -n capability resolution before any owner signal.

Failure semantics

Accepted signal delivery without an owner transition is not success. Timeout, failed state, unchanged PID/start generation, changed owner without the matching readiness acknowledgement, stale/foreign nonce or unit data, duplicate invocation, verifier launch failure, and terminal-marker write failure remain actionable non-success.

A unit with Restart=no is not auto-started: the verifier persists verified=false, exits 1, emits no success marker, and provides explicit manual start/status guidance. The updater also avoids printing durable success while owner verification is pending.

Related Issue

No exact issue covers the updater-owning cgroup/finalizer reachability defect.

Related history: #8288, #39166, and #72192.

This is not a duplicate of #47902: that PR persists dashboard web-action status through hermes_cli/web_server.py; this PR fixes updater control-flow reachability and externally verifies the updater-owning systemd gateway. Their patch IDs and ownership protocols are different.

Type of Change

• 🐛 Bug fix (non-breaking change that fixes an issue)
• ✨ New feature (non-breaking change that adds functionality)
• 🔒 Security fix
• 📝 Documentation update
• ✅ Tests (adding or improving test coverage)
• ♻️ Refactor (no behavior change)
• 🎯 New skill (bundled or hub)

Changes Made

• hermes_cli/update_cmd.py: ownership-aware finalization order, durable result semantics, privilege-gated external verifier launch, and fail-closed owner handling.
• hermes_cli/update_owner_restart.py: nonce-bound out-of-cgroup owner transition/readiness verifier.
• gateway/run.py and gateway/slash_commands.py: replacement-owner readiness acknowledgement and gateway update watcher integration.
• hermes_cli/dashboard_procs.py and hermes_cli/main.py: ownership-aware dashboard/backend cleanup and managed restart verification.
• Updater, gateway, dashboard, fleet-timeout, and CLI regression tests covering owner-first/middle/last ordering, accepted-signal/no-transition, Restart=no, system/user scopes, marker state, stale/foreign requests, and full _cmd_update_impl ordering.

How to Test

1. Run the focused owner/finalization suite:

   scripts/run_tests.sh \
     tests/hermes_cli/test_update_owner_finalization.py \
     tests/hermes_cli/test_update_owner_restart_verifier.py \
     tests/gateway/test_update_command.py \
     tests/hermes_cli/test_cmd_update.py -q

   Result: 71 passed, 0 failed across 4 files.

2. Run the broader updater/gateway suite:

   scripts/run_tests.sh \
     tests/hermes_cli/test_update*.py \
     tests/hermes_cli/test_cmd_update*.py \
     tests/gateway/test_update*.py \
     tests/cli/test_update_command.py -q

   Result: 251 passed, 0 failed across 27 files.

3. Run static and patch checks:

   ruff check <13 changed Python files>
   python -m py_compile <13 changed Python files>
   git diff --check origin/main..HEAD

   Result: Ruff 0.15.10 passed, py_compile passed, git diff --check passed, and the structured added-line secret scan reported 0 findings.

4. Disposable Linux user-systemd verification from the independently reviewed candidate:

   • restart-capable owner: replacement PID/start generation plus readiness acknowledgement verified;
   • Restart=no owner: remained inactive with MainPID 0, verifier exited 1, verified=false, no success marker, and no temporary unit/path residue.

Checklist

Code

• I've read the Contributing Guide
• My commit messages follow Conventional Commits (fix(scope):, feat(scope):, etc.)
• I searched for existing PRs to make sure this isn't a duplicate
• My PR contains only changes related to this fix/feature (no unrelated commits)
• I've run pytest tests/ -q and all tests pass
• I've added tests for my changes (required for bug fixes, strongly encouraged for features)
• I've tested on my platform: Linux user systemd; user/system scope behavior is also covered by focused mocks

Documentation & Housekeeping

• I've updated relevant documentation (README, docs/, docstrings) — N/A; no public command/config syntax changed
• I've updated cli-config.yaml.example if I added/changed config keys — N/A; no config keys changed
• I've updated CONTRIBUTING.md or AGENTS.md if I changed architecture or workflows — N/A; no contributor workflow changed
• I've considered cross-platform impact (Windows, macOS) per the compatibility guide — the new owner verifier is systemd-gated; non-systemd paths retain their existing behavior
• I've updated tool descriptions/schemas if I changed tool behavior — N/A; no model tool surface changed

Screenshots / Logs

No screenshots. The reproducible journal evidence and disposable canary results are summarized above without including private runtime identifiers.