fix(update): restart supervised and manually-started backends after update

[teknium1]

Summary

After hermes update, killed dashboard/serve backends now come back automatically: systemd-supervised PIDs get their owning unit restarted, and manually-started processes are respawned with the argv they were launched with. Previously only the well-known hermes-dashboard.service unit was restarted — a remote backend under a custom unit name, or any manually-launched hermes serve, stayed dead until the operator SSHed in (#68934, Desktop loses connectivity after remote update).

Root cause: our SIGTERM reads as a clean stop to systemd, so Restart=on-failure never fires; and the manual-launch path never recorded launch args, so nothing could relaunch it.

Changes

• hermes_cli/main.py:
  • _get_pid_cgroup_path / _get_systemd_service_for_pid / _extract_scope_from_cgroup / _try_restart_systemd_service — snapshot each killed PID's owning systemd unit from /proc/<pid>/cgroup before the kill, restart it after (user/system scope aware). Salvaged from fix(update): restart systemd-supervised remote backends after killing stale processes #69029 (@webtecnica, authorship preserved; unrelated profiles.py hunk dropped).
  • _dashboard_cmdline_for_pid / _respawn_dashboard_processes — capture manually-started processes' argv (/proc/<pid>/cmdline on Linux, ps -o command= on macOS) and respawn them detached post-update, --no-open forced, output to the active profile's logs/dashboard-restart.log. Salvaged from fix: auto-restart dashboard after update when argv is recoverable #41508 (@Variable85, authorship preserved) with the sweeper-review fixes: serve matching kept, profile-aware log path via get_hermes_home(), capture gated to the update path.
  • Restarts only run with restart_managed=True (the update path); hermes dashboard --stop remains a plain stop.
• tests/hermes_cli/test_update_stale_dashboard.py: 11 new tests (unit restart + dedupe + failure hint, argv respawn + --no-open + failure fallback, /proc + ps capture, --stop never restarts). All 11 fail without the fix; the 26 pre-existing tests pass unchanged.

Validation

	Before	After
Custom systemd unit (e.g. hermes-serve.service)	left dead (clean-stop)	unit restarted via correct scope
Manually-started hermes serve / dashboard	killed + manual hint	respawned with original argv
hermes dashboard --stop	stops	stops (unchanged)
tests	26 pass	37 pass; new 11 fail on main

E2E on a live system (no mocks): real child process argv captured losslessly from /proc including embedded spaces; respawn ran detached with output landing in logs/dashboard-restart.log; full kill→respawn cycle through _kill_stale_dashboard_processes killed pid A and brought the same command back as pid B.

Closes #68934. Closes #40449.
Salvages #69029 and #41508 with contributor authorship preserved.

Infographic

[github-actions]

૮ >ﻌ< ა ci review

_{ran on 7906dc4}

ℹ️ Info

Desktop E2E visual evidence · View test artifacts · View job

1 visual diff.

_{inline evidence upload failed.}

Failed to upload diff-665a0833239e-onboarding-overlay-diff.png with gh image (exit code 1): Error uploading /home/runner/work/_temp/e2e-evidence/diff-665a0833239e-onboarding-overlay-diff.png: step 0 (get upload token): uploadToken not found on repo page — do you have write access to NousResearch/hermes-agent? (or, if NousResearch enforces SAML SSO, authorize at https://github.com/orgs/NousResearch/sso)

[alt-glitch]

This was generated by AI during triage.

Related: #69029 and #69046 address the same update-triggered supervised-backend outage with different lifecycle mechanisms. This salvage uses post-kill cgroup/service restart and additionally covers manual process respawn; a maintainer should select the intended mechanism.

[webtecnica]

Closing — this fix was absorbed as part of the gateway reconnect-watchdog cluster (#70987) by @teknium1. The backend restart logic was cherry-picked and preserved with authorship credited. The reconnect watcher now handles post-update restart correctly on main.