Skip to content

fix(update): a successful update no longer credits or ignores surviving unmanaged serve runtimes (#100479, salvage #100490 + #100493) - #100928

Merged
teknium1 merged 4 commits into
mainfrom
p1/update-ci-fleet
Sep 2, 2026
Merged

teknium1 merged 4 commits into
mainfrom
p1/update-ci-fleet

Conversation

@teknium1

@teknium1 teknium1 commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

A successful hermes update no longer reports a clean success while an unmanaged hermes serve / hermes dashboard runtime (sshd-spawned serve --isolated, manual CLI serve — no systemd unit) keeps running pre-update code: the survivor is warned about, reconciled as unaccounted, and the update exits 1 with a partial receipt (#100479; salvage of #100490 + #100493).

Changes

  • hermes_cli/update_inventory.pymatch_runtime_outcomes (cherry-pick @chelsealong fix(update): stop crediting unmanaged serve runtimes with a gateway's restart #100490, widened): serve/dashboard runtimes never borrow the gateway's bookkeeping. The original PR restricted the bare hermes-gateway unit fallback to kind == "gateway"; the widening applies the same rule to the two sibling sites it missed — relaunched_profiles / externally_supervised_profiles (a manually relaunched default gateway credited the serve too) and the profile-substring unit match (hermes-gateway-work.service credited the work serve). Serve/dashboard rows now reconcile against their own hermes-serve* / hermes-dashboard* unit vocabulary (exact names, user/ scope prefix tolerated) or, when the caller supplies the (pid, create_time) survivor probe (stale_serve_pids=), by incarnation liveness: gone ⇒ restarted, still alive ⇒ unaccounted, probe failed ⇒ fail closed.
  • hermes_cli/update_cmd.py — success path (cherry-pick @twotnguyen fix(update): warn surviving pre-update serve and dashboard runtimes on success (#100479) #100493, widened): _surviving_pre_update_serve_runtimes() + _warn_stale_serve_runtimes() now run on the clean success path (previously only reachable from the abort-recovery except handler, so dead on every successful update). The widening feeds those rows into the Phase-2 reconciliation so a survivor escalates (exit 1 + partial receipt + runtime_outcomes row) instead of being printed and forgotten under a .
  • report_unaccounted_runtimes: an unaccounted serve/dashboard names the serve remedy (systemctl --user restart hermes-serve.service / relaunch hermes serve) instead of hermes gateway restart, which cannot reach it.
  • contributors/emails/nguyenngoctinh011258@gmail.comtwotnguyen (attribution audit).

Slots into #91277 Phase 2 (plan-vs-execution reconciliation) — observability/tripwire fix, no restart-behavior change; it does not stop or relaunch unmanaged serves (no relaunch authority), that remains #99450 / Phase 2 proper.

Validation

Check Result
tests/hermes_cli/test_restart_plan_reconciliation.py 17 passed (6 new)
tests/hermes_cli/test_update_fleet_restart_pending.py 15 passed (2 new: warn-on-success from #100493, end-to-end escalate: warn + unaccounted + exit 1 + receipt)
Sabotage — update_inventory.py reverted to main 6 new reconciliation tests fail ('restarted' == 'unaccounted')
Sabotage — update_cmd.py reverted to main both update-path tests fail ('pid 5555' in out, exit code)
Neighbors: test_windows_update_restart_reconciliation 4, test_update_autostash 30, test_serve_runtime_inventory 12, test_update_inventory 11, test_update_serve_generation_recovery 59, test_fleet_matrix_down_state 7, test_update_restart_recovery 17 all passed
test_update_head_moved_gate.py::test_update_success_when_head_moves 1 failed — pre-existing on origin/main (fails identically with main's update_cmd.py/update_inventory.py checked out), unrelated
ruff check on touched files clean
scripts/audit_pr_attribution.py all emails mapped

Live repro: real imports from this worktree, isolated temp HERMES_HOME, real spawn-ledger file, a REAL child process registered as an unmanaged serve (no unit, no live spawner), plus a systemd gateway whose hermes-gateway.service the restart phase restarted; driven through the real hermes_cli.main.cmd_update success path with the repo's git mocks — before (origin/main): match_runtime_outcomes → serve outcome: 'restarted' on both the unit-name path and the relaunched_profiles path, report_unaccounted_runtimes → False, no "still run pre-update code" line printed, receipt runtime_outcomes credits the serve as restarted; after: serve outcome: 'unaccounted' on both paths, ⚠ These serve/dashboard processes still run pre-update code … pid <child> printed, ⚠ Planned runtimes the restart phase never touched: ✗ serve [default] pid <child> with the serve remedy, cmd_update exit 1, receipt outcome: partial.

Closes #100479
Closes #100490
Closes #100493

Credit: @chelsealong (#100490, first — reconciliation guard + test, cherry-picked with authorship preserved) and @twotnguyen (#100493 — success-path survivor warning + update-path test, cherry-picked with authorship preserved). Thanks to @teamster22 for the field report and the AST-verified root cause in #100479.

Infographic

update-surviving-serve-reconciliation

@alt-glitch alt-glitch added type/bug Something isn't working comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management area/install-update Installer, updater, packaging, wheels, doctor P1 High — major feature broken, no workaround sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Sep 2, 2026
chelsealong and others added 4 commits September 2, 2026 00:25
… restart

match_runtime_outcomes() treats any default-profile runtime as covered
once the bare "hermes-gateway" unit restarts, regardless of the
runtime's own kind. An sshd-spawned `serve --isolated` backend (no
systemd unit, supervisor "manual-serve") shares the default profile
and gets silently marked "restarted" even though its own PID was never
touched — so the #91277 Phase 2 unaccounted-runtime tripwire never
fires for it and `hermes update` reports success while it keeps
running pre-update code (#100479).

Restrict the "hermes-gateway" special case to kind == "gateway" so a
serve/dashboard runtime under the same profile falls through to
"unaccounted" instead of borrowing the gateway's outcome.
…ry and escalate survivors (#100479)

Widen the two salvaged fixes (#100490, #100493) to the whole class:

- match_runtime_outcomes: serve/dashboard rows never borrow gateway
  bookkeeping at ANY site — not just the bare hermes-gateway unit name
  (#100490) but also relaunched_profiles / externally_supervised_profiles
  and the profile-substring unit match (hermes-gateway-work credited the
  'work' serve). They reconcile against hermes-serve*/hermes-dashboard*
  units (exact names, scope prefix tolerated) or, when the caller passes
  the (pid, create_time) survivor probe result, by incarnation liveness.
- update_cmd success path: the survivor rows from #100493's new call now
  feed the Phase-2 reconciliation, so a surviving unmanaged serve is
  'unaccounted' -> exit 1 + 'partial' receipt, not warn-and-exit-0.
- report_unaccounted_runtimes: a serve/dashboard miss names the serve
  remedy instead of 'hermes gateway restart', which cannot reach it.

Tests: 6 reconciliation cases (sibling sites, unit vocabulary, exact-name
guard, incarnation probe, remedy text) + an end-to-end cmd_update case
asserting warn + unaccounted + exit 1 + receipt runtime_outcomes.
@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

ran on 42c089f — chore: map contributor email for salvaged #100493

⚠️ Warnings

OSV vulnerability scan · View job

13 known vulnerabilities found in pinned dependencies.

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.


debug info

CI timings

CI timings · View report · View job

Wall time 5m56s vs 5m18s (+11.9%). 3 job(s) slower, 11 faster,

  • OS-specific tests / Windows-only tests: -53.0s
  • Check no committed infographics / check-no-committed-infographics: -47.0s
  • Python lints / ruff enforcement (blocking): -41.0s
  • OSV scan / Scan lockfiles / osv-scan: -28.0s
  • OSV scan / Emit review status: +24.0s

@teknium1
teknium1 merged commit 2b7132c into main Sep 2, 2026
37 checks passed
@teknium1
teknium1 deleted the p1/update-ci-fleet branch September 2, 2026 07:42
cervantesh added a commit to cervantesh/hermes-agent that referenced this pull request Sep 3, 2026
Split the self-update implementation into bounded owner modules while preserving the update_cmd compatibility facade and current runtime behavior. Keep the shipped recovery and reconciliation contracts from NousResearch#100165 and NousResearch#100928 intact, and pin the complete frozen updater execution graph.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/install-update Installer, updater, packaging, wheels, doctor comp/cli CLI entry point, hermes_cli/, setup wizard comp/cron Cron scheduler and job management P1 High — major feature broken, no workaround sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

4 participants