fix(update): a successful update no longer credits or ignores surviving unmanaged serve runtimes (#100479, salvage #100490 + #100493) - #100928
Merged
Conversation
… restart match_runtime_outcomes() treats any default-profile runtime as covered once the bare "hermes-gateway" unit restarts, regardless of the runtime's own kind. An sshd-spawned `serve --isolated` backend (no systemd unit, supervisor "manual-serve") shares the default profile and gets silently marked "restarted" even though its own PID was never touched — so the #91277 Phase 2 unaccounted-runtime tripwire never fires for it and `hermes update` reports success while it keeps running pre-update code (#100479). Restrict the "hermes-gateway" special case to kind == "gateway" so a serve/dashboard runtime under the same profile falls through to "unaccounted" instead of borrowing the gateway's outcome.
…ry and escalate survivors (#100479) Widen the two salvaged fixes (#100490, #100493) to the whole class: - match_runtime_outcomes: serve/dashboard rows never borrow gateway bookkeeping at ANY site — not just the bare hermes-gateway unit name (#100490) but also relaunched_profiles / externally_supervised_profiles and the profile-substring unit match (hermes-gateway-work credited the 'work' serve). They reconcile against hermes-serve*/hermes-dashboard* units (exact names, scope prefix tolerated) or, when the caller passes the (pid, create_time) survivor probe result, by incarnation liveness. - update_cmd success path: the survivor rows from #100493's new call now feed the Phase-2 reconciliation, so a surviving unmanaged serve is 'unaccounted' -> exit 1 + 'partial' receipt, not warn-and-exit-0. - report_unaccounted_runtimes: a serve/dashboard miss names the serve remedy instead of 'hermes gateway restart', which cannot reach it. Tests: 6 reconciliation cases (sibling sites, unit vocabulary, exact-name guard, incarnation probe, remedy text) + an end-to-end cmd_update case asserting warn + unaccounted + exit 1 + receipt runtime_outcomes.
teknium1
force-pushed
the
p1/update-ci-fleet
branch
from
September 2, 2026 07:25
42bdd8c to
42c089f
Compare
Contributor
૮ >ﻌ< ა ci reviewran on 42c089f — chore: map contributor email for salvaged #100493
|
This was referenced Sep 2, 2026
Closed
10 tasks
9 tasks
cervantesh
added a commit
to cervantesh/hermes-agent
that referenced
this pull request
Sep 3, 2026
Split the self-update implementation into bounded owner modules while preserving the update_cmd compatibility facade and current runtime behavior. Keep the shipped recovery and reconciliation contracts from NousResearch#100165 and NousResearch#100928 intact, and pin the complete frozen updater execution graph.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A successful
hermes updateno longer reports a clean success while an unmanagedhermes serve/hermes dashboardruntime (sshd-spawnedserve --isolated, manual CLI serve — no systemd unit) keeps running pre-update code: the survivor is warned about, reconciled asunaccounted, and the update exits 1 with apartialreceipt (#100479; salvage of #100490 + #100493).Changes
hermes_cli/update_inventory.py—match_runtime_outcomes(cherry-pick @chelsealong fix(update): stop crediting unmanaged serve runtimes with a gateway's restart #100490, widened): serve/dashboard runtimes never borrow the gateway's bookkeeping. The original PR restricted the barehermes-gatewayunit fallback tokind == "gateway"; the widening applies the same rule to the two sibling sites it missed —relaunched_profiles/externally_supervised_profiles(a manually relaunched default gateway credited the serve too) and the profile-substring unit match (hermes-gateway-work.servicecredited theworkserve). Serve/dashboard rows now reconcile against their ownhermes-serve*/hermes-dashboard*unit vocabulary (exact names,user/scope prefix tolerated) or, when the caller supplies the(pid, create_time)survivor probe (stale_serve_pids=), by incarnation liveness: gone ⇒restarted, still alive ⇒unaccounted, probe failed ⇒ fail closed.hermes_cli/update_cmd.py— success path (cherry-pick @twotnguyen fix(update): warn surviving pre-update serve and dashboard runtimes on success (#100479) #100493, widened):_surviving_pre_update_serve_runtimes()+_warn_stale_serve_runtimes()now run on the clean success path (previously only reachable from the abort-recoveryexcepthandler, so dead on every successful update). The widening feeds those rows into the Phase-2 reconciliation so a survivor escalates (exit 1 +partialreceipt +runtime_outcomesrow) instead of being printed and forgotten under a✓.report_unaccounted_runtimes: an unaccounted serve/dashboard names the serve remedy (systemctl --user restart hermes-serve.service/ relaunchhermes serve) instead ofhermes gateway restart, which cannot reach it.contributors/emails/nguyenngoctinh011258@gmail.com→twotnguyen(attribution audit).Slots into #91277 Phase 2 (plan-vs-execution reconciliation) — observability/tripwire fix, no restart-behavior change; it does not stop or relaunch unmanaged serves (no relaunch authority), that remains #99450 / Phase 2 proper.
Validation
tests/hermes_cli/test_restart_plan_reconciliation.pytests/hermes_cli/test_update_fleet_restart_pending.pyunaccounted+ exit 1 + receipt)update_inventory.pyreverted to main'restarted' == 'unaccounted')update_cmd.pyreverted to main'pid 5555' in out, exit code)test_windows_update_restart_reconciliation4,test_update_autostash30,test_serve_runtime_inventory12,test_update_inventory11,test_update_serve_generation_recovery59,test_fleet_matrix_down_state7,test_update_restart_recovery17test_update_head_moved_gate.py::test_update_success_when_head_movesupdate_cmd.py/update_inventory.pychecked out), unrelatedruff checkon touched filesscripts/audit_pr_attribution.pyLive repro: real imports from this worktree, isolated temp
HERMES_HOME, real spawn-ledger file, a REAL child process registered as an unmanagedserve(no unit, no live spawner), plus a systemd gateway whosehermes-gateway.servicethe restart phase restarted; driven through the realhermes_cli.main.cmd_updatesuccess path with the repo's git mocks — before (origin/main):match_runtime_outcomes→ serveoutcome: 'restarted'on both the unit-name path and therelaunched_profilespath,report_unaccounted_runtimes → False, no "still run pre-update code" line printed, receiptruntime_outcomescredits the serve as restarted; after: serveoutcome: 'unaccounted'on both paths,⚠ These serve/dashboard processes still run pre-update code … pid <child>printed,⚠ Planned runtimes the restart phase never touched: ✗ serve [default] pid <child>with the serve remedy,cmd_updateexit 1, receiptoutcome: partial.Closes #100479
Closes #100490
Closes #100493
Credit: @chelsealong (#100490, first — reconciliation guard + test, cherry-picked with authorship preserved) and @twotnguyen (#100493 — success-path survivor warning + update-path test, cherry-picked with authorship preserved). Thanks to @teamster22 for the field report and the AST-verified root cause in #100479.
Infographic