Repository navigation
fix(update): after the commit point nothing fails hermes update (C3) - #132386
Merged
Merged
Conversation
૮ >ﻌ< ა ci reviewran on 189ce63 — chore: retrigger CI (zero-job dispatch failure, auto-heal) debug infoCI timingsCI timings · View report · View jobWall time 28m51s vs 25m38s (+12.5%). 18 job(s) slower, 12 faster, 1 unchanged.
|
alt-glitch
added
type/bug
andrexibiza
reviewed
Oct 3, 2026
andrexibiza
left a comment
andrexibiza
left a comment
There was a problem hiding this comment.
Review: one P1 — keep commit success separate from permission to mutate gateway topology
Reviewed head: 12ca0d5cd962865aadd6d48a00394208814f451d. The exit-zero contract is not the objection. The problem is replacing a runtime-safety signal with a constant at a different decision boundary. One focused inline identifies the change; the RCA, cross-surface analysis, and validation are consolidated here.
P1: the selected-completion path bypasses the existing SQLite migration guard
_complete_selected discards the result of complete_source_checkout and now passes update_complete=True into fleet verification. This is not merely normalizing an exit status. The return value produced by _complete_locked remains tail_done and all(verdict): a completed tail with an unsafe SQLite verdict returns False, even though repeating the tail is not the remedy.
The downstream _verify_fleet_after_update still has an explicit if not update_complete: return immediately before multiplex auto-migration, with the documented reason that the selected runtime's SQLite is unsafe. The new constant makes that guard ineffective for this caller.
Trigger: completion reports unsafe SQLite; restart is not explicitly deferred/skipped; fleet verification otherwise finishes without marking the fleet incomplete; the host is eligible for automatic migration. The updater then enters maybe_auto_migrate_after_update, which can call apply_migration to consolidate standalone profile gateways into the default multiplexer despite the already-negative runtime verdict. The separate automatic-migration guards check service domains, UNIX users, and home-tree boundaries; those are not a replacement for the dropped SQLite condition. This exposes an otherwise healthy fleet to an optional topology mutation on a runtime the updater already considers unsafe.
RCA
There are three distinct facts: source adoption committed, retryable tail work is complete, and the selected runtime is safe enough for additional topology changes. _complete_selected correctly separates source-tail debt from the update exit contract, but also overwrites the signal consumed by the migration safety gate. Committed ⇒ exit zero does not imply committed ⇒ safe to migrate topology.
The same loss occurs when completion raises and is caught: verification still receives True without an affirmative completion verdict. That is another manifestation of this finding, not a second independent issue.
SSA — producer, consumers, and recovery boundaries
Verdict producer and install consumer. The shared source-completion implementation deliberately retains the negative SQLite verdict for installation while keeping it distinct from source-tail work. Do not globally redefine that return value to mean only “the tree moved.”
Update exit and marker consumers. The completion worker separately controls the source marker, the gateway watcher's success code, restart deferral, and verification. Keep the exit-zero behavior and the tail-debt-based marker policy. Neither requires passing True into migration eligibility.
Fleet and migration consumers. The fleet verifier independently checks stale/down/unaccounted runtimes and owed restarts before reaching the runtime-safety guard. Preserve both gates: fleet currentness does not establish interpreter safety. The migration hook and its plan application must not be authorized by an exit-status constant.
Validation and closure
I executed an isolated probe of the transcribed, pinned _complete_selected function, with imports and side effects stubbed. This was not a repository test-suite run and did not stop or migrate real gateways.
| Completion fixture | Value passed to fleet verification | Source marker cleared |
|---|---|---|
True, no tail debt |
True |
Yes |
False, unsafe-SQLite shape with no tail debt |
True |
Yes |
False, build follow-up owed |
True |
No |
Raises RuntimeError |
True |
No |
Explicit no-restart and restart-skip controls did not invoke verification, so the finding is scoped to the reachable verification path, not every update. The probe establishes the lost argument; the downstream consequence is source-traced through the linked guard and migration hook, not claimed as a live outage reproduction.
For the bounded fix, retain the actual completion verdict, initialize it conservatively before the exception boundary, and pass that verdict to the fleet verifier, while continuing to return successful update status after commit. A structured result is also valid, provided runtime safety is explicit rather than inferred from an empty retry list.
Regression coverage should combine the caller with the real verification guard and a mocked migration hook: unsafe SQLite + otherwise-current fleet must retain exit zero and never call migration; a caught completion failure must likewise not authorize migration; healthy completion must still allow it when eligible. Preserve the existing tail-marker and no-restart behavior in those tests.
The current CI report separately reports failures in desktop build-fail E2E, Python tests (2/2), and git upgrade E2E. I am not treating those red checks as three additional diagnosed bugs, nor claiming full-suite, Windows, or service-manager integration validation here.
andrexibiza
reviewed
Oct 3, 2026
There was a problem hiding this comment.
Supplemental C3 review: successful code updates still lose or misreport deferred work
Reviewed head 12ca0d5cd962865aadd6d48a00394208814f451d against base 5d3c05977bb3c8b7cfd6b3e39d96f6e35a9e0662. These findings accept the proposed post-commit exit-zero contract. The defects are in retaining the owed work, preserving runtime-health gates, and propagating the richer outcome into existing consumers. Six additional P2 correctness findings remain. The existing P1 SQLite review arrived during this analysis; F3 below adds executed consumer-path evidence to that finding and does not create another inline.
F1 — [P2] Real failed tail steps bypass the new follow-up adapter and clear retry debt
The new owed_step adapter sees only escaped exceptions. Its actual config migration and profile sync callees consume those errors and return normally. An ordinary filesystem failure therefore reaches tail_done = not owed and stamp publication with an empty list; completion clears the pending marker.
Observed: inject OSError("disk full") below the adapter, first at the actual version-only migrate_config call and then at seed_profile_skills. Imported completion → maintenance → real migration/profile helpers return True, caller and receipt follow-ups are both [], and the completed-stamp writer is invoked. The new test replaces the whole config helper with a raising mock, so it excludes the failure-consumption behavior that matters here.
There is a second hole in the same retry protocol: maintenance produces profile_sync, but TAIL_FOLLOWUPS omits it. A separate caller probe supplied False plus [('profile_sync', 'disk full')]; _complete_selected still deleted its pending marker.
RCA / repair: normal return is not a completion verdict for these helpers. Preserve explicit accumulated failure results while continuing independent steps and profiles; record every owed step and include profile_sync in the tail classification. The inner swallowing predates this PR; this is an incomplete implementation of the new retry guarantee, not a claim that disk-full failures are new.
Regression: inject failure at the real inner operations rather than replacing the adapter's callee. Require the later steps to run, a correlated follow-up to exist, no completed stamp, and retry debt to remain through the next launch.
F2 — [P2] Root-scoped receipts break managed SSH updates launched from a profile
The receipt producer now resolves the root home, while the SSH observation script still reads the launch home's receipt directory. Named-profile launch homes are explicitly supported, and the remote home probe preserves an explicit HERMES_HOME.
Observed: execute the exact embedded observation script with launch home <root>/profiles/coder, correlated exit marker 0, and a successful correlated terminal receipt in <root>/logs/update_receipts. It returns exitCode: 0, receipt: null. Put the identical receipt in the old profile location and it is recognized. With the marker clear, the waiter rejects the successful update after the receipt grace period.
RCA / repair: the writer's artifact scope changed without migrating its readers. Use the already-derived install root for receipts, retaining the launch home for correlated status/intent files. The sibling update.log move has the same mismatch: the tee writes at root, but logs.tail_log reads the active profile. The probe returns exit 1/“Log file not found” despite the root update log existing.
Regression: exercise root → named profile → root with an actual root receipt/log, a matching correlation, absent update marker, and successful status. The SSH waiter must accept it and logs update must open the update log. Do not move the profile's status/intent files as a side effect.
F3 — Existing [P1] SQLite finding: additional consumer-path verification
_complete_selected discards the completion verdict and passes update_complete=True. That value controls the retained guard which leaves gateway topology alone on an unsafe SQLite runtime.
Observed: execute actual _finish → _complete_selected → fleet verification with the source-completion boundary returning False and reporting sqlite_runtime, and a healthy fleet snapshot. Exit and terminal receipt are correctly 0/success, but multiplex migration is invoked. The probe substitutes the migration effect with a recorder; it does not claim an observed session-corruption incident.
RCA / repair: committed-code success, completed tail, and runtime safety are different values. Preserve the runtime/completion verdict for the migration gate while keeping the command exit zero. The pre-PR caller passed the real verdict; changing the exit policy does not require making the runtime gate unconditional.
Regression: unsafe SQLite + current fleet must produce successful code-update status and the runtime follow-up, with zero topology-migration calls. A safe, finished completion is the positive control.
F4 — [P2] Restart failures still overwrite gateway IPC with a contradictory terminal failure
Completion now prepublishes .update_exit_code = 0, but the failed-service branch, abort recovery, and Windows resume still write 1 for restart debt. The notification consumer immediately sends its terminal notice and clears markers when that file appears.
Observed: actual _finish routing with the real failed-systemd and failed-Windows-resume branches produces exit_code: 0, terminal outcome success, gateway_restart follow-up, and marker contents 1. Services and process effects were supplied as deterministic boundary inputs. A later detached-wrapper rewrite to zero cannot repair a notification already consumed during the verification window.
RCA / repair: independent IPC writers retain the previous meaning of failure after the command/receipt contract changes. Post-commit restart debt belongs in the follow-up and obligation channels; every gateway status writer must preserve the committed update's success meaning. Keep true pre-commit/rollback/interrupt failures distinguishable.
Regression: hold verification/finalization behind a barrier, inject service/recovery/resume failure, and run the notification reader before the wrapper's terminal rewrite. Require consistent successful code-update status plus visible restart debt; no premature failure notice or lost follow-up.
F5 — [P2] A killed running receipt drops carried manual-restart obligations on the next run
Receipt begin copies prior plan/pending rows into carried_manual_serves and replaces latest.json. If that run dies before publishing a new plan, the next begin copies only plan.runtimes and pending_manual_serves; it cannot read the carried rows in the running schema it just wrote. The finalizer then receives an empty carried list.
Observed: start from one untransferred manual-serve obligation, begin a receipt in a separate child, abruptly exit that child before plan publication, then begin/finalize in another child. Counts are pending=1 → running carried=1 → final pending=0, final outcome success. The same experiment against the fetched base receipt implementation retains pending=1 after interruption. Reminder transfer is deliberately made to fail, which is exactly the condition the retained debt must survive.
RCA / repair: the new durable running representation is not closed under recovery/replay. Preserve and merge carried debt across every generation, or persist it under the existing public pending key from the start. Retention and startup warning readers must understand the same representation; another receipt rotation must not turn failed transfer into completion.
Regression: interrupt before plan publication and at subsequent stage boundaries while the reminder transfer fails. Repeated begin/reconcile/finalize must retain the same obligation until successful transfer or independently verified recovery. Deduplicate by runtime identity rather than dropping the row.
F6 — [P2] Desktop's detached result hides deferred builds behind “Update complete”
The new build guard records a build follow-up and leaves completion owed, but the POSIX handoff still derives “Update complete” from exit zero and serializes ok without the deferred work. Relaunched Desktop shows a warning only for ok && manual; plain ok is log-only.
Observed in the independently inspected Desktop build-fail CI job: the real builder fails, the receipt records build, hermes update exits zero, and the relaunched app logs detached update finished OK (branch main). The handoff publishes done and closes its update UI, so the deferred build never reaches the existing manual-action warning path.
RCA / repair: successful command status is being projected into “all update work finished.” Read the correlated terminal receipt and propagate its owed build/result into a success-with-follow-ups/manual-action result. Preserve exit zero; expose the remaining build/relaunch requirement in the Desktop UI. Inspect the Windows and SSH handoff projections for the same lost fields before declaring this consumer class closed.
Regression: change the existing build-fail E2E to require successful committed-code status, preserved previous launchable app, an owed build, and a visible deferred-build warning. Merely replacing its old FAILED assertion with finished OK would retain the defect.
F7 — [P2] An unmapped stopped gateway's new restart debt disappears at the next startup
Verification now keeps stopped unmapped PIDs owed, but inventory persistence records only named gateways from the pre-update plan. A scoped argv-discovered gateway without socket/status/profile-PID metadata can be found and stopped by the manual restart path while the plan has no named gateway row. Its obligation therefore remains inventory-less.
Observed: execute actual collection → manual-stop bookkeeping → receipt recording → verification with such a scoped PID (signal effect mocked), then invoke the real startup debt query. Immediately after verification the marker exists and a gateway_restart follow-up is recorded. On the next query, gatewayless discharge sees no profile status or mapped runtime, clears the marker, and reports no owed restart. Output: inventoried_gateway_count=0, owed_after_verification=true, startup_reports_owed=false, owed_after_startup_probe=false. The normal earlier restart receipt also has incomplete=false; it does not rescue the later stopped-PID debt.
RCA / repair: the persisted owed set comes from a narrower discovery authority than the restart effect. Record an explicit unresolved stopped-gateway obligation from the actual restart outcome, with observed identity/scope where available; do not invent a named profile or let lack of metadata prove recovery. Keep the restart warning until successor evidence or an explicit recovery transition settles it.
Regression: extend the new unmapped-stop test beyond same-run cleared == [] to a fresh startup and an already-current update. Empty discovery must retain this recorded debt. This is an unclosed V7/V12 guarantee, not newly introduced process stopping.
RCA and SSA: close the outcome protocol across owners
The common root is a partial protocol migration. C3 changes the command verdict, while tail helpers, persistence schemas, fleet gates, IPC writers, and UI readers continue to use different meanings of “success.” A printed warning alone cannot preserve a retry obligation or reach a detached consumer.
| Boundary | Current failure | Complete repair/verification obligation |
|---|---|---|
| Tail helper → completion owner | F1: failure consumed, or profile_sync filtered out |
Explicit per-step results; consistent tail classification; debt survives subsequent launch |
| Receipt/log writer → profile/SSH reader | F2: root output, profile-only reader | Resolve update artifacts at root, while keeping launch-scoped correlation files at launch home |
| Completion verdict → fleet migration | F3: exit success substituted for runtime validity | Runtime-health gate independent of command exit and follow-up presence |
| Restart owner → notification IPC | F4: terminal marker contradicts receipt | One committed-update meaning at every writer and reader, including observation before wrapper exit |
| Running receipt → reclaimer/finalizer | F5: private carried schema not replayable | Same outstanding debt readable and retained before/after crash and receipt rotation |
| Receipt → detached Desktop result | F6: follow-up discarded from UI projection | Correlated receipt debt reaches the existing warning/manual-action path |
| Restart effect → persisted owed set → startup | F7: actual stopped gateway absent from plan-derived inventory | Persist the effect's owed set and require recovery evidence to settle it |
F1's sibling audit also found launcher publication warning/return branches and configured-feature dependency repair consuming PM failures. These are source-verified members of the same failure-result mismatch; they were not independently fault-injected here. Active/sibling config checks, profile enumeration/skill seeding/env backfill/Honcho sync must be included in the accumulated result protocol, with independent profile progress preserved.
The diagrams below model the paths described in the evidence. Their embedded legends identify node roles; evidence confidence remains explicit in the prose. The original diagrams were rendered with Mermaid CLI to SVG and PNG and visually inspected before this link/palette update.
flowchart TD
C["Committed checkout"] --> T["Independent completion<br/>steps"]
T --> E["Step error escapes"]
E --> O["Follow-up and retry debt"]
O --> R["Next launch retries"]
T --> H["Helper consumes error"]
H --> N["Returns with no<br/>follow-up"]
N --> S["Stamp written; debt<br/>cleared"]
T --> V["SQLite verdict false"]
V --> A["Caller passes true"]
A --> M["Topology migration runs"]
subgraph QA_LEGEND["Legend — roles; evidence stays explicit"]
Q_teal["Durable state"]
Q_amber["Transitions / pending"]
Q_purple["Ownership / policy"]
Q_red["Failures"]
end
classDef teal fill:#134e4a,stroke:#2dd4bf,color:#f0fdfa,stroke-width:2px
class C,Q_teal teal
classDef amber fill:#451a03,stroke:#fbbf24,color:#fffbeb,stroke-width:2px
class T,O,R,Q_amber amber
classDef purple fill:#3b0764,stroke:#c084fc,color:#faf5ff,stroke-width:2px
class V,Q_purple purple
classDef red fill:#450a0a,stroke:#f87171,color:#fef2f2,stroke-width:2px
class E,H,N,S,A,M,Q_red red
click C "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/update_completion.py#L197-L217" "Pinned source"
click T "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/source_completion.py#L103-L145" "Pinned source"
click E "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/update_cmd_maint.py#L993-L1000" "Pinned source"
click O "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/source_completion.py#L103-L145" "Pinned source"
click R "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/venv_sync.py#L398-L417" "Pinned source"
click H "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/update_cmd_config.py#L179-L207" "Pinned source"
click N "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/source_completion.py#L118-L145" "Pinned source"
click S "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/update_completion.py#L209-L213" "Pinned source"
click V "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/source_completion.py#L139-L145" "Pinned source"
click A "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/update_completion.py#L236-L238" "Pinned source"
click M "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/update_cmd_fleet.py#L2124-L2133" "Pinned source"
Source paths: completion ownership, tail classification, runtime gate.
flowchart TD
P["Successful update with<br/>debt"] --> D["Root receipt"]
D --> S["SSH reads profile<br/>directory"]
S --> X["Missing correlated<br/>receipt"]
P --> Z["Exit code zero"]
Z --> U["Desktop reports complete"]
D --> B["Running carried debt"]
B --> K["Updater interrupted"]
K --> N["Next begin ignores<br/>carried rows"]
N --> L["Restart debt disappears"]
subgraph QA_LEGEND["Legend — roles; evidence stays explicit"]
Q_blue["Entry / observations"]
Q_teal["Durable state"]
Q_green["Successful outcomes"]
Q_gray["Context / transient"]
Q_red["Failures"]
end
classDef blue fill:#172554,stroke:#60a5fa,color:#eff6ff,stroke-width:2px
class P,Q_blue blue
classDef teal fill:#134e4a,stroke:#2dd4bf,color:#f0fdfa,stroke-width:2px
class D,B,Q_teal teal
classDef green fill:#052e16,stroke:#4ade80,color:#f0fdf4,stroke-width:2px
class Z,Q_green green
classDef gray fill:#1e293b,stroke:#94a3b8,color:#f8fafc,stroke-width:2px
class K,Q_gray gray
classDef red fill:#450a0a,stroke:#f87171,color:#fef2f2,stroke-width:2px
class S,X,U,N,L,Q_red red
click P "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/update_completion.py#L197-L217" "Pinned source"
click D "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/update_receipt.py#L196-L203" "Pinned source"
click S "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/apps/desktop/electron/managed-ssh-update.ts#L418-L426" "Pinned source"
click X "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/apps/desktop/electron/managed-ssh-update.ts#L673-L681" "Pinned source"
click Z "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/scripts/desktop-update/posix.sh#L887-L889" "Pinned source"
click U "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/scripts/desktop-update/posix.sh#L887-L889" "Pinned source"
click B "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/update_receipt.py#L288-L302" "Pinned source"
click K "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/update_receipt.py#L288-L302" "Pinned source"
click N "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/update_receipt.py#L288-L302" "Pinned source"
click L "https://github.com/NousResearch/hermes-agent/blob/12ca0d5cd962865aadd6d48a00394208814f451d/hermes_cli/update_receipt.py#L432-L439" "Pinned source"
Source paths: root receipt, SSH reader, receipt rotation, Desktop result consumer.
Validation and CI
The six runnable probe files below execute the reviewed Python functions or the exact embedded SSH observation script. They use temporary homes and explicit substitutes for service/process effects, migration effects, unrelated maintenance, and failed reminder transfer. They do not establish live native-systemd/launchd/Windows behavior. F6 additionally has real Desktop build-failure E2E evidence in the linked CI job.
A focused local selection across test_update_post_commit_followups.py, test_source_completion_stamp.py, test_source_build.py, and test_update_completion_process.py passed 46 tests, with 3 skipped and 5 deselected, on Python 3.12.14. The full selection initially left five process/liveness cases unvalidated because this scratch runtime lacks the matching psutil/native dependency environment and hit the test live-system guard; those cases are not claimed passing. The component probes above are the finding evidence, rather than a green-suite argument.
The single PR commit is not green. The inspected CI lanes ran merge a8832dca0d074651fe2378c05070679f5ec43548 (12ca0d5 into the stated base):
| CI lane | Result and required treatment |
|---|---|
| Python shard 2 | Two resolver-allowlist failures: _build_desktop_product is unlisted and build_update_products is stale. Move the existing justified allowlist row with the extracted resolver, or adopt the canonical resolver. This is an extraction bookkeeping failure, not evidence of a newly unsafe lookup. |
| Git upgrade E2E | Conflict test still rejects rc=0 through reported_success. Its inspected receipt says the named stash is parked after restore conflicts; no patch-loss claim. Update the assertion to the explicit C3 outcome while retaining stash-recovery assertions. |
| Desktop build-fail E2E | Old FAILED expectation conflicts with exit-zero C3, but F6 remains. Assert successful code status and visible deferred build debt. |
To repeat the existing Desktop cell, use the repository's setup-pm test-environment toolchain, full git checkout, npm ci, usable bubblewrap namespaces, Xvfb/Electron dependencies, then from apps/desktop: CI=true HERMES_E2E_PYTHON=<test-python> HERMES_E2E_UPDATE_ROOT=/tmp/hdu xvfb-run -a --server-args="-screen 0 1280x1024x24" npx playwright test -c e2e/update/playwright.config.ts e2e/update/build-fail.spec.ts. The focused Git cell is scripts/run_tests.sh --include-integration tests/e2e/core/upgrade/git/test_local_modifications.py in the provisioned test environment.
Runnable reproduction receipts
Save the following files outside the checkout. Each accepts the checkout path as its first argument. The final assertions state the required contract and fail at this head after printing the observed state. They are review probes, not proposed committed tests. Run each with Python 3.12+ against a checkout pinned to 12ca0d5cd962865aadd6d48a00394208814f451d; fleet_status.py also accepts one of sqlite, failed-systemd, or failed-windows-resume. Baseline comparison for interrupted_manual_carry.py retained one obligation, while the head loses it.
tail_failures.py
Run: python tail_failures.py /absolute/path/to/checkout
"""Imported head helpers; inject ordinary filesystem failures below the adapter boundary.
Only unrelated maintenance, external config/profile sources and builds are mocked.
The completion, maintenance, config migration, profile-sync and receipt adapters are real.
"""
from contextlib import ExitStack, redirect_stdout
import io
import json
import os
from pathlib import Path
import sys
import tempfile
import types
from unittest.mock import patch
ROOT = Path(sys.argv[1]).resolve()
sys.path.insert(0, str(ROOT))
from hermes_cli import source_completion, source_build, update_cmd_maint as maint, update_cmd_config as migration, update_receipt
def stub_module(name, **attrs):
result = types.ModuleType(name)
result.__dict__.update(attrs)
return result
results = []
for scenario in ['config_write_failure', 'profile_skill_sync_failure']:
events = []
owed = []
def fail_migrate(**kw):
events.append('config migration attempted')
raise OSError('disk full')
def fail_sync(*a, **kw):
events.append('profile sync attempted')
raise OSError('disk full')
noop = lambda *a, **k: None
cfg = stub_module('hermes_cli.config', get_missing_env_vars=lambda **k: [],
get_missing_config_fields=lambda: [],
check_config_version=lambda **k: (1, 2) if scenario == 'config_write_failure' else (2, 2),
migrate_config=fail_migrate)
cmd = stub_module('hermes_cli.update_cmd',
_check_and_apply_config_migration=migration._check_and_apply_config_migration,
_migrate_sibling_profile_configs=lambda: [],
_no_prompt_git_kwargs=lambda: {},
_m=lambda: types.SimpleNamespace(PROJECT_ROOT=ROOT))
profile_rows = [] if scenario == 'config_write_failure' else [types.SimpleNamespace(name='team', path=ROOT / 'profiles/team')]
profiles = stub_module('hermes_cli.profiles', list_profiles=lambda: profile_rows,
seed_profile_skills=fail_sync, backfill_profile_envs=lambda **k: [])
stubs = {
'hermes_cli.config': cfg, 'hermes_cli.update_cmd': cmd, 'hermes_cli.profiles': profiles,
'hermes_cli.venv_sync': stub_module('hermes_cli.venv_sync', publish_launchers=noop),
'hermes_cli._subprocess_compat': stub_module('hermes_cli._subprocess_compat', expose_pm_git=noop),
'hermes_cli.source_stamp': stub_module('hermes_cli.source_stamp', write_source_stamp=lambda _: events.append('completed stamp written')),
'hermes_cli.gitlock': stub_module('hermes_cli.gitlock', fetch_full_commit_graph=lambda *a, **k: False),
'hermes_cli.model_catalog': stub_module('hermes_cli.model_catalog', seed_cache_from_checkout=lambda _: False),
'hermes_cli.macos_tcc_anchor': stub_module('hermes_cli.macos_tcc_anchor', ensure_tcc_anchor=noop),
}
with tempfile.TemporaryDirectory(prefix='completion-probe-') as directory, ExitStack() as stack:
stack.enter_context(patch.dict(os.environ, HERMES_HOME=directory, HERMES_DISABLE_LAZY_INSTALLS='1'))
stack.enter_context(patch.dict(sys.modules, stubs))
stack.enter_context(patch.object(source_build, 'build_update_products', noop))
stack.enter_context(patch.object(migration, '_restore_snapshot_safety_nets', noop))
for name in ['_verify_and_restore_state_dbs_post_update', '_invalidate_live_plugin_catalog_caches',
'_print_bundled_skills_sync_report', '_print_post_update_notices_and_self_heals']:
stack.enter_context(patch.object(maint, name, noop))
stack.enter_context(patch.object(maint, '_update_complete_message', lambda _: 'update complete'))
stack.enter_context(patch.object(maint, '_print_verified_update_completion', lambda _: True))
current_receipt = update_receipt.UpdateReceipt()
update_receipt._current.set(current_receipt)
out = io.StringIO()
with redirect_stdout(out):
result = source_completion._complete_locked(ROOT, desktop=False, assume_yes=True, gateway_mode=False,
pre_update_snapshot_id=None, pre_update_version=None,
completion_message=None, announce=None, followups=owed)
results.append(dict(scenario=scenario, completion_result=result, followups=owed,
receipt_followups=current_receipt.data.get('followups', []),
events=events, stdout=out.getvalue()))
update_receipt._current.set(None)
print(json.dumps(results, indent=2))
assert all(r["followups"] and "completed stamp written" not in r["events"] for r in results), "failed tail was stamped complete without retry debt"profile_sync_filter.py
Run: python profile_sync_filter.py /absolute/path/to/checkout
import contextlib, importlib, json, pathlib, sys, tempfile, types
from unittest.mock import patch
sys.path.insert(0, str(pathlib.Path(sys.argv[1]).resolve()))
import hermes_cli
from hermes_cli import source_completion, update_completion, update_receipt
def mod(name, **attrs):
m=types.ModuleType(name);m.__dict__.update(attrs);return m
with tempfile.TemporaryDirectory() as td, contextlib.ExitStack() as stack:
root=pathlib.Path(td);pending=root/'source-completion-pending';pending.write_text('owed')
main=mod('hermes_cli.main',PROJECT_ROOT=root)
cmd=mod('hermes_cli.update_cmd',_sweep_bytecode_after_update=lambda *a:None)
config=mod('hermes_cli.update_cmd_config')
venv=mod('hermes_cli.venv_sync',clear_completion=lambda root:pending.unlink())
for name,m in [('main',main),('update_cmd',cmd),('update_cmd_config',config),('venv_sync',venv)]:
stack.enter_context(patch.dict(sys.modules,{m.__name__:m}))
stack.enter_context(patch.object(hermes_cli,name,m,create=True))
def incomplete(*a,followups,**kw):
followups.append(('profile_sync','disk full'));return False
stack.enter_context(patch.object(source_completion,'complete_source_checkout',incomplete))
stack.enter_context(patch.object(update_receipt,'record_stage',lambda *a,**kw:None))
update_completion._complete_selected(dict(source=str(root),sibling_snapshots=[],plan=None,
branch='main',desktop=False,assume_yes=True,gateway_mode=False,snapshot_id=None,
pre_update_version=None,no_gateway_restart=True))
print(json.dumps({'completion_result':False,'reported_followup':'profile_sync',
'retry_marker_exists':pending.exists(),'in_tail_followups':'profile_sync' in update_receipt.TAIL_FOLLOWUPS}))
assert "profile_sync" in update_receipt.TAIL_FOLLOWUPS, "profile_sync retry debt is not classified as owed tail work"receipt_paths.py
Run: python receipt_paths.py /absolute/path/to/checkout
"""Read-only PR 132386 root/profile receipt and log consumer regression.
Run from workspace: python receipt-analysis/reproduce_receipt_paths.py source
Only temporary fixture homes are written. Runs the PR's exact observation script.
"""
import contextlib
import io
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile
source = Path(sys.argv[1]).resolve()
text = (source / 'apps/desktop/electron/managed-ssh-update.ts').read_text()
script = text.split('const OBSERVATION_SCRIPT = String.raw`', 1)[1].split('`', 1)[0]
sys.path.insert(0, str(source))
from hermes_cli import logs
correlation = '12345678-1234-4678-9234-567812345678'
with tempfile.TemporaryDirectory(prefix='pr132386-receipt-') as td:
root = Path(td)
profile = root / 'profiles/coder'
profile.mkdir(parents=True)
os.environ['HERMES_HOME'] = str(profile)
(profile / ('.update_exit_code.' + correlation)).write_text('0')
receipt = {'update_id': 'sample', 'correlation_id': correlation, 'outcome': 'success',
'started_at': '2026-10-03T18:00:00Z', 'finished_at': '2026-10-03T18:01:00Z'}
receipts = root / 'logs/update_receipts'
receipts.mkdir(parents=True)
(receipts / 'update_sample.json').write_text(json.dumps(receipt))
(root / 'logs/update.log').write_text('=== hermes-update completed ' + 'a' * 32 + ' ===\n')
observed = subprocess.run([sys.executable, '-c', script, str(profile), correlation],
text=True, capture_output=True, check=True)
root_observation=json.loads(observed.stdout)
print('ROOT_RECEIPT_PROFILE_LAUNCH', observed.stdout.strip())
capture = io.StringIO()
log_exit = 0
with contextlib.redirect_stdout(capture):
try:
logs.tail_log('update')
except SystemExit as error:
log_exit = error.code
print('ROOT_UPDATE_LOG_PROFILE_READER', json.dumps({'exit_code': log_exit, 'output': capture.getvalue()}))
old = profile / 'logs/update_receipts'
old.mkdir(parents=True)
(old / 'update_sample.json').write_text(json.dumps(receipt))
observed = subprocess.run([sys.executable, '-c', script, str(profile), correlation],
text=True, capture_output=True, check=True)
print('PROFILE_RECEIPT_CONTROL', observed.stdout.strip())
assert root_observation["receipt"] is not None, "root terminal receipt was invisible to profile launch"
assert log_exit == 0, "root update log was invisible to profile launch"interrupted_manual_carry.py
Run: python interrupted_manual_carry.py /absolute/path/to/checkout
"""PR 132386: a killed running receipt drops untransferred manual-serve debt.
Run: python receipt-analysis/reproduce_interrupted_manual_carry.py source
Uses real receipt begin/persist/reconcile/finalize, with only unavailable identity
and the failing manual-reminder write supplied as deterministic boundary probes.
"""
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile
source = Path(sys.argv[1]).resolve()
if len(sys.argv) > 2:
os.environ['HERMES_HOME'] = sys.argv[2]
sys.path.insert(0, str(source))
from hermes_cli import update_receipt as ur, update_serve_obligations as uso
# Failed transfer is the state pending_manual_serves specifically preserves.
uso.defer_manual_serve = lambda row, **kw: False
ur._code_identity = lambda **kw: {}
# The first child exited before the second starts: its owner is gone.
ur._owner_alive = lambda record: False
ur.begin_update_receipt()
if sys.argv[3] == 'crash':
os._exit(7)
path = ur.finalize_update_receipt('success')
print('FINALIZED', path)
raise SystemExit(0)
with tempfile.TemporaryDirectory(prefix='pr132386-manual-carry-') as td:
home = Path(td)
directory = home / 'logs/update_receipts'
directory.mkdir(parents=True)
row = {'kind': 'serve', 'supervisor': 'manual-serve', 'restart_via': 'respawn-argv',
'pid': 40001, 'profile': 'default', 'detail': {}}
(directory / 'latest.json').write_text(json.dumps({'outcome': 'success', 'pending_manual_serves': [row]}))
before = json.loads((directory / 'latest.json').read_text())
crashed = subprocess.run([sys.executable, __file__, str(source), td, 'crash'], capture_output=True, text=True)
running = json.loads((directory / 'latest.json').read_text())
finished = subprocess.run([sys.executable, __file__, str(source), td, 'finish'], capture_output=True, text=True)
after = json.loads((directory / 'latest.json').read_text())
print(json.dumps({'first_child_exit': crashed.returncode, 'second_child_exit': finished.returncode,
'before_pending_count': len(before.get('pending_manual_serves', [])),
'running_carried_count': len(running.get('carried_manual_serves', [])),
'after_pending_count': len(after.get('pending_manual_serves', [])),
'after_carried_count': len(after.get('carried_manual_serves', [])),
'final_outcome': after['outcome'], 'second_child_stdout': finished.stdout,
'second_child_stderr': finished.stderr}, indent=2))
assert len(after.get("pending_manual_serves", [])) == 1, "interrupted receipt discarded owed manual restart"fleet_status.py
Run: python fleet_status.py /absolute/path/to/checkout
"""Isolated correctness probes against PR 132386 source; no live gateways or services."""
import contextlib
import importlib
import io
import json
from pathlib import Path
import sys
import tempfile
import types
from unittest.mock import patch
ROOT = Path(sys.argv[1]).resolve()
sys.path.insert(0, str(ROOT))
import hermes_cli
from hermes_cli import update_cmd_fleet as fleet
from hermes_cli import update_completion as completion
from hermes_cli import update_receipt as receipt
from hermes_cli import update_cmd_maint as maintenance
from hermes_cli import source_completion
from hermes_cli import update_cmd_windows as windows
def stub_module(name, **attributes):
module = types.ModuleType(name)
module.__dict__.update(attributes)
return module
def outcome(**overrides):
values = dict(incomplete=False, phase_errors=[], pre_restart_gateway_pids=[],
restarted_services=[], failed_or_stale_units=[], relaunched_profiles=[],
externally_supervised_profiles=[], killed_pids=set())
values.update(overrides)
return fleet._GatewayRestartOutcome(**values)
def request(home):
return dict(source=str(ROOT), home=str(home), sibling_snapshots=[], plan=None,
branch="main", desktop=False, assume_yes=True, gateway_mode=True,
snapshot_id=None, pre_update_version="old", windows_resume=None,
receipt={"update_id": "fleet-c3-review-probe"})
def run_case(name):
with tempfile.TemporaryDirectory(prefix="fleet-c3-probe-") as temp, contextlib.ExitStack() as stack:
home = Path(temp)
events = {"followups": [], "terminal_outcomes": [], "migrations": []}
fake_main = stub_module("hermes_cli.main", PROJECT_ROOT=ROOT)
fake_config = stub_module("hermes_cli.update_cmd_config")
fake_venv = stub_module("hermes_cli.venv_sync", clear_completion=lambda root: None)
fake_gateway = stub_module(
"hermes_cli.gateway", is_macos=lambda: False,
find_gateway_pids=lambda **kw: [], find_profile_gateway_processes=lambda **kw: [],
_prepare_profile_gateway_update_restart=lambda *a: None,
_get_service_pids=lambda **kw: set(), _wait_for_gateway_exit=lambda **kw: None,
)
fake_migrate = stub_module(
"hermes_cli.gateway_migrate",
maybe_auto_migrate_after_update=lambda: events["migrations"].append("invoked"),
)
fake_update = stub_module(
"hermes_cli.update_cmd", get_hermes_home=lambda: home,
_m=lambda: types.SimpleNamespace(_fleet_probe_expected_runtimes=fleet._fleet_probe_expected_runtimes),
_sweep_bytecode_after_update=lambda branch: None,
_fleet_restart_skip_reason=lambda plan: None,
_pending_fleet_restart_needed=lambda: False,
_surviving_pre_update_serve_runtimes=lambda plan: [],
_warn_stale_serve_runtimes=lambda rows: None,
_resume_windows_gateways_and_merge_outcome=lambda *a: None,
_resume_windows_gateways_after_update=lambda *a: None,
_write_gateway_update_exit_code=fleet._write_gateway_update_exit_code,
)
modules = {m.__name__: m for m in [fake_main, fake_config, fake_venv, fake_gateway, fake_migrate, fake_update]}
stack.enter_context(patch.dict(sys.modules, modules))
for attr, module in [("main", fake_main), ("update_cmd", fake_update), ("update_cmd_config", fake_config)]:
stack.enter_context(patch.object(hermes_cli, attr, module, create=True))
stack.enter_context(patch.object(fleet, "_print_legacy_units_warning", lambda: None))
stack.enter_context(patch.object(fleet, "_collect_fleet_snapshot", lambda *a: [
dict(profile="default", pid=1001, state="current", code_sha="head")]))
stack.enter_context(patch.object(fleet, "_named_gateways_still_owed", lambda: False))
stack.enter_context(patch.object(fleet, "_clear_fleet_restart_pending_marker", lambda: None))
stack.enter_context(patch.object(maintenance, "_refresh_dashboard_after_update", lambda **kw: None))
stack.enter_context(patch.object(receipt, "record_stage", lambda *a, **kw: None))
stack.enter_context(patch.object(receipt, "record_followup", lambda step, reason, **kw: events["followups"].append(step)))
stack.enter_context(patch.object(receipt, "finalize_update_receipt", lambda result, **kw: events["terminal_outcomes"].append(result)))
stack.enter_context(patch.object(receipt, "finalize_pending_update_receipt", lambda *a: None))
stack.enter_context(patch.object(completion, "_resume_receipt", lambda *a: None))
stack.enter_context(patch.object(completion, "_read_terminal_receipt", lambda *a: {"outcome": "success"}))
stack.enter_context(patch.dict(sys.modules, {"pm.receipt": stub_module("pm.receipt", accept_worker_receipt=lambda *a: None)}))
stack.enter_context(patch.object(fleet, "_record_owed_gateway_inventory", lambda plan: None))
if name == "sqlite":
def false_completion(*a, **kw):
receipt.record_followup("sqlite_runtime", "unsafe SQLite runtime")
return False
stack.enter_context(patch.object(source_completion, "complete_source_checkout", false_completion))
fake_update._restart_gateway_fleet_after_update = lambda *a: outcome()
elif name == "failed-systemd":
stack.enter_context(patch.object(source_completion, "complete_source_checkout", lambda *a, **kw: True))
stack.enter_context(patch.object(fleet, "_gateway_drain_budget", lambda: 0.0))
stack.enter_context(patch.object(fleet, "_scoped_manual_gateway_pids", lambda pids, **kw: pids))
stack.enter_context(patch.object(fleet, "_restart_manual_gateways", lambda *a: None))
stack.enter_context(patch.object(fleet, "_warn_incomplete_gateway_fleet_restart", lambda *a: None))
stack.enter_context(patch.object(fleet, "_force_kill_stuck_gateways", lambda *a: None))
def failed_systemd(restarted, failed, *a):
failed.append("hermes-gateway.service")
stack.enter_context(patch.object(fleet, "_restart_systemd_gateway_units", failed_systemd))
fake_update._restart_gateway_fleet_after_update = fleet._restart_gateway_fleet_after_update
else:
stack.enter_context(patch.object(source_completion, "complete_source_checkout", lambda *a, **kw: True))
def failed_resume(token):
raise RuntimeError("gateway service restart failed")
fake_update._m = lambda: types.SimpleNamespace(
_fleet_probe_expected_runtimes=fleet._fleet_probe_expected_runtimes,
_resume_windows_gateways_after_update=failed_resume,
)
fake_update._restart_gateway_fleet_after_update = lambda *a: outcome()
fake_update._resume_windows_gateways_and_merge_outcome = windows._resume_windows_gateways_and_merge_outcome
fake_update._verify_fleet_after_update = fleet._verify_fleet_after_update
captured = io.StringIO()
with contextlib.redirect_stdout(captured):
final_code = completion._finish(request(home), home / "result.json")
events.update(case=name, gateway_marker=(home / ".update_exit_code").read_text().strip(),
completion_exit_code=final_code, response=json.loads((home / "result.json").read_text()),
stdout=captured.getvalue())
assert final_code == 0
if name == "sqlite":
assert events["followups"] == ["sqlite_runtime"]
assert events["migrations"] == ["invoked"]
assert events["terminal_outcomes"] == ["success"]
else:
assert events["gateway_marker"] == "1"
assert events["followups"] == ["gateway_restart"]
assert events["terminal_outcomes"] == ["success"]
return events
if __name__ == "__main__":
cases=[sys.argv[2]] if len(sys.argv)>2 else ["sqlite","failed-systemd","failed-windows-resume"]
results=[run_case(name) for name in cases]
print(json.dumps(results, indent=2))
for row in results:
if row["case"] == "sqlite":
assert row["migrations"] == [], "unsafe runtime reached topology migration"
else:
assert row["gateway_marker"] == "0", "successful committed update published failure IPC"unmapped_debt.py
Run: python unmapped_debt.py /absolute/path/to/checkout
"""Show the new unmapped-stop follow-up loses its inventory-less obligation on startup."""
import contextlib
import io
import json
from pathlib import Path
import sys
import tempfile
import types
from unittest.mock import patch
ROOT = Path(sys.argv[1]).resolve()
sys.path.insert(0, str(ROOT))
import hermes_cli
from hermes_cli import update_cmd_fleet as fleet
from hermes_cli import update_host_obligation as host
from hermes_cli import update_inventory as inventory
from hermes_cli import update_receipt as receipt
from hermes_cli import update_cmd_maint as maintenance
from hermes_cli import update_cmd_fleet_gatewayless as gatewayless
def main():
with tempfile.TemporaryDirectory(prefix="unmapped-obligation-probe-") as temp, contextlib.ExitStack() as stack:
home = Path(temp)
marker = home / "host-update-restart.json"
marker.write_text(json.dumps({"version": 1, "expected_sha": "head"}))
events = {"followups": [], "signals": []}
fake_update = types.ModuleType("hermes_cli.update_cmd")
fake_update.get_hermes_home = lambda: home
fake_update._current_checkout_sha = lambda: "head"
fake_update._m = lambda: types.SimpleNamespace(
_fleet_probe_expected_runtimes=fleet._fleet_probe_expected_runtimes,
_clear_marker_file=lambda path, **kw: path.unlink(missing_ok=True),
)
fake_update._surviving_pre_update_serve_runtimes = lambda plan: []
fake_update._warn_stale_serve_runtimes = lambda rows: None
fake_gateway = types.ModuleType("hermes_cli.gateway")
fake_gateway.find_gateway_pids = lambda **kw: [101]
fake_gateway.find_profile_gateway_processes = lambda **kw: []
fake_gateway._prepare_profile_gateway_update_restart = lambda *a: None
fake_gateway._get_service_pids = lambda **kw: set()
fake_gateway._wait_for_gateway_exit = lambda **kw: None
fake_status = types.ModuleType("gateway.status")
fake_status.read_runtime_status = lambda path: None
fake_status.live_gateway_pid_for_home = lambda home: None
stack.enter_context(patch.dict(sys.modules, {fake_update.__name__: fake_update, fake_gateway.__name__: fake_gateway,
fake_status.__name__: fake_status}))
stack.enter_context(patch.object(hermes_cli, "update_cmd", fake_update, create=True))
stack.enter_context(patch.object(host, "host_obligation_path", lambda: marker))
stack.enter_context(patch.object(fleet, "_current_checkout_sha", lambda: "head"))
stack.enter_context(patch.object(fleet, "_scoped_manual_gateway_pids", lambda pids, **kw: pids))
stack.enter_context(patch.object(fleet.os, "kill", lambda pid, sig: events["signals"].append(pid)))
stack.enter_context(patch.object(fleet, "_print_legacy_units_warning", lambda: None))
stack.enter_context(patch.object(fleet, "_collect_fleet_snapshot", lambda *a: []))
stack.enter_context(patch.object(maintenance, "_refresh_dashboard_after_update", lambda **kw: None))
stack.enter_context(patch.object(receipt, "collect_fleet_versions", lambda **kw: []))
stack.enter_context(patch.object(receipt, "record_followup", lambda step, reason, **kw: events["followups"].append(step)))
stack.enter_context(patch.object(receipt, "record_gateway_restart", lambda **kw: events.update(gateway_restart_record=kw)))
stack.enter_context(patch.object(receipt, "finalize_update_receipt", lambda *a, **kw: None))
# Host evidence after the unmapped process cleanly stopped: no published profile record
# and no mapped runtime. Same identity omission that made this PID unmapped before stop.
stack.enter_context(patch.object(receipt, "_profile_homes", lambda: [("default", home)]))
stack.enter_context(patch.object(receipt, "_socket_identity", lambda home: None))
stack.enter_context(patch.object(inventory, "_supervisor_classifier", lambda: lambda pid: "manual"))
stack.enter_context(patch.object(inventory, "collect_runtime_inventory", lambda: inventory.UpdatePlan(runtimes=[])))
plan = inventory.UpdatePlan()
inventory._collect_gateway_runtimes(plan, [("default", home)], set())
events["inventoried_gateway_count"] = len(plan.runtimes)
out = fleet._GatewayRestartOutcome(incomplete=False, phase_errors=[], pre_restart_gateway_pids=[101],
restarted_services=[], failed_or_stale_units=[], relaunched_profiles=[],
externally_supervised_profiles=[], killed_pids=set())
output = io.StringIO()
with contextlib.redirect_stdout(output):
fleet._restart_manual_gateways(out, 0.0)
out.record_receipt() # Normal restart-phase order, before verification adds incomplete.
fleet._verify_fleet_after_update(out, _pre_update_plan=plan, _windows_gateway_resume=None, update_complete=True)
events["owed_after_verification"] = marker.exists()
events["record_after_verification"] = json.loads(marker.read_text())
final_receipt = {"outcome": "success", "fleet": [], "plan": {"runtimes": []},
"gateway_restart": events["gateway_restart_record"],
"followups": [{"step": step} for step in events["followups"]]}
events["startup_reports_owed"] = fleet._update_owes_fleet_restart(receipt=final_receipt, pending_manual=[])
events["owed_after_startup_probe"] = marker.exists()
events["stopped_unmapped_pids"] = sorted(out.stopped_unmapped_pids)
events["stdout"] = output.getvalue()
assert events["signals"] == [101]
assert events["followups"] == ["gateway_restart"]
assert events["owed_after_verification"]
assert not events["startup_reports_owed"]
assert not events["owed_after_startup_probe"]
print(json.dumps(events, indent=2))
assert events["startup_reports_owed"] and marker.exists(), "startup erased stopped-gateway debt"
if __name__ == "__main__":
main()
teknium1
force-pushed
the
fix/update-post-commit-never-fails
branch
from
October 4, 2026 05:16
fb7af08 to
729d352
Compare
teknium1
force-pushed
the
fix/update-post-commit-never-fails
branch
from
October 4, 2026 07:48
9f15458 to
2e545ae
Compare
andrexibiza
reviewed
Oct 4, 2026
andrexibiza
left a comment
andrexibiza
left a comment
There was a problem hiding this comment.
Current-head deep security and code review: one P1 and eleven P2s remain
Reviewed exact head e095f5b4649d7250ce088fc24ce42ce4882486d3 against true merge base 819cc3cbe02104c420ea32f1f1a924247d42eaf8; the inspected default-branch snapshot was 8b66a51036c1e20920a17cdd049fdf55c968d683. I read the 49-file diff, the complete changed production modules, their process/receipt/UI consumers, all prior findings, the repository security policy, and the relevant changed tests. The two existing reviews target older head 12ca0d5; there was no formal review at this head before this one.
I did not confirm an OS-isolation escape, unauthorized external-surface access, or credential exfiltration under SECURITY.md §3.1. The security-relevant failures below are updater integrity, process authority, durable-evidence, and availability failures on supported local/remote update paths. They remain blocking under the repository's own C3 and receipt contracts.
P1 — the isolated Windows completion process cannot print the protocol it now relies on
record_followup prints ⚠, and normal completion prints ✓, from the selected child that run_completion starts with -I and stdout redirected to a pipe. On the repository's managed Python 3.14.7 under native Windows, that exact isolated/redirected shape selected cp1252; printing ✓ Update complete! raised UnicodeEncodeError and exited 1. Adding -X utf8 was the healthy control (encoding=utf-8, exit 0).
The canonical Windows run reproduced the consequence: 408 passed, 38 skipped, 8 failed, all in test_update_completion_process.py; the failures include the healthy checkout transition, post-commit build follow-up, dependency follow-up, and both progress cells. The merge-base version of the same file also failed 8/23 on this host, so the transport flaw is inherited, but this PR makes the Unicode follow-up/result protocol load-bearing while claiming that no post-commit step fails the update. The selected child needs an encoding fixed independently of the host code page (for example -X utf8) and a native-Windows redirected-stdout regression with the inherited PYTHON* environment absent.
F1 — P2 — profile synchronization still returns success without durable debt
The version-only active config write is repaired at this head: an actual late write failure now becomes config_migration, keeps the tail pending, and withholds the stamp. The rest of F1 remains. _sync_profiles_after_update consumes per-profile skill, env-backfill, and Honcho failures and returns normally, so the new owed_step sees success. A real injected seed_profile_skills OSError("disk full") printed work: error (...), but both caller and receipt follow-ups stayed empty and maintenance returned true.
There is a second independent hole: TAIL_FOLLOWUPS omits profile_sync. Supplying an escaped profile_sync follow-up made _complete_selected clear source-completion-pending. The repair needs explicit per-profile results to reach the owner and profile_sync to remain tail debt until a later successful retry.
F2 — P2 — named-profile managed SSH still reads receipts from the old location
The CLI/debug log readers are repaired, but F2 remains on the managed SSH path. The producer now writes update receipts at the root home, while the exact embedded observer still scans home/logs/update_receipts. For <root>/profiles/research, a correlated root receipt plus profile-local exit marker produced exitCode: 0, receipt: null; putting the identical receipt at the old profile path made it visible. The waiter then rejects the committed update. Receipt lookup needs the already-derived install_root, while status/intent files remain launch-profile scoped.
F4 — P2 — restart debt still rewrites committed-success gateway IPC to failure
The selected completion publishes gateway IPC 0, but failed systemd/abort/Windows-resume branches still write 1 at update_cmd_fleet.py:1710-1711, 1789-1793, and update_cmd_windows.py:1337-1342. The new verifier then records gateway_restart, finalizes outcome: success, and returns.
Executed production routing for failed-systemd and failed-Windows-resume produced the same contradiction: completion returned success / receipt success + gateway_restart, while .update_exit_code contained 1. The gateway notification reader can consume that immediately; a later wrapper write cannot repair a notice already sent. Every post-commit IPC writer must preserve committed success and carry restart debt only through the follow-up/obligation channels; the regression needs the notification reader interleaved before final return.
F7 — P2 — stopped unmapped gateway debt is erased by the next startup probe
The same-run change at update_cmd_fleet_verify.py:310-327 correctly records a stopped unmapped PID as restart debt. The persistence helper, however, records only identified gateway rows from the pre-update plan. A scan-only gateway therefore leaves an inventory-less host obligation. On the next startup, an empty status/inventory state is accepted by the inherited gateway-less discharge.
The composed probe observed: armed before verification = true; armed after verification = true; receipt follow-up = gateway_restart; startup reports owed = false; marker armed after startup = false. Persist an unresolved stopped-runtime identity/effect from the actual restart outcome and require successor or explicit recovery evidence before discharge. The current unmapped test stops at same-run non-clear and does not exercise this transition.
P2 — receipt storage failure still turns a committed update into exit 1
C3 depends on the receipt, but receipt publication itself is still allowed to change the command verdict. With a writable root receipt directory, the exact new _settle_after_commit path returned 0 with a terminal receipt. Replacing that directory with an ordinary file made finalize_update_receipt("success") warn and return None; _settle_after_commit then returned 1. The prepared _finish path has the same terminal-receipt dependency.
This is after the tree moved, so the output again says failure for committed code. Keep the fail-closed correlation requirement, but return the finalized in-memory correlated terminal result when archive publication fails, while preserving the loud evidence-storage warning and armed obligations. The regression needs writable and unwritable/disk-full receipt controls through both _settle_after_commit and _finish.
P2 — parent-side Windows recovery rewrites a terminal success back to running
After a dependency follow-up, the selected child can already have finalized the terminal receipt. If parent-side Windows resume then raises, record_followup mutates the parent's stale pre-child receipt context first. Because record_followup persists running state, it overwrites the child's terminal archive and latest.json; amend_terminal_followup then appends to that overwritten record.
The exact receipt-seam replay produced success → running, finished_at: null, and two windows_resume follow-ups, while _complete_source_update returned success. Amend the correlated terminal archive directly after child acceptance; do not call the generic running-context writer after a terminal receipt exists. A regression should assert one follow-up, terminal success, and unchanged finished_at after a parent resume failure.
P2 — interrupted autostash debt exists only in memory until another stage happens to persist it
record_user_action updates only the ContextVar clone. Unlike record_followup, it does not refresh the running archive. The completion process records the stash debt before product work; an interrupt during that window makes finalize_interrupted_update_receipt hydrate the older on-disk record and discard the action.
Executed sequence: begin → record_user_action("local_changes", ...) → inspect archive → interrupted finalize. The in-memory action was absent from the running file, and the terminal interrupted file still had no user_action. Persist the action immediately, then retain it when hydrating a further-along running record. The regression should interrupt between the action record and the next stage boundary and require the stash identity/instructions to survive.
P2 — the running-receipt owner test misclassifies live and recycled owners in opposite directions
The new _owner_alive has two independent identity errors. It skips a PID equal to the current process, so a nested begin_update_receipt() marks the still-open outer receipt interrupted despite the function's nested-update contract. It also validates pid with pid_create_time, then rechecks the same writer_pid with no creation time; a reused PID can overrule an already-proven incarnation mismatch.
Both were executed: nested begin changed the outer archive from running to interrupted; the reuse probe called the matcher with (4242, 10.0) → False and then (4242, None) → True, yielding _owner_alive == True. Record creation identity for every distinct writer, deduplicate identical PID roles, and treat the current process as live unless the receipt is explicitly the run being replaced. Regressions need nested same-process and reused-PID controls.
P2 — profile-scoped update locks do not serialize the new root-wide receipt pointer
UpdateLock resolves under the process profile home, while this PR moves all update receipts to one root directory and every profile mutates one checkout. Two named-profile locks were both acquired simultaneously at <root>/profiles/a/.hermes-update-in-progress and <root>/profiles/b/.hermes-update-in-progress. In separate receipt contexts, B finalized latest.json as success; a later A stage replaced it with A's running record.
The checkout-concurrency hole is inherited; the new shared receipt pointer makes the evidence collision immediate and reproducible. Serialize update ownership at the install/root/checkout identity and guard latest.json against older/in-progress writers replacing a newer terminal run. The regression needs two profile processes, one winner, and a terminal pointer that cannot regress.
P2 — Desktop recognizes only two textual forms of owed build debt
The new POSIX projection at posix.sh:891-900 requires the truncated warning line to contain desktop app build or Node dependencies. source_build_env(explicit=True) can fail before either named step; multiple aggregated failures can also push the Desktop phrase beyond record_followup's 500-character cap.
An exact-head probe made npm acquisition fail before dependency preparation. The real completion produced followups=[("build", "npm unavailable before Node dependency preparation")], no stamp, and the expected warning; the script's exact grep returned 1 while the Node dependencies control returned 0. The app therefore takes plain success although no Desktop rebuild ran. Project structured receipt data into the handoff result instead of parsing presentation text, and cover a generic pre-build failure plus a long compound failure. PR #132354 also modifies this file; the heads conflict on posix.sh, so this structured projection must survive that resolution.
P2 — a root completion marker can certify another profile's failed dashboard action
get_action_status now reads the shared root update.log and treats any durable completion ID as exit 0, without correlating it to this dashboard action. With profile B's local action log showing a later failure before the CLI tee and profile A's completion marker at root, the actual endpoint returned exit_code: 0 and A's action ID for B. Removing the root marker made the control return exit_code: null.
The checkout is shared, but action identity is not interchangeable: B's request did not complete. Persist/carry the action ID through the profile's process/result state and accept a root marker only when it equals that action. A regression needs two profile action logs and one shared root update log with distinct IDs.
Prior-finding reconciliation
- Earlier SQLite/topology P1: repaired. Both current and historical completion paths keep runtime safety separate from exit status; unsafe SQLite keeps exit 0 and vetoes migration, while the healthy control still enters migration.
- F1: partially repaired. Active version-write failure is owed; profile/sibling/config-check failure channels remain open as described above.
- F2: partially repaired. CLI/debug root log readers moved; managed SSH receipt lookup remains open.
- F4: still present, freshly reproduced at this head.
- F5 carried-manual-serve loss: repaired.
carried_manual_servessurvives running-record rotation and current readers consume it. - F6 lone Desktop compiler failure: repaired for that message shape; generic/truncated build debt remains open.
- F7: still present, freshly reproduced through the next startup transition.
Verification and limits
- Local canonical runner on Windows 11 / Python 3.14.7: 408 passed, 8 failed, 38 skipped across 16 touched test files. Fifteen files passed; all failures were in the fresh completion-process file and trace to the isolated child's
cp1252Unicode failure. Equivalent merge-base file: 12 passed, 8 failed, 3 skipped. - Current hosted checks at this head: 46 success, 24 skipped, 0 failing/pending. Hosted Windows being green did not exercise this machine's redirected
cp1252runtime. - Five deterministic disposable-home probes reproduced the receipt, marker, consumer, and profile-state transitions above. No real gateway/service was stopped and no production home was used.
- Related PR #132354 overlaps only
scripts/desktop-update/posix.sh;git merge-treereports a content conflict. A semantic scratch resolution preserving both the marker-lock work and this PR's C3 build projection passesbash -n. Native Linux handoff tests and the heavyweight Desktop E2E were not rerun on this Windows host; this PR's hosted Desktop build-fail E2E is green for the lone compiler-failure case.
Acceptance requires closing each surviving transition above, retaining the repaired SQLite/F5/lone-build behavior, and exercising the negative controls at the actual parent/child, profile/root, gateway-notification, and Desktop handoff boundaries.
teknium1
force-pushed
the
fix/update-post-commit-never-fails
branch
from
October 4, 2026 22:39
e46e325 to
b822fae
Compare
Open
13 of 14 tasks
Collaborator
|
The new raise in update_cmd_config.py:210 records the active migration failure, but also skips sibling migration and the snapshot safety-net checks below it. A controlled probe of _check_and_apply_config_migration() ran active migration, sibling migration and restoration with the original snapshot on the healthy path. When the active version write failed, neither of the latter steps ran. The next-launch completion command does not carry the original snapshot ID or sibling snapshot map, and the restoration helpers return immediately without a snapshot ID. Please keep the independent migration/restoration work running while accumulating the active failure, and preserve the original snapshot references for any deferred restoration. A regression should fail the active write, prove sibling and safety-net progress, then retry in a fresh process with the original snapshot evidence still available. |
teknium1
force-pushed
the
fix/update-post-commit-never-fails
branch
from
October 5, 2026 07:05
7138d12 to
2e3b716
Compare
kshitijk4poor
left a comment
kshitijk4poor
left a comment
There was a problem hiding this comment.
Reviewed at
b822fae6ba4; the branch has since been rebased onto main (head below); I have not re-reviewed it, so check each finding still applies.
Review at b822fae: request changes (rebase needed, plus three regressions)
The branch is 264 commits behind main. It conflicts with main in hermes_cli/AGENTS.md, with #132361 / #132346 in tests/hermes_cli/test_update_zip_completion.py, and with #132354 in scripts/desktop-update/posix.sh. The posix.sh clash is about meaning, not just text: #132354 reports post-commit work via add_warning → WARNINGS[] → "warnings":[…], while this PR uses DONE_NOTE only. I found no stale-base reverts: main's later changes in update_completion.py / update_cmd.py (libatomic pre-install, settle_partial_clone_maintenance) survive the merge.
Regressions vs main
- A gateway proven to run old code now passes silently.
update_cmd_fleet_verify.py~310-334 records agateway_restartfollow-up and finalizessuccess.posix.sh~891 only reacts to build follow-ups, so gateway_restart, windows_resume, channel_adoption and completion all show "Update complete." The gateway/updatewatcher reads.update_exit_code= 0. After this, the CLI startup warning is the only signal. Every post-commit follow-up should reach the Desktop result as a warning. - Managed SSH loses the receipt.
_receipt_dir()moved to the root home (update_receipt.py~200-207), butapps/desktop/electron/managed-ssh-update.ts:419still readshome/logs/update_receipts. A named-profile launch getsreceipt=null, so the update is rejected. .update_exit_codeis inconsistent. It still writes1after the commit point atupdate_finish.py~165 and themain.pyboundary (~2535/2546/2552). The IPC says 1 while the receipt sayssuccess.
Rebase against #132361
_failed_result(code or 1),_finish's exit-1BaseExceptionhandling, and themain()boundaryexcept BaseException → 1all conflict with_settle_after_commit.- The custody
refusal_notice()is gated onif code, so it would never print under C3. _settle_after_commitreturning 1 on a missing receipt trips #132361's "did not publish" check.- The
hermes_cli/AGENTS.mdline "A provably-stale gateway fails the update (exit 1)" must be rewritten to the C3 rule, not just merged.
Open threads
- Fixed:
- Still reproduce:
- F1:
owed_steponly sees escaped exceptions. - F2: SSH reader (regression 2 above).
- F5: carried rows.
- F6: only build follow-ups reach the Desktop.
- UTF-8 under
-I. - F4: IPC=1 (regression 3 above).
- F7: stopped-unmapped.
- Root
latest.jsonvs the per-profile lock. - Desktop phrase grep.
actions.pyroot log correlation.
- F1:
- Not traced: #8.
Looks good
- The Ctrl-C →
interrupted/130 split. - An unrestored stash stays
partial/1. - No
SystemExitescapes after the commit point. - Windows resume runs once.
Method: read the source at this head vs current main (93c9360), checked open review threads against this head, ran the touched-area test files on main + ①②④⑤⑥⑦⑧ merged (every failing file also fails on main or is environment-only: home-guard / macOS bash 3.2 mapfile), plus targeted probes.
teknium1
force-pushed
the
fix/update-post-commit-never-fails
branch
2 times, most recently
from
October 5, 2026 11:14
c67c824 to
1860f08
Compare
Collaborator
Author
|
Replies to @kshitijk4poor's review, item by item. Commits cited are on the current PR heads.
|
andrexibiza
reviewed
Oct 5, 2026
There was a problem hiding this comment.
Reviewed ce5028d against merge base 7157422. Four reproducible P2 issues remain. The intended C3 distinction—committed code succeeds while unfinished work remains owed—is sound, but action identity, client-visible debt, and recovery evidence still diverge at the consumers below. This is a COMMENT review, not an approval or a request-changes event.
R1 · P2 · The receipt fallback bypasses the new dashboard action correlation
Trigger: Profile B's update fails while its dashboard is offline and its in-memory subprocess result is lost. Profile A subsequently completes an update. B's local action log still names B, while the shared root log and latest receipt now belong to A. B's own failed receipt can still exist in the archive.
The new comparison at actions.py:374–379 correctly rejects A's log marker. But the route then loads the root-wide latest receipt unconditionally, and _completed_exit_code converts any success receipt to 0. The summary omits both receipt identities, so there is no matching check left. HERMES_ACTION_ID also does not enter receipt provenance: the receipt's launcher correlation reads HERMES_UPDATE_CORRELATION_ID only.
Executable probe: Use the real receipt writer to finalize B as failed, then A as success under their respective <root>/profiles/<name> homes; change HERMES_HOME back to B; write B's === hermes-update started ... <b-id> === profile header and A's root completion marker; clear the four action registries; query the authenticated /api/actions/hermes-update/status endpoint using TestClient. The identical producer/endpoint probe gives:
| Evidence | Merge base | Current head |
|---|---|---|
| B failed, A later succeeded, B has no retained process result | exit_code=null |
exit_code=0, no returned action_id |
| Remove the receipt through the reader boundary | exit_code=null |
exit_code=null |
| B's own matching completion marker, no receipt | — | exit_code=0, action_id=b-id |
This is a cross-profile regression from moving the receipt store to the root; the earlier root-log finding survives through a different evidence path. The web UI treats 0 as success, and Desktop also accepts it before its own receipt fallback. Desktop's separate receiptProvesOutcome check uses only a 60-second timestamp allowance, so merely correcting the backend exit mapping would leave another identity-free consumer.
Repair / regression contract: Persist the dashboard caller's action identity on the run, retrieve that action's terminal archive, and require an exact identity match before inferring its result. Keep unrelated latest receipts available as general history without letting them certify B. Cover A-success/B-unfinished, A-success/B-failed, rotated logs, a matching archived B receipt when A is latest, and the Desktop fallback. Preserve the distinct PM update identity.
R2 · P2 · Success-with-debt becomes plain success in dashboard and managed SSH clients
The dependency settlement deliberately publishes exit 0 / outcome success with a dependencies follow-up. The same applies to other owed steps. However, both the dashboard summary and the actual managed SSH observer discard followups and user_action. The local POSIX/Windows handoff's repaired text parser does not cover these paths.
Executable probe: Produce a terminal receipt through the real writer with record_followup("dependencies", "synthetic selected-interpreter sync failed"), outcome success, and a valid managed-SSH UUID. Execute the unchanged embedded Python observation program under a named-profile home with the matching correlated exit marker. Feed its actual JSON through the bundled production parseRemoteUpdateObservation, runManagedSshUpdate, and managedSshUpdateAllRow, injecting only transport/restore callbacks. Also query the actual dashboard status endpoint.
durable producer: followups=[dependencies], outcome=success
dashboard status receipt: followups absent
SSH observer receipt: followups absent, exitCode=0
Desktop managed result: ok=true, updateOk=true, outcome=updated
detail="Remote Hermes updated and every managed SSH profile is ready."
The source-backed web consumer likewise selects its success toast from exit_code === 0; the local Desktop backend consumer calls finishBackendApply(true) on the same condition. The execution above proves the SSH formatter, not native Electron rendering or a live SSH transaction. A healthy backend restoration does not prove that an owed config/build/dependency step finished.
Repair / regression contract: Carry structured follow-up and user-action facts through both summary protocols, their shared response types, and the final client result/toast. Keep C3's committed-success exit status and keep scope restoration independent, while reporting the owed step and its remedy. Exercise non-build debt with a healthy restored scope, ordinary debt-free success, multiple owed steps, and partial/user-action receipts. Do not rely on scraping the remote log to recover fields the observer dropped.
R3 · P2 · An existing restart inventory prevents recording newly stopped unmapped gateways
_record_owed_gateway_inventory:144–145 returns whenever the obligation already has an inventory. A same-SHA catch-up intentionally preserves the standing record. If that retry also stops an unmapped gateway and the mapped gateway remains down, verification records a new gateway_restart follow-up but never adds the newly stopped PID to the durable owed set. Once the mapped gateway recovers, startup can clear the old inventory without any successor for the unmapped one.
Executable probe: Use the real host-obligation writer with SHA S and mapped gateway A; invoke the actual _verify_fleet_after_update with a plan for A and restart bookkeeping containing stopped-unmapped PID 7999. Inject external fleet discovery as A-down, plus dashboard/serve probes and signal delivery boundaries. The verifier finalizes a success receipt with gateway_restart. Then let the actual startup settlement consumer observe only a current successor for A at S:
| Initial obligation | Inventory after verification | Startup after A recovers |
|---|---|---|
| No inventory (positive control) | A and stopped-unmapped 7999 | settled=false, still armed |
| Existing inventory containing A | A only; 7999 missing | settled=true, obligation removed |
No live service was stopped. This exercises the real verifier, durable store, and settlement consumer; fleet/process observations are injected. The important precondition is that A is still down during verification: when A is already current and no other error exists, the earlier named-gateway check can settle/re-arm the record and mask this omission.
Repair / regression contract: Monotonically add newly incurred obligations to a readable inventory for the same target SHA, preserving the old owed rows and restart provenance. Keep malformed/foreign-target records fail-closed. A mapped successor must not discharge newly stopped-unmapped debt; retain it through a fresh startup and same-SHA update until its distinct local successor is proved. The existing inventory-less test alone cannot cover this retry cell.
R4 · P2 · Recording an active migration failure skips independent restoration and loses its snapshot context
The new raise at update_cmd_config.py:210–211 escapes before sibling migration and _restore_snapshot_safety_nets(pre_update_snapshot_id). The caller records config_migration debt, but the pending-tail marker stores only text. The next-launch finish-update invocation and retry consumer carry neither the original active snapshot ID nor the sibling snapshot map. These restoration helpers return without an ID.
This confirms the existing migration/snapshot comment at the current head. It is separate from the deliberately best-effort profile-sync policy.
Executable probe: Seed actual active/sibling configs at v48 and one cron job in each home; create real pre-update quick snapshots; seed the valid-but-empty cron files that these safety nets explicitly protect; execute the real migration and backup code while raising OSError only at the active config's atomic_config_replace storage boundary. No migration or restore helper is replaced. Run the same probe on base and head, then retry head in a fresh process without snapshot arguments, as next-launch completion does:
| Result | Merge base | Current head | Fresh head retry after storage recovers |
|---|---|---|---|
| Active cron jobs restored | 1 | 0 | 0 |
| Sibling cron jobs restored | 1 | 0 | 0 |
| Sibling config version | 49 | 48 | 49 |
| Original snapshot available to the call | yes | yes, but never consumed | no |
The injected cron loss is the documented independent mutation-window trigger; the probe does not claim a config write failure itself deletes jobs. It shows this PR suppresses their existing repair and that a later successful migration does not recover it.
Repair / regression contract: Accumulate the active failure, execute independent sibling/snapshot work, then report the owed migration to the caller. If restoration itself must be deferred, durably retain the original snapshot references and consume them only after successful restoration. Cover active storage failure with real job/model snapshots, sibling progress, and a separate-process retry; preserve the new follow-up rather than swallowing the failure again.
Reconciliation of earlier material reports
Compared the two reviews at 12ca0d5, the review at e095f5b, the later stack review, all 34 inline comments/replies, the migration comment, and the author's itemized response. Superseded manifestations are not repeated as new blockers.
| Earlier issue | Current-head disposition |
|---|---|
| Original SQLite/runtime-safety P1 | Corrected: the completion retains the real/unknown verdict and topology migration requires it. The five fleet-completion cases pass locally. |
| Windows UTF-8 P1 | Withdraw the earlier production-failure claim: normal hermes_cli initialization already reconfigures streams, so the empty-package fixture was not sufficient. Both isolated child argvs now additionally contain -X utf8; the focused argv test passes. Native cp1252 Windows was not rerun here. |
| F1 swallowed profile sync; active version-write failure | Per-profile best effort is explicitly retained by design; escaping profile_sync is tail debt. Do not reopen that policy as a new defect. The active-write debt is now recorded, but its new early exit leaves R4. |
| F2 named-profile SSH receipt location | Corrected: the unchanged current observer finds the real root receipt in the local execution. R2 concerns fields lost after that successful lookup. |
| F4 gateway IPC overwritten by restart/late-boundary errors | Writers and command-scope terminal guard corrected; focused post-commit follow-up/gateway-marker tests pass. Historical takeover tests pass on Linux. Native Windows recovery remains outside local verification. |
| F5 interrupted carried manual-serve rows | Begin and readers now consume carried_manual_serves. The live-PID old-schema probe could not be validated in this environment; no claim of native closure is made. |
| F6 / Desktop phrase truncation / later review regression 1 | Local handoffs now parse every follow-up step and a whole Desktop-owed line; the eight handoff tests and focused protocol case pass. R2 is the uncovered dashboard/managed-SSH route. |
| F7 stopped-unmapped debt | Inventory-less path repaired and positive control retained; R3 reproduces the existing-inventory retry form. |
| Receipt-store failure changes exit status | In-memory terminal fallback added; focused settlement/finish storage-failure cases pass. This is evidence for same-process status, not a promise that an unwritable store has persisted a receipt. |
| Parent Windows recovery rewrites success / duplicate resume | Terminal-archive guard, child receipt adoption, and settled-token handling added. Relevant non-native cases pass; no native Windows reproduction here. |
| User action exists only in memory | Immediate running-record persistence added; the separate-process durability case passes. |
| Live/recycled receipt-owner identity | Own-PID and duplicate-identity logic changed, with writer incarnation recorded. Two local durability tests are inconclusive because of the PID/procfs mismatch described below. |
| Root latest.json vs profile-local locks | Still a stack prerequisite: #132365 is open at 7117ab64a4fb8ae87823aa5324674b09885cfd38, not included here. Land it before/with this PR and rerun two-profile ownership/receipt tests on the resulting tree. I did not certify the composite or duplicate the existing lock finding inline. |
| Root completion marker certifies another action | Log-only path corrected; R1 reproduces the receipt fallback and a base/head cross-profile difference. |
| Old stale-base/rebase and #132361 custody conflict reports | This head is rebased onto the pinned base; the obsolete standalone _failed_result/boundary reports do not describe this tree. The author's composite custody resolution and #132354 shell integration still need validation on the actual merged stack; no composite was executed in this review. |
Verification and limits
Used an isolated clean checkout, a PM-built dev/test environment with Python 3.14.7 for the repository's 3.14 pin, the canonical scripts/run_tests.sh, real ASGI endpoints and receipt/backup stores, and Node 24.19.0 to bundle/execute the actual SSH lifecycle module. No implementation changes or production service actions were made.
- Ten focused files: 107 passed, 6 failed, 7 skipped, plus one teardown error. The failures involve live PID/incarnation or child cleanup. This environment returns a namespace PID different from
/proc/self/status(os.getpid()=2versus procfsPid=16719in the diagnostic), sopsutil.Process(os.getpid())raisesNoSuchProcessand the live-system guard cannot identify test children. I do not count those failures as PR defects or claim those native identity cases passed. - Four additional migration/reconciliation/restart-obligation files: 92 passed. Fleet-completion/runtime-verdict file: 5 passed. Total passed test cases across those runs: 204, with the above unresolved environment failures retained.
- All four reported functional probes completed at this head; R1 and R4 include identical merge-base comparisons. R3 stubs external fleet/process observations; R2 stubs SSH transport and successful restoration. Neither is a live remote update or native Electron/Windows test.
- At the refreshed pinned head, GitHub reports 47 successful and 24 skipped checks, including All required checks pass. Green CI does not exercise the four demonstrated consumer/retry cells.
Receipt-to-consumer model
The diagram scopes R1/R2 and the repaired local handoff. Teal denotes durable state, blue incoming observations, green successful outcomes, red failures, and amber pending work. Evidence confidence is stated explicitly in the node labels and review text. Source anchors: receipt producer, dashboard consumer, SSH consumer, local handoff, lock prerequisite. Mermaid was parsed/rendered with mmdc and visually checked in a dark palette before publication.
%%{init: {'theme':'dark','themeVariables':{'background':'#0d1117','primaryTextColor':'#e6edf3','lineColor':'#94a3b8'}}}%%
flowchart TB
subgraph R["C3 evidence at ce5028d0d2dd"]
C["Committed code"]:::owner --> P["Success receipt with followups"]:::owner
P --> S["Shared root latest.json"]:::owner
S --> A["Unmatched receipt fallback"]:::failure
A --> B["Other dashboard action reports exit 0"]:::failure
P --> D["Dashboard and SSH summaries drop followups"]:::failure
D --> U["Client reports plain success"]:::failure
P --> T["Printed followup protocol"]:::normal
T --> H["Local Desktop handoff names debt"]:::repair
K["Shared update lock in prerequisite PR"]:::uncertain -.-> S
end
R ~~~ L
L["Legend<br/>Blue: entry / incoming<br/>Amber: transition / pending<br/>Teal: durable state<br/>Green: success<br/>Red: failure"]
classDef normal fill:#172554,stroke:#60a5fa,color:#eff6ff,stroke-width:2px
classDef owner fill:#134e4a,stroke:#2dd4bf,color:#f0fdfa,stroke-width:2px
classDef repair fill:#052e16,stroke:#4ade80,color:#f0fdf4,stroke-width:2px
classDef failure fill:#450a0a,stroke:#f87171,color:#fef2f2,stroke-width:2px
classDef uncertain fill:#451a03,stroke:#fbbf24,color:#fffbeb,stroke-width:2px
style R fill:#0b1020,stroke:#64748b,color:#f8fafc
style L fill:#1e293b,stroke:#94a3b8,color:#f8fafc,font-size:11px
linkStyle default stroke:#94a3b8,stroke-width:2px,color:#f8fafc
linkStyle 9 stroke:none,fill:none
click C "https://github.com/NousResearch/hermes-agent/blob/ce5028d0d2dd96a897fe659b22426e76998c4828/hermes_cli/update_receipt.py#L375-L403" "Committed-code follow-up contract"
click P "https://github.com/NousResearch/hermes-agent/blob/ce5028d0d2dd96a897fe659b22426e76998c4828/hermes_cli/update_receipt.py#L375-L403" "Receipt follow-up producer"
click S "https://github.com/NousResearch/hermes-agent/blob/ce5028d0d2dd96a897fe659b22426e76998c4828/hermes_cli/update_receipt.py#L225-L261" "Root receipt store"
click A "https://github.com/NousResearch/hermes-agent/blob/ce5028d0d2dd96a897fe659b22426e76998c4828/hermes_cli/web_routers/actions.py#L340-L353" "Receipt fallback"
click B "https://github.com/NousResearch/hermes-agent/blob/ce5028d0d2dd96a897fe659b22426e76998c4828/hermes_cli/web_routers/actions.py#L384-L410" "Dashboard status response"
click D "https://github.com/NousResearch/hermes-agent/blob/ce5028d0d2dd96a897fe659b22426e76998c4828/apps/desktop/electron/managed-ssh-update.ts#L418-L438" "SSH summary; dashboard source in prose"
click U "https://github.com/NousResearch/hermes-agent/blob/ce5028d0d2dd96a897fe659b22426e76998c4828/hermes_cli/web_routers/actions.py#L384-L410" "Status consumed by clients; evidence in review"
click T "https://github.com/NousResearch/hermes-agent/blob/ce5028d0d2dd96a897fe659b22426e76998c4828/hermes_cli/update_receipt.py#L375-L390" "Printed follow-up protocol"
click H "https://github.com/NousResearch/hermes-agent/blob/ce5028d0d2dd96a897fe659b22426e76998c4828/scripts/desktop-update/posix.sh#L897-L918" "Local handoff reports owed work"
click K "https://github.com/NousResearch/hermes-agent/pull/132365" "Referenced prerequisite; integration evidence remains in review"
This was referenced Oct 6, 2026
teknium1
force-pushed
the
fix/update-post-commit-never-fails
branch
from
October 6, 2026 01:09
d52f0f2 to
558e8e3
Compare
teknium1
added a commit
that referenced
this pull request
Oct 6, 2026
…sult On main (#132386) a --prepared child that exits without a result file is a lost completion: _prepare settles it as owed and re-arms the tail for the next launch. The L3 test's fake child returned 0 and wrote nothing, so after the rebase the recorded events gained a trailing arm_completion from that settle path. The fake now behaves like a successful child and writes its result; the ordering assertion the test exists for (lease first, then everything that writes the checkout) is unchanged.
andrexibiza
reviewed
Oct 6, 2026
andrexibiza
left a comment
andrexibiza
left a comment
There was a problem hiding this comment.
Post-merge continuity pass on 189ce63 (merged as 648a309), continuing the R5/R6 review rather than restarting from the original diff.
I re-traced the repaired consumers and receipt boundaries that were still open in my last pass.
R5 — durable dashboard action identity: closed
The repair no longer derives identity solely from the bounded presentation log. _spawn_hermes_action atomically persists the action id in the per-action sidecar, and status recovery uses the visible header first, then the sidecar, then the live registry. The regression matrix covers short output, >2,000 lines, rotation, success, failed, unfinished, and a newer receipt from another profile. That closes the exact failure I reproduced at 558e8e3c373: log retention can no longer turn a matching terminal archive into exit_code=null, and an unrelated root receipt still cannot certify the action.
R6 — debt projection across non-happy verdicts: closed
managedUpdateMessage now computes receipt debt before composing the update/restore verdict. A committed update with failed profile restoration retains its dependency debt while updateOk / restoreOk remain independent. A partial receipt with user_action carries the producer's exact parked-local-change instruction and does not substitute “rerun hermes update.” The dashboard and local Desktop consumers were also extended to render debt only when the receipt action id matches the action they started.
Additional receipt-store review: closed
The named-profile split reported later in review is repaired without moving the canonical updater archive away from the root store. The updater mirrors latest.json into the profile store, readers reconcile root/profile latest pointers by modification time, and action lookup searches both archive locations. The real two-home subprocess regression exercises both directions: PM sees the updater receipt; updater consumers see a later PM receipt; pre-move profile archives remain discoverable.
The archive-reader consolidation also fixes a real stale-parent hazard: lost-completion settlement now resumes the child's persisted run record before appending the completion debt, instead of finalizing the parent's pre-child snapshot and erasing stages. Malformed/non-object archives are treated as missing at this post-commit boundary rather than escaping the C3 contract.
Reconciliation
| Finding | Final disposition |
|---|---|
| R1 cross-profile latest receipt certifies another action | closed |
| R2 client protocols drop followups/user action | closed |
| R3 same-SHA catch-up loses newly incurred gateway debt | closed |
| R4 active config write failure skips sibling/snapshot safety nets | closed |
| R5 bounded log tail loses action identity | closed |
| R6 restore-failed / partial SSH result hides recovery debt | closed |
| named-profile updater/PM receipt-store split | closed |
| duplicate follow-up/archive-reader paths | consolidated; stale-snapshot case repaired |
Head CI is green across CI, Install & Update E2E, Nix flake check, and Docker Build, Test, and Publish. I found no new P0–P3 finding in the continuity surface after the final repair series.
This is a COMMENT post-merge closure review, not an approval event.
teknium1
added a commit
that referenced
this pull request
Oct 6, 2026
…sult On main (#132386) a --prepared child that exits without a result file is a lost completion: _prepare settles it as owed and re-arms the tail for the next launch. The L3 test's fake child returned 0 and wrote nothing, so after the rebase the recorded events gained a trailing arm_completion from that settle path. The fake now behaves like a successful child and writes its result; the ordering assertion the test exists for (lease first, then everything that writes the checkout) is unchanged.
andrexibiza
reviewed
Oct 6, 2026
There was a problem hiding this comment.
Post-merge CR — two recovery-message gaps remain
Reviewed head: 189ce63209e. Merge: 648a3092554. Main snapshot compared: 9d4dfde8172, October 6, 2026. The relevant losses remain in that main snapshot; this is not a review of an obsolete pre-fix head.
Two P2 findings: R7 and R8. The latest repairs do preserve recovery information in the managed result and in terminal receipt fields. The remaining failures are downstream: both Update-all notification consumers discard the repaired message, and the shared/dashboard/Desktop outcome contract omits a real post-commit terminal state, interrupted. Neither finding alleges that the stash was deleted or the checkout rolled back.
R7 · [P2] Preserve the repaired message through both Update-all consumers
Changed anchor: managed result composition, line 1007.
The R6 formatter repair works at its own boundary. managedUpdateMessage now retains follow-ups and the producer's manual instruction when restoration fails or the update is partial. However, managedSshUpdateAllRow places that message in detail while assigning every unsuccessful managed row a nonempty error: the raw error, or result.outcome when there is none.
Both final consumers then select error before detail:
| Consumer | Exact selection | Consequence |
|---|---|---|
| Update everything | `row.error | |
| Settings → Update all | `new Error(row.error |
Executed: source-extracted runManagedSshUpdate → managedSshUpdateAllRow → the complete runEverythingUpdate function and the actual Settings updateAll callback. Transport/restore operations, bridge results, atoms, notification calls and React callback binding were injected; the actual notification arguments were captured. This was not a live SSH transaction or mounted Electron/React window.
| Input | Result/row before presentation | Notification argument in both consumers |
|---|---|---|
| Clean success | No debt, ok=true |
Normal ready message — control passes. |
Success + dependencies follow-up; restoration succeeds |
detail names the step and rerun remedy |
Same complete recovery message — control passes. |
Same receipt; restoreScope throws Error("restore refused") |
updateOk=true, restoreOk=false; detail still names dependencies and its remedy |
restore refused only. |
Exit 1 / partial + local_changes user action; restoration succeeds |
updateOk=false, restoreOk=true; detail contains the exact stash recovery instruction |
update-failed only. |
The manual-action fixture was Run \git stash apply stash@{0}` in /opt/hermes to restore your local changes.Removing only the row'serror` in the diagnostic control restores the complete message; no producer change is needed to make the information available.
RCA and attribution: the formatter now fulfills its local contract, but the repair stops one consumer boundary too early. The existing priority expressions predate this addition; they are an unadapted consumer of the new recovery message, not evidence that the repaired formatter still drops it. The new committed-owed-step test asserts the failed rows' detail, not what either UI sends to its notification API. That is why its passing cases do not close this behavior.
Required repair: compose the recovery detail with the specific raw diagnostic through one presentation helper used by both fan-out entrypoints. Merely swapping priority can hide the useful restore error; preserve both without duplication. Keep updateOk/restoreOk independent, preserve clean/skipped behavior, and never substitute updater-rerun advice for a parked-stash instruction.
Regression boundary: feed the real converted row through each actual notification consumer for the restore-failed and partial/manual-action cases above. Require both the diagnostic and every owed-step/manual instruction in the captured notification argument. A .detail assertion alone is insufficient.
R8 · [P2] Carry interrupted through terminal proof and debt presentation
Changed anchor: update-debt.ts, lines 37–38.
The new helper equates a committed run with success or partial. The producer explicitly has another post-commit terminal outcome: finalize_interrupted_update_receipt finalizes interrupted, normally with exit 130, after adopting the run's durable record. UpdateReceipt.finalize keeps the supplied followups and user_action; the earlier immediate user-action persistence repair is not the loss being reported here.
For an action-correlated update interrupted after commit, three classifiers disagree with that producer:
| Boundary | Current behavior |
|---|---|
| Dashboard durable exit classification | With no live/in-memory result or completion marker, recognizes only success/partial; the interrupted receipt yields null. |
Desktop receiptProvesOutcome |
Rejects the finished interrupted receipt even when it belongs to the POST's exact action. |
Shared updateDebt |
Returns null, discarding both recovery fields for presentation. |
Consequently, the Desktop polling/failure path can wait out its deadline and still give no local-change recovery instruction. A known exit 130 avoids that wait but still reaches the same missing-debt presentation. The web toast also reads this shared helper and therefore omits the preserved instruction.
Executed: the complete upstream shared helper, the source-extracted receipt finalize method with fixture clock/code-identity lookups, the action-status helpers, and the complete source-extracted backend poller/proof/failure path with injected API/state and fake clock/timers. The helper's complete source matched Git blob 3e03b162bd11f85364ded0f277e05be003f663ab.
| Controlled input | Observed result |
|---|---|
Own finished partial receipt, exit null |
One poll; returned failure message retains follow-up and exact manual instruction. |
Otherwise equivalent own finished interrupted receipt, exit null, no completion marker |
240 polls / 360,000 simulated ms, then Backend update failed.; instruction absent. |
| Own interrupted receipt, known exit 130 | One poll, but still Backend update failed. without the instruction. |
| Another action's partial receipt, exit 1 | No foreign instruction — identity control passes. |
| Only change shared-helper input from interrupted to partial | Both already-present recovery fields become visible. |
The six minutes are fake-clock elapsed time, not a wall-clock or native-update test. The finalization experiment begins with the debt fields supplied; it does not independently repeat the author's three-process persistence/kill test. The web consequence is source-traced through the exercised shared helper, not browser-mounted here.
RCA and attribution: terminal outcome, action identity and recovery debt are distinct dimensions. The identity check is correct; the incomplete terminal-state vocabulary suppresses valid instructions after it. The new shared predicate introduces this missing presentation cell, while the dashboard/Desktop proof predicates remain unadapted to the post-commit interrupted outcome. This is not a reason to relabel interrupted as successful or accept another profile's latest receipt.
Required repair: treat an own, finished, interrupted receipt as terminal non-success and expose its owed work without waiting for a success marker. Carry the recorded interruption/exit semantics, retain exact action matching, and keep the manual instruction separate from retryable follow-ups. Regress both known exit 130 and restarted/rotated-log exit_code:null, plus running/mismatched-receipt controls. Test the final notification/state message, not just persisted fields.
SSA / RCA — producer to last visible consumer
Every operational node links to the reviewed source. Green denotes the preservation boundary actually exercised, not a claim that the whole subsystem passed. The restart-recovery branch reaches the debt-message path after its polling deadline; the known-exit/web branch can reach it immediately.
%%{init: {"theme":"dark","themeVariables":{"fontFamily":"Arial","fontSize":"16px","background":"#0d1117","primaryTextColor":"#e6edf3","lineColor":"#94a3b8"},"flowchart":{"htmlLabels":true,"curve":"linear","nodeSpacing":25,"rankSpacing":40}}}%%
flowchart TB
subgraph CORE[" "]
direction TB
subgraph A["R7 · Update-all"]
direction TB
P["Correlated SSH proof<br/>follow-ups or manual action"]:::authority
M["managedUpdateMessage<br/>recovery text preserved"]:::verified
R["managedSshUpdateAllRow<br/>detail + nonempty error"]:::normal
N["Update everything<br/>error wins; detail lost"]:::failure
S["Settings Update all<br/>error wins; detail lost"]:::failure
P --> M --> R --> N
R --> S
end
subgraph B["R8 · Interrupted"]
direction TB
W["Receipt finalization<br/>debt fields preserved"]:::verified
I["Terminal outcome<br/>interrupted"]:::normal
T["Restart recovery proof<br/>interrupted rejected"]:::failure
D["updateDebt<br/>returns null"]:::failure
U["Web / Desktop message<br/>recovery text omitted"]:::failure
W --> I --> T --> D --> U
I -->|known exit / web| D
end
A ~~~ B
end
CORE ~~~ K
K["Legend<br/>Purple: ownership / policy<br/>Green: success<br/>Red: failure<br/>Gray: context / transient"]
style CORE fill:#0d1117,stroke:#30363d,color:#e6edf3
style A fill:#182330,stroke:#516477,color:#ffffff
style B fill:#182330,stroke:#516477,color:#ffffff
style K fill:#1e293b,stroke:#94a3b8,color:#f8fafc,font-size:11px
classDef authority fill:#3b0764,stroke:#c084fc,color:#faf5ff,stroke-width:2px
classDef normal fill:#1e293b,stroke:#94a3b8,color:#f8fafc,stroke-width:2px
classDef verified fill:#052e16,stroke:#4ade80,color:#f0fdf4,stroke-width:2px
classDef failure fill:#450a0a,stroke:#f87171,color:#fef2f2,stroke-width:2px
click P "https://github.com/NousResearch/hermes-agent/blob/189ce63209efe615e4fdfb8a3cddb5299952b7e9/apps/desktop/electron/managed-ssh-update.ts#L985-L1010" "Receipt-to-result boundary"
click M "https://github.com/NousResearch/hermes-agent/blob/189ce63209efe615e4fdfb8a3cddb5299952b7e9/apps/desktop/electron/managed-ssh-update.ts#L513-L548" "Recovery message composition"
click R "https://github.com/NousResearch/hermes-agent/blob/189ce63209efe615e4fdfb8a3cddb5299952b7e9/apps/desktop/electron/managed-ssh-update.ts#L1059-L1072" "Result-to-row conversion"
click N "https://github.com/NousResearch/hermes-agent/blob/189ce63209efe615e4fdfb8a3cddb5299952b7e9/apps/desktop/src/store/updates.ts#L1154-L1179" "Notification selection"
click W "https://github.com/NousResearch/hermes-agent/blob/189ce63209efe615e4fdfb8a3cddb5299952b7e9/hermes_cli/update_receipt.py#L220-L225" "Receipt.finalize"
click I "https://github.com/NousResearch/hermes-agent/blob/189ce63209efe615e4fdfb8a3cddb5299952b7e9/hermes_cli/update_receipt.py#L739-L758" "Post-commit interruption finalizer"
click D "https://github.com/NousResearch/hermes-agent/blob/189ce63209efe615e4fdfb8a3cddb5299952b7e9/apps/shared/src/update-debt.ts#L31-L48" "Shared outcome filter"
click U "https://github.com/NousResearch/hermes-agent/blob/189ce63209efe615e4fdfb8a3cddb5299952b7e9/web/src/contexts/SystemActions.tsx#L57-L82" "Web presentation consumer"
click S "https://github.com/NousResearch/hermes-agent/blob/189ce63209efe615e4fdfb8a3cddb5299952b7e9/apps/desktop/src/app/settings/connections-registry.tsx#L506-L535" "Settings notification selection"
click T "https://github.com/NousResearch/hermes-agent/blob/189ce63209efe615e4fdfb8a3cddb5299952b7e9/apps/desktop/src/store/updates.ts#L862-L886" "Desktop terminal receipt classifier"
Reconciliation — credit the repairs without declaring the class closed
| Earlier finding | This pass's disposition |
|---|---|
| R1: another action's receipt certifies this one | Exact-action checks are present; mismatched-action presentation controls reject foreign debt. The full cross-profile ASGI/archive experiment was not rerun here. |
| R2: recovery fields omitted by summary protocols | Fields and successful-result recovery messages are present; the success-with-debt control passes. Final-consumer/state coverage remains incomplete in R7/R8. |
| R3: same-SHA inventory loses newly stopped runtime | Author reports the monotonic inventory repair in 245571c82d1 with verifier/startup regression. That lifecycle test was not independently rerun in this pass. |
| R4: active config failure skips sibling/snapshot recovery | Author reports deferred raising in 05fcd5c1310, after the independent restoration work. The config/snapshot experiment was not independently rerun here. |
| R5: identity disappears beyond the bounded log tail | The sidecar/header helpers pass short, 2,001-line and rotated-log fixtures; a newer visible header beats an older sidecar. This is helper-level revalidation, not a full dashboard restart. Not carried forward as the old defect. |
| R6: formatter drops debt for restore failure/partial | Result/row composition is repaired and exercised. R7 is the downstream message-selection loss; R8 is the omitted interrupted state. Neither restates the repaired early return. |
Older findings and duplicate threads — retained provenance and verification limits
These are not silently promoted from a resolved thread to independently proven closure. Unless specified above, the earlier native/lifecycle experiments were not rerun in this pass.
| Earlier item | Reconciliation |
|---|---|
| Unsafe SQLite / forced completion verdict | Author's 372966f86c4 separates the runtime/completion safety verdict from committed exit success. No new native-runtime experiment here. |
| Config/profile-sync swallowed errors | Author reports version-only migration and escaping profile-sync repairs; per-profile best-effort handling is deliberately retained/documented, not silently counted as repaired error propagation. |
| Named-profile SSH receipt reader | Root-store reader repair is present in inspected source. The embedded observer was not executed in this pass. |
| Restart IPC contradiction, later duplicate | Author reports removal of failure writers for retryable restart debt in b138ed5; no fresh gateway-notification/Windows routing replay here. |
| Carried manual-serve debt across repeated crashes | Author reports carried-row retention repair in 2d58ac01787; crash replay not rerun here. |
| Detached Desktop build warning, pre-build/truncation refinement | Author reports whole-line Desktop-build-owed protocol and both handoffs in 24da1e5ca66. No native shell/Windows handoff validation claimed here. |
| Stopped-unmapped obligation, later duplicate | Author's initial inventory repair and later R3 extension are distinct from the message consumers examined here. Startup discharge replay not rerun. |
| UTF-8 child-boundary report | Author supplied a production-import-order rebuttal: the old fixture emptied hermes_cli.__init__, bypassing its UTF-8 setup. Do not retain the old fixture failure as proof of a current production crash. Native rebuttal/hardening not independently rerun here. |
| Receipt storage failure changes exit | Finalized in-memory fallback is present in inspected source; disk-full/unwritable-store experiment not rerun. |
| Stale Windows parent overwrites terminal child record | Terminal-write guard is present; author reports the resume/follow-up repair. No native Windows replay here. |
| User action lost before interrupted finalization | Immediate persistence is present. R8 concerns downstream interpretation of preserved fields, not renewed on-disk loss; the author's three-process probe is not relabeled as this pass's execution. |
| PID incarnation / same-process ownership | Author reports writer incarnation/deduplication repair; no ownership-process replay here. |
| Root receipt pointer vs profile-local update lock | Companion #132365 is now verified merged. This confirms dependency status, not a fresh two-process exclusion/CAS test. |
| Uncorrelated root completion marker | Followed through R1 and R5 above; no return to a foreign-latest receipt fallback is requested. |
Evidence boundaries and regression delivery
The local commands used Python 3.13.5, Node 22.16.0 and TypeScript 5.8.3 in isolated scratch. Function extracts retain their upstream logic and use explicit harness adapters; they are not a whole-application import. TypeScript emission of those extracts is not a project typecheck. All retained defect/control probes completed successfully in the sense that they observed the reported failing behavior and distinguishing controls; no repair was implemented.
The observed exact-head CI, Install & Update E2E, Nix, and Docker runs report success. Those statuses do not substitute for exercising the two final notification consumers or the interrupted receipt state. No full canonical suite, native installer, live SSH host, or Windows/macOS environment was run by this review.
The Mermaid source was rendered and visually inspected offline with installed Mermaid 11.12.2 in Chromium: ten source hyperlinks, embedded legend, zero render errors. This is browser-render validation, not a claimed mmdc run.
Closure criterion: preserve recovery information from the correlated terminal producer through every final consumer, including partial, restore-failed and interrupted outcomes. Keep truthful non-success status and precise diagnostics; do not use success-only tests or intermediate-field assertions to stand in for that contract.
| : restoreOk | ||
| ? 'The remote update failed, but every managed SSH profile was restored.' | ||
| : 'The remote update transaction could not restore every managed SSH profile.' | ||
| ...managedUpdateMessage(outcome, restoreOk, proof?.receipt) |
There was a problem hiding this comment.
R7 · [P2] Carry this recovery message through both Update-all notification consumers
The formatter repair now works: this result and managedSshUpdateAllRow(...).detail retain dependencies debt on restore failure and the manual stash instruction on a partial run. However, the row converter also gives every failed managed row a nonempty error. Both Update everything and Settings → Update all prefer that error to detail.
Executed the source-extracted lifecycle → row → both actual notification callbacks with injected transport/bridge/state. A success receipt with dependencies debt plus restoreScope throwing Error("restore refused") sends only restore refused to both notification APIs. Exit 1 / partial with a verbatim local_changes recovery instruction sends only update-failed. In each case .detail still has the complete recovery text. Clean and success-with-debt controls pass; removing only .error restores the message.
Compose the detail and specific diagnostic through a shared presentation helper used by BOTH consumers; keep update/restore status independent and do not tell users an updater rerun restores a parked stash. Extend the regression past .detail to capture actual notification arguments. This anchors the unadapted downstream consumer contract at the changed producer; it does not claim managedUpdateMessage still has the repaired early return. No live SSH or mounted UI was exercised. Full matrix, RCA and limits are in R7 of the review.
| return null | ||
| } | ||
|
|
||
| if (receipt.outcome !== 'success' && receipt.outcome !== 'partial') { |
There was a problem hiding this comment.
R8 · [P2] Do not drop recovery instructions from an own interrupted terminal receipt
The producer explicitly finalizes post-commit interruption as interrupted / exit 130, retaining followups and user_action. This success/partial-only predicate returns null for that valid action-matched terminal record. The immediate user-action persistence repair is present; this is a consumer-state loss, not renewed stash/receipt deletion.
The sibling dashboard exit classifier yields null after process/result loss, and Desktop receipt proof rejects interrupted too. Executing the complete shared helper and source-extracted backend poller with injected API/state and fake time: own partial/null-exit returns the manual instruction after one poll; otherwise equivalent own interrupted/null-exit takes 240 polls / 360,000 simulated ms, then returns Backend update failed. without it. Known exit 130 takes one poll but still drops the instruction. Foreign-action controls remain rejected; changing only interrupted to partial restores the already-present debt.
Carry own, finished interrupted receipts through terminal non-success classification and owed-work presentation, keeping action identity and interruption semantics intact. Regress known exit 130 and restarted/rotated-log null exit, with running/foreign controls, and assert final messages in both clients. Do not relabel interrupted as success or replace manual stash instructions with rerun advice. This was an isolated source-extracted/fake-clock experiment, not a native interrupt or six-minute wall-clock run; the full RCA and evidence limits are in R8.
andrexibiza
reviewed
Oct 6, 2026
There was a problem hiding this comment.
Verification amendment — evidence for the landed revision
This supersedes the blanket closure language in my preceding continuity review. That pass inspected changes and workflow summaries; it did not independently execute the regressions. Presenting its reconciliation as an exhaustive closure was stronger than the evidence. This amendment distinguishes source inspection, repository CI execution, and work not performed.
Exact revision and execution evidence
The final PR head is 189ce63209e; the landed merge is 648a3092554.
The four previously named final-head workflows do report success: CI, Install & Update E2E, Nix, and Docker. Those PR checks used the synthetic merge 4611eca38b2016e584355c2634f3df9a48b23cdc, not the later landed merge.
There is also a successful push CI run for the actual landed merge. Its TypeScript job log records checkout of 648a309 and runs npm run test:typescript. The Python 3.11 and Python 3.12 jobs also report success. These are repository CI results, not a local rerun by this reviewer.
What the inspected repairs establish
R5 — action identity: the landed status/spawn implementation persists the per-action sidecar and uses it when the bounded log no longer contains the start header. The added regression defines five cells, not a full Cartesian matrix: short/success, long/success, rotated/success, long/failed, and long/unfinished, each with another profile's newer receipt. My earlier wording did not make that coverage limit clear.
R6 — non-happy outcome debt: the formatter repair and regression retain debt when restoration fails and preserve the exact manual instruction for a partial, parked-local-changes outcome. The regression uses the real embedded observer/parser/lifecycle/result formatter with injected transport and restoration callbacks. That is not a live SSH-host or Electron-window test. The Desktop and dashboard repairs also gate receipt debt on the caller's action identity.
The earlier repairs remain identifiable in the final series: R1 correlation, R2 debt propagation, R3 monotonic gateway inventory, and R4 deferred migration error. The root/profile receipt-store repair, archive-reader consolidation, and stale-parent snapshot repair likewise have concrete changes and regression definitions. Their presence is not a substitute for freshly executing every earlier failure scenario across the integrated stack.
Evidence boundary
%%{init: {"theme":"dark","themeVariables":{"background":"#0d1117","primaryTextColor":"#e6edf3","lineColor":"#8b949e"}}}%%
flowchart TB
subgraph CORE[" "]
direction TB
M["Landed revision: 648a3092554"] --> C["Push CI: success"]
C --> J["TypeScript job: success"]
C --> P["Python 3.11 / 3.12 jobs: success"]
M --> R5["R5: durable action identity"]
M --> R6["R6: debt across terminal outcomes"]
R5 -. "source inspected; not rerun locally" .-> N["Independent local reproduction: not executed"]
R6 -. "source inspected; not rerun locally" .-> N
end
CORE ~~~ LEGEND
LEGEND["Legend<br/>Purple: ownership / policy<br/>Teal: durable state<br/>Green: success<br/>Gray: context / transient"]
classDef durable fill:#134e4a,stroke:#2dd4bf,color:#f0fdfa,stroke-width:2px
classDef policy fill:#3b0764,stroke:#c084fc,color:#faf5ff,stroke-width:2px
classDef verified fill:#052e16,stroke:#4ade80,color:#f0fdf4,stroke-width:2px
classDef unrun fill:#1e293b,stroke:#94a3b8,color:#f8fafc,stroke-width:2px
class M durable
class R5,R6 policy
class C,J,P verified
class N unrun
style CORE fill:#0d1117,stroke:#30363d,color:#e6edf3
style LEGEND fill:#1e293b,stroke:#94a3b8,color:#f8fafc,font-size:11px
click N "https://github.com/NousResearch/hermes-agent/pull/132386#pullrequestreview-5427886335" "Explicit reviewer evidence boundary; no local execution claimed"
click M "https://github.com/NousResearch/hermes-agent/commit/648a3092554b62a251ce1d0e904fc1a8f4f6c953" "Landed revision"
click C "https://github.com/NousResearch/hermes-agent/actions/runs/37451823192" "Actual merge CI"
click J "https://github.com/NousResearch/hermes-agent/actions/runs/37451823192/job/112230164194" "TypeScript execution log"
click P "https://github.com/NousResearch/hermes-agent/actions/runs/37451823192" "Python jobs"
click R5 "https://github.com/NousResearch/hermes-agent/blob/648a3092554b62a251ce1d0e904fc1a8f4f6c953/hermes_cli/web_routers/actions.py" "Action identity implementation"
click R6 "https://github.com/NousResearch/hermes-agent/blob/648a3092554b62a251ce1d0e904fc1a8f4f6c953/apps/desktop/electron/managed-ssh-update.ts" "Managed SSH result implementation"
Revised conclusion: the inspected repair paths and successful landed CI support the reported fixes. No additional defect was established in this continuity inspection. I withdraw the stronger implication that this constituted exhaustive, independently reproduced closure of every earlier finding. I also did not audit CI separately for all 62 commits; final-head/merge success does not establish that condition. No approval or merge-readiness certification is being issued.
teknium1
added a commit
that referenced
this pull request
Oct 6, 2026
…out_move #132386 added three ZIP tests that replace update_cmd._prepare_git_command with a no-argument lambda. This branch calls it as _prepare_git_command(checkout_move=...), so after the rebase onto main each fake died with "unexpected keyword argument 'checkout_move'". They now take **_ like every other fake of that function in the tree; the assertions are unchanged.
teknium1
added a commit
that referenced
this pull request
Oct 6, 2026
…pts checkout_move The same #132386 interaction as the ZIP completion tests: this test (from main) replaces update_cmd._prepare_git_command with a no-argument lambda inside its child process, and this branch calls it with checkout_move=..., so the child died with TypeError and the test read it as a failed update. The fake takes **_ like every other fake of it.
teknium1
added a commit
that referenced
this pull request
Oct 6, 2026
…mit-safe hand-offs While this branch was in review, #132386 landed on main: a committed `hermes update` that still owes work prints one whole "Desktop app build owed:" line and one "Update follow-up '<step>' did not finish:" line per owed step, and both Desktop hand-offs turn those into a manual outcome naming the step (never a plain "Update complete."). This branch rewrites both hand-off scripts, so the rebuild took its versions whole and this commit carries main's behaviour into them, through this branch's own post-commit design: - posix.sh: owed_followup_steps() (unchanged from main: the parser contract test runs it), the build-owed and owed-steps checks on a zero exit, each setting DONE_NOTE (finish() makes it manual) and an add_warning entry so the step also reaches the result's warnings; the Linux relaunch gate now appends to DONE_NOTE instead of replacing an owed-step note. - windows.ps1: the same two checks as Add-Followup -Manual calls before the runtime verify, with main's exact [regex]::Matches expression. Already equivalent here: the mac swap is skipped when the rebuild failed, and finish() relaunches the kept app for a manual note unless the Linux gate says skew/manual. Before: 6 of main's contract tests fail on the rebuilt branch (test_every_owed_followup_reaches_the_desktop_result x4, the truncated build-owed case, the parser contract). After: tests/scripts/desktop_update/ plus test_update_post_commit_followups.py pass, 113 passed, 0 failed.
teknium1
added a commit
that referenced
this pull request
Oct 6, 2026
…mit-safe hand-offs While this branch was in review, #132386 landed on main: a committed `hermes update` that still owes work prints one whole "Desktop app build owed:" line and one "Update follow-up '<step>' did not finish:" line per owed step, and both Desktop hand-offs turn those into a manual outcome naming the step (never a plain "Update complete."). This branch rewrites both hand-off scripts, so the rebuild took its versions whole and this commit carries main's behaviour into them, through this branch's own post-commit design: - posix.sh: owed_followup_steps() (unchanged from main: the parser contract test runs it), the build-owed and owed-steps checks on a zero exit, each setting DONE_NOTE (finish() makes it manual) and an add_warning entry so the step also reaches the result's warnings; the Linux relaunch gate now appends to DONE_NOTE instead of replacing an owed-step note. - windows.ps1: the same two checks as Add-Followup -Manual calls before the runtime verify, with main's exact [regex]::Matches expression. Already equivalent here: the mac swap is skipped when the rebuild failed, and finish() relaunches the kept app for a manual note unless the Linux gate says skew/manual. Before: 6 of main's contract tests fail on the rebuilt branch (test_every_owed_followup_reaches_the_desktop_result x4, the truncated build-owed case, the parser contract). After: tests/scripts/desktop_update/ plus test_update_post_commit_followups.py pass, 113 passed, 0 failed.
teknium1
added a commit
that referenced
this pull request
Oct 6, 2026
…out_move #132386 added three ZIP tests that replace update_cmd._prepare_git_command with a no-argument lambda. This branch calls it as _prepare_git_command(checkout_move=...), so after the rebase onto main each fake died with "unexpected keyword argument 'checkout_move'". They now take **_ like every other fake of that function in the tree; the assertions are unchanged.
teknium1
added a commit
that referenced
this pull request
Oct 6, 2026
…pts checkout_move The same #132386 interaction as the ZIP completion tests: this test (from main) replaces update_cmd._prepare_git_command with a no-argument lambda inside its child process, and this branch calls it with checkout_move=..., so the child died with TypeError and the test read it as a failed update. The fake takes **_ like every other fake of it.
teknium1
added a commit
that referenced
this pull request
Oct 6, 2026
…out_move #132386 added three ZIP tests that replace update_cmd._prepare_git_command with a no-argument lambda. This branch calls it as _prepare_git_command(checkout_move=...), so after the rebase onto main each fake died with "unexpected keyword argument 'checkout_move'". They now take **_ like every other fake of that function in the tree; the assertions are unchanged.
teknium1
added a commit
that referenced
this pull request
Oct 6, 2026
…pts checkout_move The same #132386 interaction as the ZIP completion tests: this test (from main) replaces update_cmd._prepare_git_command with a no-argument lambda inside its child process, and this branch calls it with checkout_move=..., so the child died with TypeError and the test read it as a failed update. The fake takes **_ like every other fake of it.
This was referenced Oct 6, 2026
teknium1
added a commit
that referenced
this pull request
Oct 7, 2026
hermes.update.run blamed 971 failed CLI runs in Sep 28 - Oct 5 on the apply stage while hermes.update.stage held zero failed apply rows. Stage marks are END marks: every pre-apply exit (fetch, channel, branch, merge, HEAD checks) leaves no mark for the stage it died in, so the run row inferred the stage and no stage row said it failed. A failed run whose failed_stage names a stage it never marked now gets one failed row for it. Rebuilt on the updater overhaul's followups model (#132386): a committed run that owes follow-ups is a run-level success and its failed stage already has its own mark, so only failed runs get the row; refusals never do.
teknium1
added a commit
that referenced
this pull request
Oct 7, 2026
hermes.update.run blamed 971 failed CLI runs in Sep 28 - Oct 5 on the apply stage while hermes.update.stage held zero failed apply rows. Stage marks are END marks: every pre-apply exit (fetch, channel, branch, merge, HEAD checks) leaves no mark for the stage it died in, so the run row inferred the stage and no stage row said it failed. A failed run whose failed_stage names a stage it never marked now gets one failed row for it. Rebuilt on the updater overhaul's followups model (#132386): a committed run that owes follow-ups is a run-level success and its failed stage already has its own mark, so only failed runs get the row; refusals never do.
teknium1
added a commit
that referenced
this pull request
Oct 7, 2026
hermes.update.run blamed 971 failed CLI runs in Sep 28 - Oct 5 on the apply stage while hermes.update.stage held zero failed apply rows. Stage marks are END marks: every pre-apply exit (fetch, channel, branch, merge, HEAD checks) leaves no mark for the stage it died in, so the run row inferred the stage and no stage row said it failed. A failed run whose failed_stage names a stage it never marked now gets one failed row for it. Rebuilt on the updater overhaul's followups model (#132386): a committed run that owes follow-ups is a run-level success and its failed stage already has its own mark, so only failed runs get the row; refusals never do.
teknium1
added a commit
that referenced
this pull request
Oct 7, 2026
hermes.update.run blamed 971 failed CLI runs in Sep 28 - Oct 5 on the apply stage while hermes.update.stage held zero failed apply rows. Stage marks are END marks: every pre-apply exit (fetch, channel, branch, merge, HEAD checks) leaves no mark for the stage it died in, so the run row inferred the stage and no stage row said it failed. A failed run whose failed_stage names a stage it never marked now gets one failed row for it. Rebuilt on the updater overhaul's followups model (#132386): a committed run that owes follow-ups is a run-level success and its failed stage already has its own mark, so only failed runs get the row; refusals never do.
teknium1
added a commit
that referenced
this pull request
Oct 7, 2026
hermes.update.run blamed 971 failed CLI runs in Sep 28 - Oct 5 on the apply stage while hermes.update.stage held zero failed apply rows. Stage marks are END marks: every pre-apply exit (fetch, channel, branch, merge, HEAD checks) leaves no mark for the stage it died in, so the run row inferred the stage and no stage row said it failed. A failed run whose failed_stage names a stage it never marked now gets one failed row for it. Rebuilt on the updater overhaul's followups model (#132386): a committed run that owes follow-ups is a run-level success and its failed stage already has its own mark, so only failed runs get the row; refusals never do.
teknium1
added a commit
that referenced
this pull request
Oct 7, 2026
hermes.update.run blamed 971 failed CLI runs in Sep 28 - Oct 5 on the apply stage while hermes.update.stage held zero failed apply rows. Stage marks are END marks: every pre-apply exit (fetch, channel, branch, merge, HEAD checks) leaves no mark for the stage it died in, so the run row inferred the stage and no stage row said it failed. A failed run whose failed_stage names a stage it never marked now gets one failed row for it. Rebuilt on the updater overhaul's followups model (#132386): a committed run that owes follow-ups is a run-level success and its failed stage already has its own mark, so only failed runs get the row; refusals never do.
jigo-hyunjin
added a commit
to jigo-ai-team/hermes-agent
that referenced
this pull request
Oct 8, 2026
* feat(plugin-catalog): bump stt-vocab to 1.1.0
New languages setting (de, es, fr, it, nl, pt, tr) so memory notes written
in those languages give clean names: their sentence openers, note words and
month names are no longer glued onto the names. Pin 6145d06 -> 54e83f9.
* feat(plugin-catalog): bump aux-ledger to 1.1.0
Time windows (/aux 6h, /aux 7d, any number of hours or days) and grouping by provider (/aux providers). What the plugin records is unchanged.
* feat(plugin-catalog): bump stream-speed to 1.1.0
C:/Program Files/Git/speed session, time windows (/speed 6h, /speed 7d, any number of hours or days) and grouping by provider (/speed providers). What the plugin records is unchanged.
* chore(plugin-catalog): crew 0.7.9, re-pin to 317c08c
* feat(plugin-catalog): stalkchain banner and update
* chore(catalog): update Local System One to 0.6.0
* plugin-catalog: add banner and bump pin for honcho
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
* feat(plugin-catalog): add klipper-print-watch
* chore(plugin-catalog): bump klipper-print-watch to a970970
* chore(catalog): review disclosure for klipper-print-watch
* feat(plugin-catalog): add pastdotdev memory provider
* chore(catalog): pastdotdev disclosure form + contributor map for Kiloris
* catalog: add kimchi-acp-provider (Kimchi harness over ACP stdio)
Pins getkimchi/kimchi-hermes model-providers/kimchi-acp @
f3da18f16dff7220c909227f687b5885ab729f66 (community tier, models
category, requires_env none — the harness subprocess owns auth).
Owner-submitted per the admission rules. Companion to the
kimchi-provider entry.
Co-Authored-By: Kimchi <noreply@kimchi.dev>
* catalog: pin kimchi-hermes@d6ffe628 — make --yolo opt-in per review
* chore(catalog): review disclosure for kimchi-acp-provider
* chore(plugin-catalog): bump pinned-folders entry sha to v0.1.4
* feat(plugin-catalog): add alice-voice plugin
* chore(catalog): review disclosure for alice-voice
* Re-pin agora to v2.0.10
* feat(plugin-catalog): add gbrain-pointer (community GBrain memory provider)
* chore(catalog): review disclosure for gbrain-pointer
* chore: map contributor email for praggybuilds
* feat(catalog): add Jot notes for Hermes Desktop
* feat(catalog): pin Jot 0.2.0 with current screenshots
Use the reviewed main commit with refreshed bilingual Desktop captures
and precise documentation of undoing the latest round of agent edits.
Keep the banner as the card image and English screenshots in the gallery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(catalog): add Kinprove genealogy research recipes
* chore: map contributor email for lazyants
* feat(plugins): a portable package can ask Hermes to gate its MCP server
Agent Plugins v1 has no trust field, and Hermes's portable mcp.json reader
accepts only type/url/headers for a remote server, so a package server always
ran at the default trust: full. A package whose tools trade, spend or send
(senpi resolving trade approvals on a user's Hyperliquid account) could only
ask the model to check with the user; Hermes itself never prompted.
plugin.json's Hermes extension now accepts
extensions.com.nousresearch.hermes.servers.<name>.trust: untrusted
which is applied to that mcp.json server exactly like a config.yaml
'trust: untrusted': every write-capable call asks first and fails closed
when unattended. 'full' (the default) is accepted and changes nothing, any
other value refuses the package, so a package can narrow access but never
widen it, and a config.yaml server with the same name still replaces the
package's entry. 'hermes plugins validate' lists the gated servers.
* feat(plugin-catalog): add switchbot-control
* chore(plugin-catalog): bump switchbot-control to bb68cb4
* chore(catalog): review disclosure for switchbot-control
* fix(update): a stash with no Python is never rejected by the import probe (#130101)
The stash-restore health check compared two independent critical-module
import probes and rejected the restore on any difference, even when the
stash restored no Python at all. The probe's outcome also moves with
install state between the two runs (launch preparation relaunching under
the live update surfaces as SystemExit(0); a dependency sync triggered by
the first probe), so a docs-only stash was rolled back, the stash parked,
the gateway restart skipped and the update exited 1.
Only compare imports when the restore touched Python (the same
restored-Python set the syntax check already uses). A restored .py that
breaks an import is still rejected.
* fix(update): skip the restore import check only for non-runtime files
The #130101 exemption skipped the post-restore import comparison whenever
the stash restored no .py file. A restored native extension (toolsets.so
shadowing toolsets.py), bytecode, a .pth or a data file read at import is
not .py, so such a restore was accepted, the stash dropped, and run_agent,
model_tools and toolsets were left failing to import.
Skip the comparison only when every path the stash restores (read from the
stash commit: worktree, index and untracked trees, renames split) has a
closed non-runtime suffix (docs, images), is not a symlink, and lives in a
directory HEAD already tracks. Anything else is compared as before. A
docs-only restore still succeeds.
* docs: stop presenting Desktop Light as a shipped download
The installation page said "Light is a remote-only build variant" next to
the download instructions, which read as if a remote-only Desktop client
can be downloaded. Light exists only as a build target with no release
leg, and there are no plans to publish it. Replace the sentence with how
remote use actually works today (install a package, connect from Settings
-> Gateways), drop "Light packages" from the uninstall note, and mark the
variant as unpublished in the desktop README.
* plugin-catalog: add Agent Hold Em
* chore(catalog): review disclosure for agent-hold-em
* feat(plugin-catalog): add hermes-field-notes — pitfalls and local core patches
* plugin-catalog: hermes-field-notes 1.2.0 (patch recipes)
* plugin-catalog: hermes-field-notes 1.2.1 (review fixes)
* plugin-catalog: hermes-field-notes — README author section
* chore(catalog): review disclosure for hermes-field-notes
* chore: map alrcatraz for salvage of #132751
* fix(agent): let an explicit tool_reason label a soft interrupt
interrupt() derived the published tool-interrupt reason from whether a message
was attached, so on the soft path a system producer had no way to label itself:
passing a message booked the stop as "user sent a new message", passing none
booked it as "user interrupt". Both live in USER_INTERRUPT_REASONS, so
interrupt_issuer() returned None and the turn exit reason became
interrupted_by_user — a system abort recorded as a human stop.
An explicit tool_reason now wins on the soft path as it already did for
hard_cancel, keeping the message heuristic only as the fallback.
(cherry picked from commit 495f1a84a15a4bf88295af1e06f225e25d57715c)
* fix(agent): attribute the terminal batch-timeout abort to the guard
The sequential-batch timeout guard stopped the turn with a bare
interrupt(message), which the soft-path heuristic read as "user sent a new
message". The abort was therefore recorded as interrupted_by_user and the
skipped calls told the user they had sent a message they never wrote.
Pass the guard's own tool_reason so the exit reason names the real issuer and
the skipped-call notice can describe it.
(cherry picked from commit a756f781b2630ea3059c8e56df132c5d8ba35844)
* fix(agent): render skipped-call notices from the recorded reason
Both sequential skip sites hardcoded a user-initiated cause ("User sent a new
message" / "skipped due to user interrupt"), so any system abort that reached
them asserted an action the user never took.
Render the notice from the published reason instead: a system issuer describes
itself, and only the genuinely user-initiated reasons keep the user-facing
wording.
(cherry picked from commit 8efe429e2a485a183c7ad46522d9a5c99cef6a9c)
* fix(agent): hide the synthetic interrupt close from the transcript
close_interrupted_tool_sequence() appends a synthetic assistant turn when an
interrupted tail is a raw tool result, so the next user message does not land as
tool -> user. The row carried a user-visible "Operation interrupted." string and
was persisted as an ordinary assistant message, so background preemption —
session reaping, watchdogs, batch guards — surfaced in the transcript as an
interruption the user was expected to act on.
Store it in the same hidden shape turn_api_call.py already uses for interrupt
placeholders: content="" plus display_kind="hidden" keeps it out of rendered
transcripts, while the api_content sidecar carries the LLM-visible text and is
substituted at API-build time. The alternation guarantee is unchanged.
(cherry picked from commit 22502b1c40c26a08eefc3d35cee067bd1d889bfc)
* test(agent): cover soft-interrupt attribution and skipped-call wording
The exit reason and the skipped-call notice must both name the real issuer: a
system producer that stops the turn softly labels itself via tool_reason, while
a message-carrying soft interrupt with no reason stays a human stop. The wording
derives from the recorded reason, so a system abort never tells the user they
did something they did not.
(cherry picked from commit ec65b95cfb312ebb7a276a26b163df0eaad65b55)
* fix(agent): render concurrent skip notices from the recorded reason
The sequential path was converted to interrupt_skip_wording() but the
concurrent batch kept three hardcoded user-blaming strings: the pre-batch
skip (execute_tool_calls_concurrent), the unfilled-slot notice
(_unfinished_tool_result) and emit_cancelled's hook message. A system
producer that stops a parallel batch — watchdog, lease loss, SSE
disconnect, or any segment routed through execute_tool_calls_segmented —
still told the model "skipped due to user interrupt" while
interrupt_issuer() booked the same turn as interrupted_by_system. Render
all three from the recorded reason like the sequential siblings; the
observability enums (error_type/hook_error_type) stay untouched.
Also harden the interpolation the wording rides on: _append_skipped_tool_results
substitutes {name} with str.replace instead of str.format, and
interrupt_skip_wording escapes braces in caller-supplied reasons, so a
free-form tool_reason can never raise KeyError/IndexError mid-turn.
(cherry picked from commit b785bccbca6fd997921411198d277c39d3d93261)
* fix(agent): hide only the placeholder close, keep caller banners visible
close_interrupted_tool_sequence hid every synthetic closing turn behind
display_kind="hidden", but four of its five callers pass a real user-facing
notice — the truncation terminal ("Response truncated — …"), the context-
overflow partial text, tool-validation partials and abort_turn_on_interrupt's
"Operation interrupted: …" lines. Hiding those removed the turn's only
explanation from every transcript surface.
Narrow the hidden shape to what it was for: an empty final_response or the
bare interrupt placeholder (which previously rendered as the "Operation
interrupted." bubble users never asked for). Any other text is rendered
verbatim as before. Alternation safety is unchanged in both branches.
(cherry picked from commit a637af2ef44c1f0213c3138c458d6f6c48650988)
* fix(agent): stop the interrupt placeholder echoing back as a reply
A hidden assistant row is hidden from the transcript, not from the request:
the projection substitutes its api_content sidecar into content on every call.
The value chosen for that slot in #88955 was "[response interrupted]" - a short
natural-language phrase sitting in the model's own prior-turn position, which
models reproduce verbatim: an ordinary instruction answered with clean
finish_reason=stop and nothing but the placeholder (#132949). The same hazard
was already recognised twice in this codebase - the redirect scaffold is
barred from assistant text because "the model echoes it" (#81841), and a
runaway repetition partial is replaced by a self-describing label rather than
replayed (#112764) - but the placeholder was never given the same treatment.
Change the placeholder to a structural label the model has no reason to say,
and retire rows already persisted under the pre-change spellings: the prep
filter that dropped scaffold ghosts now also drops hidden rows carrying the
legacy placeholder or the scaffold marker.
Dropping is not always safe. A hidden close appended after a tool result is
the only thing separating that tool message from the next user turn; removing
it recreates the tool -> user alternation violation that makes strict providers
ignore prior context (#48879) - the exact failure the close exists to prevent,
and one repair_message_sequence deliberately leaves alone. When the neighbours
would form that pair, rewrite the row's text instead of dropping it. The
rewrite targets the current placeholder, so a neutralised row never re-triggers
the filter.
(cherry picked from commit 227e9a1ed64e0d75594d031423ecd2f5c74cd0b9)
* test(agent): cover replay-echo retirement, and pin behaviour not wording
The retirement filter gets five cases: a legacy placeholder row between two
user turns is dropped; the same row after a tool result is kept with its text
rewritten (dropping it would hand the provider the tool -> user pair that
close_interrupted_tool_sequence exists to prevent); a row carrying the current
placeholder is left untouched, which is what keeps #88955's re-heal guard
intact and proves the rewrite cannot re-trigger itself; a scaffold ghost still
drops as before; and a visible caller banner is never treated as a ghost.
Verified discriminating: reverting the filter to the scaffold-only version
fails all five, restoring it passes.
The existing #88955 assertions compared against the literal
"[response interrupted]" in four places. That couples the tests to one chosen
spelling - any wording change, including this one, breaks them for the wrong
reason. They now compare against the constant, asserting what the contract
actually says: the durable row stays empty and hidden, the wire copy carries
the neutral payload the writer stamped, and neither is ever the interrupt
scaffold (#81841).
(cherry picked from commit e4ebf0138b43547dbf1db36eaa84beb5c80bbfb7)
* fix(agent): drop the diagnostic message from the batch-guard interrupt
The terminal batch-timeout guard still passed "terminal batch tool did not
complete" as the interrupt message. That text lands in _interrupt_message,
which the gateway (_run_agent_drain_pending) and the CLI
(_chat_resolve_interrupt) re-queue as the user's next turn, so a guard abort
injected a fake user message. Call interrupt(tool_reason=...) with no
message, and use a slug-safe reason so the issuer reads
interrupted_by_system(terminal_batch_timeout) rather than
terminal_batch_aborted:_call_timed_out.
Message-less guard shape from #130209.
Co-authored-by: Yuan Li <dskwelmcy@163.com>
* test(agent): align skip-notice fixtures with the recorded-reason wording
Skipped-call notices now render from _tool_interrupt_reason instead of a
hardcoded "skipped due to user interrupt". The concurrent pre-flight stub
never recorded a reason, so it fell to the "Turn interrupted" fallback and
the stale assertion failed; record the new-message reason the real soft
path publishes and assert its wording. The ACP fixture is cosmetic but
should show the shape the executor actually emits.
* fix(agent): retire replay-echo ghosts without mutating the transcript
_neutralise_replay_echo_ghosts rewrote the caller's list in place
(seq[:] = out) and overwrote content/api_content on durable row dicts.
Those dicts are the live session rows the SessionDB flush cursor and
_DB_PERSISTED_MARKER track, so an in-place edit silently diverges the
in-memory row from what state.db holds. BASE built a new list here; keep
that contract: return a filtered list and neutralise a shallow copy of the
row, which only shapes this request's replay.
* test(agent): trim the soft-interrupt salvage tests to two invariants
Keep one attribution test that goes red on BASE (a tool_reason soft
interrupt is booked to the system issuer with no requeued message) and one
replay-echo test (a legacy placeholder row next to a tool tail is
neutralised on a copy, never dropped into tool -> user, and the durable row
is left untouched). The remaining wording/brace/close-row cases pin
implementation detail already covered by the updated finalizer, steer and
concurrent-interrupt tests.
* fix(agent): render skip wording verbatim instead of brace-escaping it
Fold finding 3 (B02 gate): interrupt_skip_wording doubled braces on the
assumption the wording reaches str.format, but every consumer substitutes
{name} with str.replace, so a reason like "a{b}" rendered as "a{{b}}".
Drop the escape and its docstring claim, and look user-stop wording up
directly in _USER_STOP_WORDING.
* fix(agent): attribute Ctrl-C tool cancellation to a user interrupt
Fold finding 2 (B02 gate, sibling of #130207): both KeyboardInterrupt
handlers in tool_executor call agent.interrupt("keyboard interrupt").
With a message and no tool_reason, interrupt() records "user sent a new
message", so the skipped-call notices for a Ctrl-C claimed the user had
sent a new message. Pass tool_reason=_REASON_USER_INTERRUPT so the
recorded reason (and the rendered wording) is "User interrupt"; the
issuer stays None, so attribution to the human is unchanged.
* fix(agent): neutralise legacy interrupt placeholders instead of dropping them
Fold finding 1 (B02 gate, regression): the replay-echo filter now matches
the legacy "[response interrupted]" placeholder and DROPPED the hidden row
unless its neighbours were tool -> user. The common persisted redirect
shape user -> hidden(placeholder) -> user(correction with checkpoint
api_content) then became user -> user, and _merge_consecutive_users merged
the pair: the correction's checkpoint sidecar was discarded and the first
stored user row was rewritten in place.
Always neutralise a copy ({content: "", api_content: placeholder}) and
never drop: role-safe in every neighbour shape, and the echoable text is
still gone. The kept replay test now also covers the redirect shape
(3 rows survive, checkpoint intact, stored dicts untouched).
* refactor(agent): tidy the interrupt-placeholder fold-ups
Fold finding 4 (B02 gate, low): behaviour-neutral cleanups.
- close_interrupted_tool_sequence: the hidden branch only runs when the
stripped text is empty or already the placeholder, so `stripped or
_INTERRUPTED_PLACEHOLDER` is always the placeholder; say so.
- tool_executor: compute interrupt_skip_wording once for the cancelled
outcome instead of twice.
- move _neutralise_replay_echo_ghosts out of prepare_iteration to module
level (lazy imports kept inside to avoid the conversation_loop cycle).
- the finalizer test asserts the exact placeholder, not just non-empty.
* test(agent): follow the renamed interrupt placeholder in the send-time pad tests
Gate r2 finding 1 (High, regression): a4c687126c renamed the non-final
empty-turn placeholder from "[response interrupted]" to
"[interrupt: no assistant output for this turn]", but two send-time pad
tests in test_partial_stream_finish_reason.py still asserted the old
literal and went red on the stack head. Compare against
agent.agent_runtime_helpers._INTERRUPTED_PLACEHOLDER so the tests track
the constant, not a spelling.
A full grep of tests/ and prod for the old wordings also found a healed
prefill fixture in test_thinking_only_sanitizer.py (still green, but it
models a stub the repair can no longer produce) and a stale docstring in
test_steer.py; both now use the current placeholder.
* test(agent): drive the real batch-timeout guard in the soft tool_reason test
Gate r2 finding 2 (toothless): 482a748431 stopped the terminal batch-timeout
guard from passing a diagnostic message to interrupt(), because
_interrupt_message is what the gateway drain and the CLI re-queue as the
user's next turn. The existing test only called agent.interrupt() directly,
so reverting the guard hunk left it green.
Route the existing test through _run_sequential_tool_execution_middleware
with a prepared terminal call whose worker never settles, so the guard
itself publishes the interrupt; the issuer and `_interrupt_message is None`
asserts now go red when the guard hunk is reverted.
* fix(agent): publish the Ctrl-C reason before the sequential cancel hook
Gate r2 finding 3: in _run_sequential_call's KeyboardInterrupt handler,
ref.emit_cancelled() ran before agent.interrupt(..., tool_reason=user
interrupt), so the post_tool_call hook rendered its wording from an unset
reason and reported "Tool execution cancelled. Turn interrupted" while the
concurrent handler (and 47db3231e2's intent) report "User interrupt".
Publish the interrupt first, matching the concurrent handler.
The existing cancelled-hook test now pins the hook's error_message.
* test(agent): pin the visible-banner branch of close_interrupted_tool_sequence
Gate r2 finding 4 (toothless): f8aea0e81e keeps a caller-supplied banner
("Response truncated — …", partial-delivery text) visible and only hides
the bare placeholder close, but nothing exercised the banner branch, so
reverting it to the always-hidden shape stayed green.
Extend the existing alternation test: a banner close must land as plain
visible content with no display_kind. Red with f8aea0e81e's hunk reverted.
* refactor(agent): match the replay-echo test docstring to the neutralise-only filter
Gate r2 finding 5: d482ce366a made the prep filter neutralise legacy hidden
rows on a copy instead of dropping them, but the module docstring still
described retiring rows except where removal would form tool -> user.
State the actual contract (never drop: removal could form tool -> user or a
user -> user pair that repair merges) and drop _hidden_row's content /
api_content kwargs that no caller passes.
* refactor(agent): move interrupt placeholders into agent_runtime_helpers_placeholders to keep files under the size cap
Code-health ratchet: agent/agent_runtime_helpers.py was 3787 > 3776 and
tests/agent/test_run_agent.py 7014 > 7012 (both over the 2,000 target, so
they may only shrink vs main).
- _INTERRUPTED_PLACEHOLDER / _LEGACY_INTERRUPTED_PLACEHOLDER (and their
rationale comment) move byte-identically into the leaf sibling
agent/agent_runtime_helpers_placeholders.py. Every importer (runtime
helpers via late import, conversation_loop, message_sanitization,
turn_api_call, turn_iteration_prep and five test files) now reads the
sibling; no shim is left in the facade. Nothing patches the old path.
- test_keyboard_interrupt_emits_cancelled_post_tool_hook moves unchanged
into tests/agent/test_run_agent_interrupt_hook.py, reusing the facade's
agent fixture.
* fix(agent): only relabel a prepared-batch abort as a batch timeout on timeout
The batch-guard tail after _poll_sequential_future is shared by the
"timeout" and "interrupted" outcomes. On a user stop (Ctrl-C, /stop, a
new message) whose prepared terminal worker had not settled after the 3s
grace, the unconditional agent.interrupt(tool_reason="terminal batch
timeout") republished the interrupt: the turn was booked
interrupted_by_system(terminal_batch_timeout), _interrupt_message (the
user's queued next message) and _pending_redirect were nulled, and the
wrong reason was re-fanned to workers.
Keep batch.close() unconditional, but only call interrupt() on the
timeout branch; on the interrupted branch the stop is already published.
Public review blocker on #133271 (@ahrazzle, executed repro; @Enough1122).
* refactor(agent): correct the stale placeholder-consistency comment
The moved comment on _INTERRUPTED_PLACEHOLDER claimed it was "kept
identical to the stub placeholder in chat_completion_helpers"; that
module has no such literal (the claim was already stale on main and was
moved verbatim). State what the constant actually is instead.
Trivial note from the public review on #133271 (@ahrazzle).
* fix(agent): stop terminal batch preparation from overwriting a user stop
terminal_approval_batch caught both _CancelledPreparation and TimeoutError
and called agent.interrupt(str(exc)). _CancelledPreparation is raised
because a user interrupt is already pending, so the user's queued message
and redirect were replaced by "Terminal approval preparation cancelled…",
which the gateway then re-queued as a fake user turn. The TimeoutError
branch booked a system timeout as a user stop with a fake requeue — the
#130207 bug class itself, at the sibling site the batch guard fix missed.
Now: close the batch always; only on timeout publish a message-less
interrupt with tool_reason="terminal batch preparation timeout"; do
nothing on cancellation (the stop is already published).
Gate rP finding 1 (2c).
* test(agent): pin system attribution in sequential skip notices
The two _execute_tool_calls_sequential skip notices (stop before a call,
and "remaining tool call(s)" after one) were rewritten to render the
recorded interrupt reason (6eee1fb6b0), but reverting either back to the
hardcoded "due to user interrupt" / "User sent a new message" left every
kept test green. Extend the system-attribution test to drive both sites
after a tool_reason stop; each reverted site now fails the test.
Gate rP finding 2 (2ab).
* test(agent): pin system attribution in unfinished concurrent slot results
_unfinished_tool_result renders the recorded interrupt reason for a slot
no worker filled (ec51b7d021), but hardcoding the old user wording back
left every kept test green. Extend the system-attribution test to call it
after a tool_reason stop and assert both the row and the post_tool_call
error_message say "Turn aborted — …".
Gate rP finding 3 (2ab).
* test(agent): pin the Ctrl-C attribution on the concurrent tool path
fed76c1df2 made both KeyboardInterrupt sites publish
tool_reason="user interrupt", but only the sequential one was covered:
reverting the concurrent worker's tool_reason left the suite green
(the hook read "User sent a new message"). Parametrize the existing
cancelled-hook test over the concurrent path with two calls.
Gate rP finding 4 (2ab).
* refactor(agent): derive user-stop skip wording from USER_INTERRUPT_REASONS
_USER_STOP_WORDING restated the three USER_INTERRUPT_REASONS keys with
their values capitalised, so a new human-stop reason had to be added in
two places or it would render as "Turn aborted — …". Drop the mirror
and capitalise the recorded reason; output is identical for all three
keys (probe-B02-5.py).
Gate rP finding 5 (reuse).
* refactor(agent): share the hidden interrupt placeholder row
message_sanitization.close_interrupted_tool_sequence and the
turn_api_call repetition-loop exit each spelled the same hidden
assistant row literal (content "", display_kind hidden, api_content
_INTERRUPTED_PLACEHOLDER), and the conversation loop's redirect built it
field by field. Add hidden_interrupt_placeholder_row() to the leaf
placeholders module and use it at all three sites so the shape cannot
drift between them.
Gate rP finding 6 (reuse).
* refactor(agent): import the interrupt placeholder at module level
agent_runtime_helpers (two sites) and message_sanitization imported
_INTERRUPTED_PLACEHOLDER inside the function body. The placeholders
module is a leaf with no imports, so there is no cycle to dodge; import
it once at module level (each module still imports cleanly in a fresh
interpreter). conversation_loop keeps its local import.
Gate rP finding 7 (quality).
* docs(agent): say skipped-result content uses str.replace for {name}
The _append_skipped_tool_results docstring still said content "is
formatted with {name}", but the body substitutes with str.replace so a
recorded interrupt reason containing braces renders verbatim. State
that, so nobody "simplifies" it back to str.format.
Gate rP finding 8 (quality).
* test(agent): move the brace-rendering check after the attribution scenarios
The verbatim "{x}" wording check sat between the batch-guard timeout and
the user-stop scenario, mutating _tool_interrupt_reason mid-story. Move
it to the end, alongside the other wording assertions.
Gate rP finding 9 (quality).
* refactor(agent): hoist the hidden placeholder row import and say why a timeout carries no message
Gate rQ quality Lows: the placeholders module imports nothing, so conversation_loop can import it at module level like turn_api_call; the terminal-approval comment now says callers re-queue _interrupt_message, so a system stop must not set one.
* fix(cron): fire an outage-skipped one-shot whose fire claim never landed
Follow-up to #133283 (greptile review). The due scan stamps a one-shot's
run_claim; when the later fire-claim save fails the run never starts. The
next saved scan skips it on that live claim and ends the recovery window,
so once the claim expires _retire_expired_oneshot saw a stale claim past
grace and skipped it forever: neither fired nor retired.
The scan now marks a run claim it stamps past grace (only an outage lets a
one-shot through there) with outage=true; such a claim without a fire claim
keeps the job covered, so it fires once the claim expires. A fire claim
still means the run may have started, so at-most-once is unchanged.
* fix(cron): report an unwritable tick lock in cron status and doctor
Follow-up to #133283 (greptile review). When only cron/.tick.lock is
unwritable (e.g. root-owned after a sudo run, in a writable cron dir) the
tick degrades and returns 0, so the ticker records a success; probe_store
checked only jobs.json, so `hermes cron status` said jobs would fire and
`hermes doctor` reported a writable store while nothing ran.
probe_store now treats an existing tick lock it cannot write the same way
as a read-only jobs.json target: both status and doctor surface it with the
usual fix hint.
* fix(gateway): send cron store notices for a symlinked cron/ directory
Follow-up to #133283 (greptile review). Store records are keyed by the
resolved store path, but the notice sender rebuilt the owning profile from
that path's parent. A profile whose cron/ is a symlink to another disk has
a foreign parent, so neither its outage nor recovery notice reached any
home channel; forget_homes had the same mismatch and never dropped such a
departed profile's record.
Both now match by resolving each served home's own cron/ path.
* fix(cron): keep the outage marker when a skipped one-shot's dispatch fails
clear_run_claim (the dispatch-failure path: interpreter shutdown, execution
creation failure, pool.submit failure) set run_claim to None. The claiming scan
had already ended the recovery window, and a restart drops the in-memory outage
records, so the next scan saw a past-grace one-shot with no claim and retired it
as missed: the job the outage skipped never fired.
Keep {"outage": True} instead. With no "at" it is a stale claim, so the job is
re-dispatched on the next tick, and _retire_expired_oneshot keeps it. At-most-once
is unchanged: the fire_claim CAS still fences every re-dispatch.
Gate r1 (PM12, 2c) Low finding.
* fix(cron): drop a departed profile's store state by its resolved store path
forget_homes rebuilt each store path from a departed home KEY
(Path(key) / "cron"). cron/AGENTS.md forbids rebuilding a home from a key
(hermes_home_key normcases), and the realpath only follows a symlinked cron/
while the home still exists: a profile whose home was deleted kept its degraded
record, holding the host-wide hermes.cron.store.writable gauge at 0.
register_ticked_homes now records each home's resolved store (new public
store_health.store_key) while the home exists and hands the departed stores to
forget_homes. gateway/cron_store_notices.py uses the same store_key against
record.store (already resolved), so one normaliser decides "is this the served
home's store" everywhere.
Gate r1 (PM12, 2c + quality + 2ab toothless) Low finding.
* test(cron): pin that the tick-lock probe names the lock file
The tick-lock case only asserted probe_store() returned an error, which any other
probe failure would also satisfy. Assert the error names .tick.lock so the case
proves that branch fired.
Gate r1 (PM12, quality) Low finding.
* test(cron): derive the run-claim expiry step from the TTL helper
timedelta(minutes=31) silently encoded ONESHOT_RUN_CLAIM_TTL_SECONDS = 1800; use
_oneshot_run_claim_ttl_seconds() + 60 as tests/cron/test_jobs.py already does, so
a TTL change cannot leave the step inside the live-claim window.
Gate r1 (PM12, quality) Low finding.
* fix(cron): keep store state another still-ticked profile shares through a symlink
Two profiles whose cron/ resolves to one store share a store key; when one left, its departure dropped the degraded record the remaining profile still needed (gauge flipped to writable, a second outage notice, and the earlier outage start lost for one-shot cover). Departed stores now exclude the ones still ticked. The notices test pins it (red without the set difference).
* refactor(cron): name forget_stores for what it takes; fix the clear_run_claim early-return comment
* test(cron): pass encoding= to every text read/write in the store-outage tests
* feat(telemetry): updates that stop before applying say why
Every pre-apply exit of `hermes update` records one closed token on the
receipt at the exit itself (update_receipt.record_stop_reason ->
stop_class), and shared_metrics_update.update_failure_class maps it to
the run's failure_class. aborted_before_apply stays only as the fallback
for an exit with no recorded reason.
Exits that fire before the receipt opens (update-lock refusal, Git
operation in progress, managed install) write no receipt, as before, but
now record a metrics-only row with the same closed class.
The parked receipt copy keeps stop_class, exit_code, the stop reason's
leading label and the restart/user-action flags, so a parked run
classifies the same as the in-process one. A run that committed and is
owed only an interrupt or the user's parked changes reads outcome
partial (failure_class interrupted / local_changes_parked), never failed.
* fixup: keep main.py and update_failure_class under their health caps
* test(telemetry): pre-apply exit classes, parked parity, receipt-less refusals; docs + smoke
- test_shared_metrics_update_stop_reasons: every stop token reads as its own class in-process
and through the parked copy (no free text kept), unknown tokens and post-apply tokens fall
back, a committed run that only owes parked changes is partial; the lock and Git-operation
refusals each record one row and leave the holder's latest.json byte-identical.
- test_shared_metrics_install_failures: a PermissionError stop reason now reads its own class
permission_denied (was os_error); ENOSPC reads disk_full; other OSErrors keep os_error.
- relay-shared-metrics.md: the new classes, partial outcome and parked fields.
- relay smoke: the failed update receipt names its fetch failure and the row asserts fetch_failed.
* fix(telemetry): review findings on pre-apply stop reasons
- The parked copy keeps a stop reason's label only when it is an exception type name (plus its
errno token) or one of Hermes' two fixed phrases; any other text before a colon reads '-'
(a free phrase like 'my project name:' used to survive).
- _exit_after_failed_branch_switch's branch_missing/checkout_move_failed probe cannot raise, so
the exit it names is unchanged even when git refuses.
- hermes update --plan/--check/--list-venv-holders on a managed install records no update run.
- Docs/contract say 'before the apply stage mark' and name the tokens that fire after git moved
and restored the tree; tests drive git_error_stop_class and _zip_stop_class directly.
* fix(telemetry): pre-receipt exits keep their initiator; index.lock permission errors; bare-interpreter updates park
- record_stop_without_receipt reads the update marker raw and tags initiator=desktop only when the
marker names this run's hand-off (delegate line = us, or HERMES_UPDATE_HANDOFF_PID / our parent),
so a Desktop-started early refusal reads kind=desktop and a CLI run refused by someone else's
update stays cli.
- _move_checkout_to classifies its git error with the shared classifier (git_output_stop_class):
index.lock 'File exists' -> git_index_locked, 'Permission denied'/EACCES -> permission_denied
(now a stop class), else checkout_move_failed.
- _publish_shared_metrics: an interpreter that cannot import the config reader (the -I -S bootstrap
that finalizes a dependency-preparation failure) never emits; it parks the bounded receipt in
pending_updates unless it can tell collection is off (loaded config says off, or no config.yaml),
and the next normal start applies the existing gate (report_pending_updates / begin_process purge).
* docs(telemetry): how update dashboards count a partial run
* test(desktop-update): a hand-off log read racing Add-Content retries, not fails
test_script_killed_before_publishing_the_delegate_runs_no_update failed on
the Windows arm64 lane with PermissionError reading
logs/desktop-update-handoff.log: windows.ps1 appends each line with
Add-Content, and a poll that lands while that write holds the file is a
Windows sharing violation, not a missing line. The poll helper now
retries the read (5 s budget) instead of failing the test on it.
Same test is green on main's latest arm64 run; nothing in this PR
touches the desktop-update scripts.
* test(e2e/desktop): give the build-fail updater its manual-outcome grace before asserting it exited
The build-fail spec polls 30 s for posix.sh to exit after the relaunched
app reports the outcome. That outcome is "manual" (the Desktop build is
owed), and finish() then runs launch_app (1.5 s acceptance) and
stop_ui leave-window, which sleeps the 15 s shim grace before tearing the
UI down: ~13 s of margin, and a slow Linux runner overran it (posix.sh
still alive at the 30 s mark, run 37619760207). The same code passed this
spec on bb177627828 and c4c2d9d8b54; the head that failed differs only by
a Windows test helper. 90 s keeps "no updater is left running" a real
check without racing the grace window.
* fix(cron): live-owner stale-claim reclaim measures silence, not run length
The #115692 reclaim released any running attempt whose claim was older than
max(3 x HERMES_CRON_TIMEOUT, script timeout, 2 h), even when the owner was
healthy and busy. Every cron job that legitimately ran longer than that was
marked `unknown` mid-run, its fire claim yanked from under it and the run
aborted with "lost its durable fire claim ownership" (two weekly jobs on one
install in a single day).
The run monitor now stamps `progress_at` on the execution row every 60 s
while the agent reports recent activity, and the sweep measures the stale
bound from that stamp (claim age for legacy rows). A wedged worker stops
stamping and is still reclaimed once the bound passes.
Shape: the stamper and the inactivity watchdog loop (which reads the same
idle signal) live in a new topical sibling cron/scheduler_liveness.py, so
cron/scheduler.py shrinks (4457 -> 4436 lines) instead of growing.
* fix(tools): normalize foreground terminal heartbeat
* fix(terminal): foreground heartbeat is dropped, not refused; notify intent still is
Follow-up to the cherry-picked #119201 (@KoNit-K): keep main's relocated
test file, flip the one invariant that pinned the refusal (a foreground
call with heartbeat>0 now runs with heartbeat=0 while notify=true stays
refused), reword the refusal so it no longer names heartbeat, and say in
the schema that heartbeat is ignored on foreground commands.
Why: the schema fix (4317ed0e71) only helps providers that materialize the
schema default. Models that copy a background call shape still send
heartbeat=60 on ordinary commands; this week 46 subagent sessions hit the
refusal 4 times each (173 refusals), 38 of them never ran a single shell
command, and the agents burned $316 / 1,498 model calls on retries.
* fix(desktop): setup cards stop redrawing in a loop
useSetupRows built a new row array on every render for the app-owned
lists (accent, layout, theme). That recreated the card's stage callback,
whose effect wrote $setupChooseStages, which re-rendered the card. The
rows are memoized on their inputs now, and stageSetupChoose skips the
atom write when a patch changes nothing.
* fix(onboarding): the first-task handoff follows a renamed owner profile
The setup marker saves the profile setup was made from. After that
profile was renamed, the saved name pointed at a missing directory, so
start_chat and reset targeted nothing. The owner now resolves through the
rename history (profile.yaml previous_names) that rename already records.
* fix(free-tier): a short free-tier 429 no longer blocks for a whole quota window
welcome_refusal_from_headers took the hourly bucket's reset as the wait
even when that bucket had requests left, so Retry-After: 4 with a
healthy bucket became a 2,000 s rate_limited refusal and tripped the
shared breaker. The wait now comes from an empty bucket's reset, else
Retry-After, else the body's wait.
* fix(desktop): a sign-in offer from the previous backend never opens
requestGateway keeps one identity across connection and profile
switches, so a free_tier.claim_nudge reply or status read that landed
after a switch could open the offer, or overwrite $freeTierStatus, for
the new backend. The offer timer, the claim and the finished-turn
re-read are tied to gatewayActivationEpoch() and drop a reply once the
route changes.
* fix(auth): persist profile single-use OAuth credentials when root store is empty
Fixes #103694: When a named profile runs auth add for a single-use
refresh provider (e.g. Anthropic, Codex, xAI) and the root store has
no existing rows for that provider, load_pool() sets _borrowed_root_ids = set().
Previously, add_entry() checked if borrowed_ids:, which evaluated to
False on an empty set. It fell back to _persist() -> persist_pool_entries(),
which routed to the update-only _update_root_pool_rows() on root. Because
root had no rows to update, the credential was dropped with no error and
never written to the profile's auth.json.
Checking if borrowed_ids is not None: ensures that profile-scoped pools
persist their new credential directly to the profile store via write_credential_pool(),
transitioning the profile to owning rows for that provider.
(cherry picked from commit b7467a2999832749f540d86530cf463ca0945b04)
(cherry picked from commit 9c604999e233cf6d74a83e4d7c141785b341b0d6)
* fix(auth): persist first profile OAuth row with empty root
(cherry picked from commit 5293489825af98e3e3f0e6cb485ffffa72d14833)
(cherry picked from commit 4144e14bfa7c021a92db5b0f66a75e2e0ac767d1)
* fix(auth): save a profile's first single-use login, and never print Added for an unsaved row
- Decide the profile claim when the row is written (the check from #101827)
and drop the None sentinel from #103712. On current main load_pool() leaves
a profile that borrows from an empty root at the default, so the sentinel
alone never reached the claim path (its test stays red).
- add_entry() re-reads the store after writing. A missing row raises
CredentialNotSavedError; `hermes auth add` exits with that message instead
of printing "Added", and the dashboard pool endpoint returns it as a 400.
- Tests: the two contributor tests become one invariant over every
SINGLE_USE_REFRESH_POOL_PROVIDERS provider (the row is readable, root
auth.json is byte-identical), plus the CLI success-line contract.
- Docs: profiles page, EN and zh-Hans.
Fixes #103694
(cherry picked from commit 0e250a005a3353564642ce70f100d89174a583f2)
* fix(auth): the forked-grant heal keeps a profile's own Anthropic login
`hermes -p <profile> auth add anthropic --type oauth` saves a
`manual:hermes_pkce` row in the profile. On the next load_pool(), the
forked-grant heal took every row whose source ends in `hermes_pkce` for a
copy of root's .anthropic_oauth.json whenever root's auth.json held no
Anthropic OAuth row. It moved the profile's token pair into root's file
and deleted the row from the profile: the CLI printed "Added", the login
was gone on the next turn, and root's file now held the profile's
account.
Only a `hermes_pkce` row comes from that file. `manual:hermes_pkce` is
owned by the pool and never written to the singleton
(_commit_anthropic_rotation matches the exact source for the same
reason), so the heal now matches the exact source too. The heal's own
docstring already promises to leave an independent `auth add` grant
alone.
This covers the empty-root case the previous commit enables, and main's
existing variant where root holds only an Anthropic API-key row next to
its .anthropic_oauth.json.
Co-authored-by: Enough <10966420+Enough1122@users.noreply.github.com>
* fix(auth): a profile's auth add never copies root's login into the profile
Root's login for nous, openai-codex and xai-oauth can live only in its
auth.json `providers.<id>` block, with no pool rows yet. In a profile
that borrows from root, load_pool() seeds it as a `device_code` entry
through the root fallback. The write-time claim in add_entry() kept
every entry except root's pool rows, so the profile's first `auth add`
wrote root's single-use refresh token into the profile's auth.json: a
forked grant (#100339) that the heal cannot match, because root has no
pool row for it.
- add_entry() writes only the added row when a profile claims a
single-use provider. Rows the profile seeds from its own sources are
seeded again on the next load.
- _seed_tokens_singleton() (openai-codex, xai-oauth) no longer seeds
root's providers block into a profile that owns its own rows, the
rule _seed_nous_singleton already applies. On current main such a
profile copied root's refresh token into its own pool on every load.
* fix(telemetry): update stage rows count the stage a failed run died in
hermes.update.run blamed 971 failed CLI runs in Sep 28 - Oct 5 on the apply
stage while hermes.update.stage held zero failed apply rows. Stage marks are
END marks: every pre-apply exit (fetch, channel, branch, merge, HEAD checks)
leaves no mark for the stage it died in, so the run row inferred the stage and
no stage row said it failed. A failed run whose failed_stage names a stage it
never marked now gets one failed row for it.
Rebuilt on the updater overhaul's followups model (#132386): a committed run
that owes follow-ups is a run-level success and its failed stage already has
its own mark, so only failed runs get the row; refusals never do.
* test: detached-writer custody tests read a pid from a beat that can never be empty
Mechanism: the detached build writer in the two "keeps custody until a
detached writer is gone" tests rewrote its heartbeat file in place with
pathlib.write_text, which opens with O_TRUNC and then writes. Custody
SIGKILLs that writer at an arbitrary instant; when the kill lands between
the truncating open and the write, `beat` is left empty and the teardown's
`beat.read_text().split()[0]` raises IndexError (suppress only covered
ProcessLookupError/ValueError). CI hit it on
test_a_ctrl_c_keeps_custody_until_a_detached_writer_is_gone[verbose-slow-cleanup]
(job 112764854735).
Repro: scratch copy of the writer with a 0.1 s sleep between open() and
write() -> both ctrl-c cases and the group-kill sibling fail with
IndexError at the teardown read on every run (5/5).
Fix: one shared _heartbeat_writer helper writes each beat to `<beat>.tmp`
and os.replace()s it into place, so `beat` always holds a complete
"<pid> <n>" once it exists. With the same injection moved onto the temp
write, all three cases pass 5/5. The ValueError suppress that papered over
a partial read is dropped.
* fix(tests): runner applies passthrough --ignore-glob; windows marker test stops sitting out 150s
Windows-only tests job 112764655375 (run 37612501107, a YAML-only catalog PR)
went red with "667 passed, 0 failed": test_desktop_update_windows_marker.py
hit the runner's 300s per-file cap mid-file ("(300s exceeded; process tree
SIGKILL'd)"), so its 19 tests were never counted and it was listed under
"no tests ran". Run 37612584210 died the same way. Clean runs of this file
take 220-300s (arm64 hit 296.5s and 299.8s), so it is at the cap on every run.
Two defects:
1. The desktop_updater lane gate never worked. tests-os.yml passes
--ignore-glob='*test_desktop_update_windows_*.py' after `--`, and
run_tests_parallel.py forwards it to `pytest <file>`. pytest applies
--ignore/--ignore-glob only while recursing directories, never to an
explicit file argument, so the flag did nothing. These hand-off tests ran
on every PR, including ones that never touched the updater, even though
the job printed "desktop_updater lane off: skipping ...". The runner now
applies --ignore/--ignore-glob to its own file list (pytest's fnmatch on
the absolute path, relative patterns anchored at the repo root, which is
the per-file pytest's cwd) and says how many files it excluded.
2. test_desktop_that_never_exits_is_not_relaunched_over waited out the
production 150s Desktop-exit ceiling. That one wait is half the file's
budget. The script already honours HERMES_UPDATE_DESKTOP_EXIT_SECONDS
("so the self-tests need not sit it out"), and the posix twin sets it to
2. Set it to 5 and assert that the script's refusal names 5s, so the
override is proven to reach the wait. The verdict under test (refuse
with 4, never relaunch) is the same at any ceiling.
Repro (Linux, CI-shape args):
python scripts/run_tests_parallel.py --files "<windows_marker>:<windows_cwd>:<other>" \
-- --ignore-glob='*test_desktop_update_windows_*.py' -m "platforms and not integration"
base: "Running 3 test files", both gated files run
fixed: "note: --ignore/--ignore-glob excluded 2 test files", "Running 1 test files"
New test_passthrough_ignore_drops_files_the_runner_hands_pytest_explicitly
(--files and discovery x glob/glob-spaced/path) fails 6/6 on base and
passes 6/6 fixed.
* fix(desktop): show a queued cron run row immediately after trigger
Run History only rendered sessions the backend had already materialized,
so an accepted Trigger now was invisible for the tens of seconds the
scheduler took to create the run session (#70826). The queued row is
painted from the click, settles when Run History observes a run that
started after it (any load — a fast 1s poll while queued, the cron.changed
broadcast on event-capable backends, or the visibility poll), and drops
at a bounded 90s timeout so a never-materializing run cannot wedge the
button. A failed request drops the row immediately. Pause deliberately
does not settle it: the pause response only persists enabled=false and
never cancels an already-claimed execution.
Queued design (queued-run row + polling + settle semantics) originates in
#70840 by @nv-cho; this ports it onto current main's
createCronTriggerController wiring.
Fixes #70826
* refactor(desktop-i18n): move billingBlock copy into en_billing sibling
Offset the cron queuedRun string growth in the over-cap en.ts/types.ts
facades per the code health ratchet: moved code keeps its cap.
* chore(desktop): refresh locales/_keys.desktop.json for cron.queuedRun
* fix(update): sweep stale git locks at the start of the run, not only before the fetch
The stale-lock / aborted-pack sweep ran only in the apply path, after the
snapshot. A run that died in that window left `.git/index.lock` behind with
nothing in the next run to remove it, so the following update failed with
"File exists" until an operator cleaned it up by hand. The sweep moves to the
top of `_cmd_update_impl`, ahead of the plan and the backup;
`clear_git_debris` already wraps locks and aborted-transfer pack temps (and
the partial-clone maintenance keys), and `_sweep_stale` skips everything
while a git process holds it, so the earlier position is safe.
(Salvaged from #132089 work; the receipt stop-reason half of the original
commit is now owned by the overhaul's receipt API and is not carried here.)
* fix(update): a killed git's index.lock no longer blocks the next update
`.git/index.lock` left by a killed git refused every later merge with
"File exists": the stale-lock sweep keeps any lock younger than 10 minutes,
so a `hermes update` killed mid-run (Windows crash cell `mid_fetch`, run
37573293632) or a version probe killed by its own 3 s timeout (#132089)
failed the next update at "Pulling updates...".
- `gitlock.release_dead_index_lock`: drop the lock as soon as the launch
repair's ownership proof (`_early_recovery._release_dead_index_lock`:
/proc fds + lock-keeping gits on Linux, lsof on macOS, the unlink itself on
Windows) shows no live git holds it. An interrupted tree move keeps its own
lock judgement. `_cmd_update_impl` calls it first, under the update lock.
- `version_info._git_version_info`: the read-only status probe runs as
`git --no-optional-locks status`, so a timed-out probe strands no lock.
Part of #132089 (the stale index.lock half; naming why a pre-apply exit fired is separate work)
* fix(update): on Windows a killed git's index.lock waits only for gits working in this checkout
Windows has no /proc to name a lock's holder, and its unlink-as-probe cannot see a
lock-keeping git with its fd closed (a `git commit` in the editor), so the reclaim keeps the
lock while a git.exe has its cwd or a path argument inside the checkout, and whenever the
scan cannot tell (no psutil, an uninspectable git). Any other git on the machine no longer
blocks it, unlike the age-floor sweep's machine-wide check.
* fix(update): only a killed update's index.lock is reclaimed at once
A lock nobody holds open is not proof its git is dead: a git that just created it may not have
opened it yet. The start-of-run reclaim now takes .git/index.lock only when the latest update run
died unfinished (receipt running/interrupted, owner gone) and the lock is newer than that run's
start. Every other lock is left to the age-floor sweep, and the update refuses on it truthfully
(e2e-upgrade test_live_index_lock_is_refused_truthfully went red on fd2214b0ad6).
* fix(update): the start-of-update index.lock reclaim never takes a live git's lock
Review round 2 on #133077. A killed update's receipt plus a younger lock let
release_dead_index_lock delete the lock of a live `/usr/lib/git-core/git-commit -a`
waiting in its editor: the lock fd is closed there, and the Linux holder scan only
knew `comm == git` with an argv subcommand, so the dashed form read as "no holder".
A competing `git add` then got in and the user's commit died "unable to write new
index file".
- The receipt gate stays necessary, never sufficient. The reclaim now also needs
proof that no git of ANY form works in this checkout (`_held_open(any_git=True)`):
cwd, path arguments (relative ones against the git's cwd) or GIT_DIR/GIT_WORK_TREE/
GIT_INDEX_FILE inside the checkout or its git dir, compared by path components.
A git we cannot place (unreadable cwd/environ, hidepid) keeps the lock.
- Git forms: `git`, a dashed `git-<sub>` from git-core, `git.exe`/`git-<sub>.exe`,
`-C`/`-c`/`--git-dir` globals, aliases (an unknown subcommand counts). The
launch-time repair's own scan inverts its list: only known readers are exempt.
- macOS: an empty lsof is no proof any more; `ps` lists every git (comm, so dashed
forms too) and any lock-keeping one keeps the lock.
- Windows: the psutil scan matches dashed and .exe gits, reads environ, and compares
by path components (`hermes-backup` was read as inside `hermes`).
- A failed retry no longer erases the evidence: `_killed_update_owns` reads the durable
per-run records (read-only) and asks about the last run that STARTED before the lock
was written, so a later failed run cannot discharge the killed one.
- The FIFO test helper kills its blocked git when its own setup assertion fails.
* fix(update): on macOS only a git working in this checkout keeps a dead index.lock
The lsof branch's ps scan counted every lock-keeping git on the machine,
because ps cannot say where a process works. A git commit open in any
other repository (or a sibling test worker's git on CI) therefore kept a
dead index.lock and blocked the launch-time repair after a killed update:
the macOS-only lane failed
test_a_merge_killed_writing_hermes_constants_is_repaired_by_the_next_launch
with "a running git holds the index".
Each candidate git's cwd now comes from `lsof -d cwd`, compared to the
checkout by path components (as on Linux); a candidate whose cwd cannot
be read while it still runs makes the answer unknowable, so the lock is
kept. Probe with real lsof/ps and a dashed git-commit waiting in its
editor: in another repo, 4b3c28c816c named it a holder and the fix does
not; in the checkout, both keep the lock.
* fix(update): the macOS lock scan finds ps when the launcher's PATH has none
The launch-time repair runs with whatever PATH the launcher got; the
macOS-only lane runs it with a PATH holding no git (as on the Windows
install that motivated the repair), so the bare "ps" call failed, the
scan answered "unknowable" and the dead index.lock of a killed update
was kept: test_a_merge_killed_writing_hermes_constants_is_repaired_by_the_next_launch
stayed red. ps is now resolved like lsof already is (which, then /bin/ps,
/usr/bin/ps). Probe: dead lock, PATH with neither git nor ps, at
ba2965261a0 the scan returned None (lock kept); now False (reclaimed).
* fix(update): review the macOS lock scan's lsof/ps lookups; keep it stdlib-only
The resolution-allowlist ratchet flagged the lsof lookup (moved from
_held_open to _held_open_lsof) and the new ps lookup in _ps_git_holder:
both are the stdlib-only launch-time repair asking the OS's own tools,
with an absent tool meaning "unknown" and the lock kept. Rows renamed
and added with that justification.
_still_running also imported psutil inside the launch-time repair, which
runs from hermes_bootstrap before app dependencies can import; it is now
a POSIX os.kill(pid, 0) probe (the lsof/ps branch never runs on Windows).
* fix(desktop): a setup card's typed-pick callback keeps a stable identity
The name card crashed the setup chat on a real Windows bundle (React #185,
maximum update depth): the card wrote its stage callback into
$setupChooseStages, which it also reads, and the callback depended on
rowLabels, a new object every render. 40512011b6 stopped the rows from
changing identity; this removes the dependency itself, so the callback
changes only with kind, mode, multi-select or the request. The labels
effect already stages the row names the callback used to rewrite.
The new test renders the real card: red on 17646902e1 (the crashing
build), green here and with either guard alone.
* test(ci): the CI replay names a silent step's exit code
workflow_steps.py is a helper module, not a test module, so pytest does not
rewrite its asserts: a replayed bash step that exits non-zero with empty
stdout/stderr fails as a bare 'AssertionError' with no exit code. Two
Windows arm64 os-tests runs (jobs 112635777217, 112764629268) failed this way
on unmodified baseline replays, and the NTSTATUS needed to tell a crashed
child (0xC0000142 / 0xC0000005) from a bash failure was lost. Put rc (decimal
and hex), the bash used, and both streams in the message.
* test(ci): the CI replay does not run Git Bash under x64 emulation on Windows arm64
Mechanism: Git for Windows on arm64 ships native git.exe but an x86-64 MSYS
usr/bin/bash.exe, which runs under the WoA x64 emulator. With the os-tests
lane's 16 parallel workers, a replayed step's bash dies before running a line
with 0xC000026F (STATUS_WX86_INTERNAL_ERROR) or 0xC0000005 and empty
stdout/stderr, so an unmodified baseline replay fails as a bare
AssertionError (jobs 112635777217 and 112764629268, two different nodeids).
Repro (throwaway workflows on the real runners, previous commit's diagnostics):
- windows-latest-32-arm-core, 16 copies of the file's replay tests at once,
4 rounds: 21/64 copies red, every failure 0xC000026F.
- Same load, windows-latest-32-core (x64): 0/64.
- Passing SYSTEMROOT/WINDIR or the full Windows env to the step: 12/48 and
17/48 vs 15/48 base, so the stripped env is not the cause.
- One process, 32 threads, 600 replays: 0 crashes; it takes concurrent
processes, i.e. host-wide emulator load.
Seam: _NATIVE_WINDOWS_TOO keeps platforms("linux", "windows") and adds a
skipif for win32 + native_arch() == arm64. The x64 Windows row still runs the
replay on the Windows layout (python.exe-only venv, Git Bash), Linux arm64
keeps it (native bash), and the routing guard that pins the marker passes.
* fix(free-tier): a setup chat turn does not start the sign-in offer clock
The setup chat's last turn (the fork answer and the handoff) finished clean
on the free tier, so it was recorded as the first task. The offer then came
due three minutes after setup ended, which lands in the middle of the first
real task when that task runs longer than three minutes. Seen live on a
Windows dev build: first_task_at was the setup chat's end, not the task's.
Turns in the setup profile's sessions are no longer tasks, as the offer
module already intended ("normally the task setup hands off to").
* i18n(desktop): translate cron.queuedRun in every overlay
Main added the Run History 'Queued run' label (44c1ad7f84) in English only;
this branch's overlay-completeness ratchet fails on a new untranslated key.
* Fix reasoning markup rendering in dashboard
* fix(web): scope structured reasoning UI to assistant messages
Only assistant content with known reasoning tags uses StructuredReasoning.
User and tool messages with literal <action>/<result> tags stay on Markdown.
* test(web): behaviour coverage for structured reasoning transcript rendering
RED on main: the expanded session transcript renders assistant
reasoning with literal <thinking>/<action>/<result> wrapper tags.
Keeps the user-role invariant (literal tags stay on the Markdown path).
* refactor(web): move session source config to SessionsPage_sources sibling; satisfy code-health ratchet
* fix(desktop): hide unknown cloud agent status
Omit the gateway status description when cloud discovery cannot provide a meaningful state, while preserving valid status labels. Add renderer coverage for both unknown and known states.
* test(desktop): cover empty cloud agent status
* fix(pm): reclaim dependency state of deleted checkouts (pm gc + startup worktree prune)
Every checkout that boots Hermes commits its own dependency state under
<home>/installs/<sha256(path)[:16]>/ (venv, test venv, PM runtime; ~200 MB
each) and nothing removed it when the checkout went away. On a host that
runs `hermes -w` campaigns the directory held 499 entries / 92 GB, of which
402 (80 GB) belonged to worktrees and scratch clones that no longer exist.
The state dir already records its checkout in inputs/.project-root. A new
pm/install_states.py lists entries whose recorded checkout is gone and
removes them unless something could still be reading them (install lock
taken, or a generation lease held). `pm gc` runs it and reports the count;
the startup worktree pruner runs it after its own reaping so a removed
worktree's state follows the tree instead of outliving it.
* fix(pm): orphan reclaim runs on every prune pass and keeps checkouts it cannot see
Follow-up to the review of this PR:
- The startup pruner returned early (no .worktrees/, or no tree past 24h) before the
reclaim ran, so a tree removed by hand kept its state until some other tree aged out.
_prune_stale_worktrees now runs the reclaim in a finally around the prune phases.
- A missing recorded path is not proof of deletion: a container sharing the data root
(-v ~/.hermes:/opt/data) cannot see the host's checkout, and gc there deleted the
host's live dependency environment. A state is an orphan only when its checkout sat
under the data root that owns installs/, or directly in a .worktrees/ dir that still
exists.
- PM runtime generation leases (pm-runtime/generations/*/.leases) now count as held,
so a worker still running from a deleted checkout's runtime keeps it.
* fix(pm): an interrupted orphan reclaim leaves a dir the next pass still finds
The startup reclaim runs on a daemon thread, so a hermes exit can cut a removal
short, and Windows refuses to unlink open files. shutil.rmtree takes children in
scandir order, so inputs/.project-root (the only record naming the checkout)
often went first and the remainder was invisible to every later pass.
Everything else is removed first; the record goes only once nothing else is left.
* fix(kanban): only sweep a parent workspace once the parent itself is terminal
The deferred parent sweep added for #33774 reaps a parent's scratch or
worktree workspace as soon as it has no active children left, without
checking the parent's own status (#133501). Archiving or finishing the
last active child of a parent that is still running/blocked/review
deletes the parent's live workspace from under its worker; the shared
wor…
mrkillbob
added a commit
to Coding-Sensations/hermes-agent
that referenced
this pull request
Oct 9, 2026
* chore(plugin-catalog): bump switchbot-control to bb68cb4
* chore(catalog): review disclosure for switchbot-control
* fix(update): a stash with no Python is never rejected by the import probe (#130101)
The stash-restore health check compared two independent critical-module
import probes and rejected the restore on any difference, even when the
stash restored no Python at all. The probe's outcome also moves with
install state between the two runs (launch preparation relaunching under
the live update surfaces as SystemExit(0); a dependency sync triggered by
the first probe), so a docs-only stash was rolled back, the stash parked,
the gateway restart skipped and the update exited 1.
Only compare imports when the restore touched Python (the same
restored-Python set the syntax check already uses). A restored .py that
breaks an import is still rejected.
* fix(update): skip the restore import check only for non-runtime files
The #130101 exemption skipped the post-restore import comparison whenever
the stash restored no .py file. A restored native extension (toolsets.so
shadowing toolsets.py), bytecode, a .pth or a data file read at import is
not .py, so such a restore was accepted, the stash dropped, and run_agent,
model_tools and toolsets were left failing to import.
Skip the comparison only when every path the stash restores (read from the
stash commit: worktree, index and untracked trees, renames split) has a
closed non-runtime suffix (docs, images), is not a symlink, and lives in a
directory HEAD already tracks. Anything else is compared as before. A
docs-only restore still succeeds.
* docs: stop presenting Desktop Light as a shipped download
The installation page said "Light is a remote-only build variant" next to
the download instructions, which read as if a remote-only Desktop client
can be downloaded. Light exists only as a build target with no release
leg, and there are no plans to publish it. Replace the sentence with how
remote use actually works today (install a package, connect from Settings
-> Gateways), drop "Light packages" from the uninstall note, and mark the
variant as unpublished in the desktop README.
* plugin-catalog: add Agent Hold Em
* chore(catalog): review disclosure for agent-hold-em
* feat(plugin-catalog): add hermes-field-notes — pitfalls and local core patches
* plugin-catalog: hermes-field-notes 1.2.0 (patch recipes)
* plugin-catalog: hermes-field-notes 1.2.1 (review fixes)
* plugin-catalog: hermes-field-notes — README author section
* chore(catalog): review disclosure for hermes-field-notes
* chore: map alrcatraz for salvage of #132751
* fix(agent): let an explicit tool_reason label a soft interrupt
interrupt() derived the published tool-interrupt reason from whether a message
was attached, so on the soft path a system producer had no way to label itself:
passing a message booked the stop as "user sent a new message", passing none
booked it as "user interrupt". Both live in USER_INTERRUPT_REASONS, so
interrupt_issuer() returned None and the turn exit reason became
interrupted_by_user — a system abort recorded as a human stop.
An explicit tool_reason now wins on the soft path as it already did for
hard_cancel, keeping the message heuristic only as the fallback.
(cherry picked from commit 495f1a84a15a4bf88295af1e06f225e25d57715c)
* fix(agent): attribute the terminal batch-timeout abort to the guard
The sequential-batch timeout guard stopped the turn with a bare
interrupt(message), which the soft-path heuristic read as "user sent a new
message". The abort was therefore recorded as interrupted_by_user and the
skipped calls told the user they had sent a message they never wrote.
Pass the guard's own tool_reason so the exit reason names the real issuer and
the skipped-call notice can describe it.
(cherry picked from commit a756f781b2630ea3059c8e56df132c5d8ba35844)
* fix(agent): render skipped-call notices from the recorded reason
Both sequential skip sites hardcoded a user-initiated cause ("User sent a new
message" / "skipped due to user interrupt"), so any system abort that reached
them asserted an action the user never took.
Render the notice from the published reason instead: a system issuer describes
itself, and only the genuinely user-initiated reasons keep the user-facing
wording.
(cherry picked from commit 8efe429e2a485a183c7ad46522d9a5c99cef6a9c)
* fix(agent): hide the synthetic interrupt close from the transcript
close_interrupted_tool_sequence() appends a synthetic assistant turn when an
interrupted tail is a raw tool result, so the next user message does not land as
tool -> user. The row carried a user-visible "Operation interrupted." string and
was persisted as an ordinary assistant message, so background preemption —
session reaping, watchdogs, batch guards — surfaced in the transcript as an
interruption the user was expected to act on.
Store it in the same hidden shape turn_api_call.py already uses for interrupt
placeholders: content="" plus display_kind="hidden" keeps it out of rendered
transcripts, while the api_content sidecar carries the LLM-visible text and is
substituted at API-build time. The alternation guarantee is unchanged.
(cherry picked from commit 22502b1c40c26a08eefc3d35cee067bd1d889bfc)
* test(agent): cover soft-interrupt attribution and skipped-call wording
The exit reason and the skipped-call notice must both name the real issuer: a
system producer that stops the turn softly labels itself via tool_reason, while
a message-carrying soft interrupt with no reason stays a human stop. The wording
derives from the recorded reason, so a system abort never tells the user they
did something they did not.
(cherry picked from commit ec65b95cfb312ebb7a276a26b163df0eaad65b55)
* fix(agent): render concurrent skip notices from the recorded reason
The sequential path was converted to interrupt_skip_wording() but the
concurrent batch kept three hardcoded user-blaming strings: the pre-batch
skip (execute_tool_calls_concurrent), the unfilled-slot notice
(_unfinished_tool_result) and emit_cancelled's hook message. A system
producer that stops a parallel batch — watchdog, lease loss, SSE
disconnect, or any segment routed through execute_tool_calls_segmented —
still told the model "skipped due to user interrupt" while
interrupt_issuer() booked the same turn as interrupted_by_system. Render
all three from the recorded reason like the sequential siblings; the
observability enums (error_type/hook_error_type) stay untouched.
Also harden the interpolation the wording rides on: _append_skipped_tool_results
substitutes {name} with str.replace instead of str.format, and
interrupt_skip_wording escapes braces in caller-supplied reasons, so a
free-form tool_reason can never raise KeyError/IndexError mid-turn.
(cherry picked from commit b785bccbca6fd997921411198d277c39d3d93261)
* fix(agent): hide only the placeholder close, keep caller banners visible
close_interrupted_tool_sequence hid every synthetic closing turn behind
display_kind="hidden", but four of its five callers pass a real user-facing
notice — the truncation terminal ("Response truncated — …"), the context-
overflow partial text, tool-validation partials and abort_turn_on_interrupt's
"Operation interrupted: …" lines. Hiding those removed the turn's only
explanation from every transcript surface.
Narrow the hidden shape to what it was for: an empty final_response or the
bare interrupt placeholder (which previously rendered as the "Operation
interrupted." bubble users never asked for). Any other text is rendered
verbatim as before. Alternation safety is unchanged in both branches.
(cherry picked from commit a637af2ef44c1f0213c3138c458d6f6c48650988)
* fix(agent): stop the interrupt placeholder echoing back as a reply
A hidden assistant row is hidden from the transcript, not from the request:
the projection substitutes its api_content sidecar into content on every call.
The value chosen for that slot in #88955 was "[response interrupted]" - a short
natural-language phrase sitting in the model's own prior-turn position, which
models reproduce verbatim: an ordinary instruction answered with clean
finish_reason=stop and nothing but the placeholder (#132949). The same hazard
was already recognised twice in this codebase - the redirect scaffold is
barred from assistant text because "the model echoes it" (#81841), and a
runaway repetition partial is replaced by a self-describing label rather than
replayed (#112764) - but the placeholder was never given the same treatment.
Change the placeholder to a structural label the model has no reason to say,
and retire rows already persisted under the pre-change spellings: the prep
filter that dropped scaffold ghosts now also drops hidden rows carrying the
legacy placeholder or the scaffold marker.
Dropping is not always safe. A hidden close appended after a tool result is
the only thing separating that tool message from the next user turn; removing
it recreates the tool -> user alternation violation that makes strict providers
ignore prior context (#48879) - the exact failure the close exists to prevent,
and one repair_message_sequence deliberately leaves alone. When the neighbours
would form that pair, rewrite the row's text instead of dropping it. The
rewrite targets the current placeholder, so a neutralised row never re-triggers
the filter.
(cherry picked from commit 227e9a1ed64e0d75594d031423ecd2f5c74cd0b9)
* test(agent): cover replay-echo retirement, and pin behaviour not wording
The retirement filter gets five cases: a legacy placeholder row between two
user turns is dropped; the same row after a tool result is kept with its text
rewritten (dropping it would hand the provider the tool -> user pair that
close_interrupted_tool_sequence exists to prevent); a row carrying the current
placeholder is left untouched, which is what keeps #88955's re-heal guard
intact and proves the rewrite cannot re-trigger itself; a scaffold ghost still
drops as before; and a visible caller banner is never treated as a ghost.
Verified discriminating: reverting the filter to the scaffold-only version
fails all five, restoring it passes.
The existing #88955 assertions compared against the literal
"[response interrupted]" in four places. That couples the tests to one chosen
spelling - any wording change, including this one, breaks them for the wrong
reason. They now compare against the constant, asserting what the contract
actually says: the durable row stays empty and hidden, the wire copy carries
the neutral payload the writer stamped, and neither is ever the interrupt
scaffold (#81841).
(cherry picked from commit e4ebf0138b43547dbf1db36eaa84beb5c80bbfb7)
* fix(agent): drop the diagnostic message from the batch-guard interrupt
The terminal batch-timeout guard still passed "terminal batch tool did not
complete" as the interrupt message. That text lands in _interrupt_message,
which the gateway (_run_agent_drain_pending) and the CLI
(_chat_resolve_interrupt) re-queue as the user's next turn, so a guard abort
injected a fake user message. Call interrupt(tool_reason=...) with no
message, and use a slug-safe reason so the issuer reads
interrupted_by_system(terminal_batch_timeout) rather than
terminal_batch_aborted:_call_timed_out.
Message-less guard shape from #130209.
Co-authored-by: Yuan Li <dskwelmcy@163.com>
* test(agent): align skip-notice fixtures with the recorded-reason wording
Skipped-call notices now render from _tool_interrupt_reason instead of a
hardcoded "skipped due to user interrupt". The concurrent pre-flight stub
never recorded a reason, so it fell to the "Turn interrupted" fallback and
the stale assertion failed; record the new-message reason the real soft
path publishes and assert its wording. The ACP fixture is cosmetic but
should show the shape the executor actually emits.
* fix(agent): retire replay-echo ghosts without mutating the transcript
_neutralise_replay_echo_ghosts rewrote the caller's list in place
(seq[:] = out) and overwrote content/api_content on durable row dicts.
Those dicts are the live session rows the SessionDB flush cursor and
_DB_PERSISTED_MARKER track, so an in-place edit silently diverges the
in-memory row from what state.db holds. BASE built a new list here; keep
that contract: return a filtered list and neutralise a shallow copy of the
row, which only shapes this request's replay.
* test(agent): trim the soft-interrupt salvage tests to two invariants
Keep one attribution test that goes red on BASE (a tool_reason soft
interrupt is booked to the system issuer with no requeued message) and one
replay-echo test (a legacy placeholder row next to a tool tail is
neutralised on a copy, never dropped into tool -> user, and the durable row
is left untouched). The remaining wording/brace/close-row cases pin
implementation detail already covered by the updated finalizer, steer and
concurrent-interrupt tests.
* fix(agent): render skip wording verbatim instead of brace-escaping it
Fold finding 3 (B02 gate): interrupt_skip_wording doubled braces on the
assumption the wording reaches str.format, but every consumer substitutes
{name} with str.replace, so a reason like "a{b}" rendered as "a{{b}}".
Drop the escape and its docstring claim, and look user-stop wording up
directly in _USER_STOP_WORDING.
* fix(agent): attribute Ctrl-C tool cancellation to a user interrupt
Fold finding 2 (B02 gate, sibling of #130207): both KeyboardInterrupt
handlers in tool_executor call agent.interrupt("keyboard interrupt").
With a message and no tool_reason, interrupt() records "user sent a new
message", so the skipped-call notices for a Ctrl-C claimed the user had
sent a new message. Pass tool_reason=_REASON_USER_INTERRUPT so the
recorded reason (and the rendered wording) is "User interrupt"; the
issuer stays None, so attribution to the human is unchanged.
* fix(agent): neutralise legacy interrupt placeholders instead of dropping them
Fold finding 1 (B02 gate, regression): the replay-echo filter now matches
the legacy "[response interrupted]" placeholder and DROPPED the hidden row
unless its neighbours were tool -> user. The common persisted redirect
shape user -> hidden(placeholder) -> user(correction with checkpoint
api_content) then became user -> user, and _merge_consecutive_users merged
the pair: the correction's checkpoint sidecar was discarded and the first
stored user row was rewritten in place.
Always neutralise a copy ({content: "", api_content: placeholder}) and
never drop: role-safe in every neighbour shape, and the echoable text is
still gone. The kept replay test now also covers the redirect shape
(3 rows survive, checkpoint intact, stored dicts untouched).
* refactor(agent): tidy the interrupt-placeholder fold-ups
Fold finding 4 (B02 gate, low): behaviour-neutral cleanups.
- close_interrupted_tool_sequence: the hidden branch only runs when the
stripped text is empty or already the placeholder, so `stripped or
_INTERRUPTED_PLACEHOLDER` is always the placeholder; say so.
- tool_executor: compute interrupt_skip_wording once for the cancelled
outcome instead of twice.
- move _neutralise_replay_echo_ghosts out of prepare_iteration to module
level (lazy imports kept inside to avoid the conversation_loop cycle).
- the finalizer test asserts the exact placeholder, not just non-empty.
* test(agent): follow the renamed interrupt placeholder in the send-time pad tests
Gate r2 finding 1 (High, regression): a4c687126c renamed the non-final
empty-turn placeholder from "[response interrupted]" to
"[interrupt: no assistant output for this turn]", but two send-time pad
tests in test_partial_stream_finish_reason.py still asserted the old
literal and went red on the stack head. Compare against
agent.agent_runtime_helpers._INTERRUPTED_PLACEHOLDER so the tests track
the constant, not a spelling.
A full grep of tests/ and prod for the old wordings also found a healed
prefill fixture in test_thinking_only_sanitizer.py (still green, but it
models a stub the repair can no longer produce) and a stale docstring in
test_steer.py; both now use the current placeholder.
* test(agent): drive the real batch-timeout guard in the soft tool_reason test
Gate r2 finding 2 (toothless): 482a748431 stopped the terminal batch-timeout
guard from passing a diagnostic message to interrupt(), because
_interrupt_message is what the gateway drain and the CLI re-queue as the
user's next turn. The existing test only called agent.interrupt() directly,
so reverting the guard hunk left it green.
Route the existing test through _run_sequential_tool_execution_middleware
with a prepared terminal call whose worker never settles, so the guard
itself publishes the interrupt; the issuer and `_interrupt_message is None`
asserts now go red when the guard hunk is reverted.
* fix(agent): publish the Ctrl-C reason before the sequential cancel hook
Gate r2 finding 3: in _run_sequential_call's KeyboardInterrupt handler,
ref.emit_cancelled() ran before agent.interrupt(..., tool_reason=user
interrupt), so the post_tool_call hook rendered its wording from an unset
reason and reported "Tool execution cancelled. Turn interrupted" while the
concurrent handler (and 47db3231e2's intent) report "User interrupt".
Publish the interrupt first, matching the concurrent handler.
The existing cancelled-hook test now pins the hook's error_message.
* test(agent): pin the visible-banner branch of close_interrupted_tool_sequence
Gate r2 finding 4 (toothless): f8aea0e81e keeps a caller-supplied banner
("Response truncated — …", partial-delivery text) visible and only hides
the bare placeholder close, but nothing exercised the banner branch, so
reverting it to the always-hidden shape stayed green.
Extend the existing alternation test: a banner close must land as plain
visible content with no display_kind. Red with f8aea0e81e's hunk reverted.
* refactor(agent): match the replay-echo test docstring to the neutralise-only filter
Gate r2 finding 5: d482ce366a made the prep filter neutralise legacy hidden
rows on a copy instead of dropping them, but the module docstring still
described retiring rows except where removal would form tool -> user.
State the actual contract (never drop: removal could form tool -> user or a
user -> user pair that repair merges) and drop _hidden_row's content /
api_content kwargs that no caller passes.
* refactor(agent): move interrupt placeholders into agent_runtime_helpers_placeholders to keep files under the size cap
Code-health ratchet: agent/agent_runtime_helpers.py was 3787 > 3776 and
tests/agent/test_run_agent.py 7014 > 7012 (both over the 2,000 target, so
they may only shrink vs main).
- _INTERRUPTED_PLACEHOLDER / _LEGACY_INTERRUPTED_PLACEHOLDER (and their
rationale comment) move byte-identically into the leaf sibling
agent/agent_runtime_helpers_placeholders.py. Every importer (runtime
helpers via late import, conversation_loop, message_sanitization,
turn_api_call, turn_iteration_prep and five test files) now reads the
sibling; no shim is left in the facade. Nothing patches the old path.
- test_keyboard_interrupt_emits_cancelled_post_tool_hook moves unchanged
into tests/agent/test_run_agent_interrupt_hook.py, reusing the facade's
agent fixture.
* fix(agent): only relabel a prepared-batch abort as a batch timeout on timeout
The batch-guard tail after _poll_sequential_future is shared by the
"timeout" and "interrupted" outcomes. On a user stop (Ctrl-C, /stop, a
new message) whose prepared terminal worker had not settled after the 3s
grace, the unconditional agent.interrupt(tool_reason="terminal batch
timeout") republished the interrupt: the turn was booked
interrupted_by_system(terminal_batch_timeout), _interrupt_message (the
user's queued next message) and _pending_redirect were nulled, and the
wrong reason was re-fanned to workers.
Keep batch.close() unconditional, but only call interrupt() on the
timeout branch; on the interrupted branch the stop is already published.
Public review blocker on #133271 (@ahrazzle, executed repro; @Enough1122).
* refactor(agent): correct the stale placeholder-consistency comment
The moved comment on _INTERRUPTED_PLACEHOLDER claimed it was "kept
identical to the stub placeholder in chat_completion_helpers"; that
module has no such literal (the claim was already stale on main and was
moved verbatim). State what the constant actually is instead.
Trivial note from the public review on #133271 (@ahrazzle).
* fix(agent): stop terminal batch preparation from overwriting a user stop
terminal_approval_batch caught both _CancelledPreparation and TimeoutError
and called agent.interrupt(str(exc)). _CancelledPreparation is raised
because a user interrupt is already pending, so the user's queued message
and redirect were replaced by "Terminal approval preparation cancelled…",
which the gateway then re-queued as a fake user turn. The TimeoutError
branch booked a system timeout as a user stop with a fake requeue — the
#130207 bug class itself, at the sibling site the batch guard fix missed.
Now: close the batch always; only on timeout publish a message-less
interrupt with tool_reason="terminal batch preparation timeout"; do
nothing on cancellation (the stop is already published).
Gate rP finding 1 (2c).
* test(agent): pin system attribution in sequential skip notices
The two _execute_tool_calls_sequential skip notices (stop before a call,
and "remaining tool call(s)" after one) were rewritten to render the
recorded interrupt reason (6eee1fb6b0), but reverting either back to the
hardcoded "due to user interrupt" / "User sent a new message" left every
kept test green. Extend the system-attribution test to drive both sites
after a tool_reason stop; each reverted site now fails the test.
Gate rP finding 2 (2ab).
* test(agent): pin system attribution in unfinished concurrent slot results
_unfinished_tool_result renders the recorded interrupt reason for a slot
no worker filled (ec51b7d021), but hardcoding the old user wording back
left every kept test green. Extend the system-attribution test to call it
after a tool_reason stop and assert both the row and the post_tool_call
error_message say "Turn aborted — …".
Gate rP finding 3 (2ab).
* test(agent): pin the Ctrl-C attribution on the concurrent tool path
fed76c1df2 made both KeyboardInterrupt sites publish
tool_reason="user interrupt", but only the sequential one was covered:
reverting the concurrent worker's tool_reason left the suite green
(the hook read "User sent a new message"). Parametrize the existing
cancelled-hook test over the concurrent path with two calls.
Gate rP finding 4 (2ab).
* refactor(agent): derive user-stop skip wording from USER_INTERRUPT_REASONS
_USER_STOP_WORDING restated the three USER_INTERRUPT_REASONS keys with
their values capitalised, so a new human-stop reason had to be added in
two places or it would render as "Turn aborted — …". Drop the mirror
and capitalise the recorded reason; output is identical for all three
keys (probe-B02-5.py).
Gate rP finding 5 (reuse).
* refactor(agent): share the hidden interrupt placeholder row
message_sanitization.close_interrupted_tool_sequence and the
turn_api_call repetition-loop exit each spelled the same hidden
assistant row literal (content "", display_kind hidden, api_content
_INTERRUPTED_PLACEHOLDER), and the conversation loop's redirect built it
field by field. Add hidden_interrupt_placeholder_row() to the leaf
placeholders module and use it at all three sites so the shape cannot
drift between them.
Gate rP finding 6 (reuse).
* refactor(agent): import the interrupt placeholder at module level
agent_runtime_helpers (two sites) and message_sanitization imported
_INTERRUPTED_PLACEHOLDER inside the function body. The placeholders
module is a leaf with no imports, so there is no cycle to dodge; import
it once at module level (each module still imports cleanly in a fresh
interpreter). conversation_loop keeps its local import.
Gate rP finding 7 (quality).
* docs(agent): say skipped-result content uses str.replace for {name}
The _append_skipped_tool_results docstring still said content "is
formatted with {name}", but the body substitutes with str.replace so a
recorded interrupt reason containing braces renders verbatim. State
that, so nobody "simplifies" it back to str.format.
Gate rP finding 8 (quality).
* test(agent): move the brace-rendering check after the attribution scenarios
The verbatim "{x}" wording check sat between the batch-guard timeout and
the user-stop scenario, mutating _tool_interrupt_reason mid-story. Move
it to the end, alongside the other wording assertions.
Gate rP finding 9 (quality).
* refactor(agent): hoist the hidden placeholder row import and say why a timeout carries no message
Gate rQ quality Lows: the placeholders module imports nothing, so conversation_loop can import it at module level like turn_api_call; the terminal-approval comment now says callers re-queue _interrupt_message, so a system stop must not set one.
* fix(cron): fire an outage-skipped one-shot whose fire claim never landed
Follow-up to #133283 (greptile review). The due scan stamps a one-shot's
run_claim; when the later fire-claim save fails the run never starts. The
next saved scan skips it on that live claim and ends the recovery window,
so once the claim expires _retire_expired_oneshot saw a stale claim past
grace and skipped it forever: neither fired nor retired.
The scan now marks a run claim it stamps past grace (only an outage lets a
one-shot through there) with outage=true; such a claim without a fire claim
keeps the job covered, so it fires once the claim expires. A fire claim
still means the run may have started, so at-most-once is unchanged.
* fix(cron): report an unwritable tick lock in cron status and doctor
Follow-up to #133283 (greptile review). When only cron/.tick.lock is
unwritable (e.g. root-owned after a sudo run, in a writable cron dir) the
tick degrades and returns 0, so the ticker records a success; probe_store
checked only jobs.json, so `hermes cron status` said jobs would fire and
`hermes doctor` reported a writable store while nothing ran.
probe_store now treats an existing tick lock it cannot write the same way
as a read-only jobs.json target: both status and doctor surface it with the
usual fix hint.
* fix(gateway): send cron store notices for a symlinked cron/ directory
Follow-up to #133283 (greptile review). Store records are keyed by the
resolved store path, but the notice sender rebuilt the owning profile from
that path's parent. A profile whose cron/ is a symlink to another disk has
a foreign parent, so neither its outage nor recovery notice reached any
home channel; forget_homes had the same mismatch and never dropped such a
departed profile's record.
Both now match by resolving each served home's own cron/ path.
* fix(cron): keep the outage marker when a skipped one-shot's dispatch fails
clear_run_claim (the dispatch-failure path: interpreter shutdown, execution
creation failure, pool.submit failure) set run_claim to None. The claiming scan
had already ended the recovery window, and a restart drops the in-memory outage
records, so the next scan saw a past-grace one-shot with no claim and retired it
as missed: the job the outage skipped never fired.
Keep {"outage": True} instead. With no "at" it is a stale claim, so the job is
re-dispatched on the next tick, and _retire_expired_oneshot keeps it. At-most-once
is unchanged: the fire_claim CAS still fences every re-dispatch.
Gate r1 (PM12, 2c) Low finding.
* fix(cron): drop a departed profile's store state by its resolved store path
forget_homes rebuilt each store path from a departed home KEY
(Path(key) / "cron"). cron/AGENTS.md forbids rebuilding a home from a key
(hermes_home_key normcases), and the realpath only follows a symlinked cron/
while the home still exists: a profile whose home was deleted kept its degraded
record, holding the host-wide hermes.cron.store.writable gauge at 0.
register_ticked_homes now records each home's resolved store (new public
store_health.store_key) while the home exists and hands the departed stores to
forget_homes. gateway/cron_store_notices.py uses the same store_key against
record.store (already resolved), so one normaliser decides "is this the served
home's store" everywhere.
Gate r1 (PM12, 2c + quality + 2ab toothless) Low finding.
* test(cron): pin that the tick-lock probe names the lock file
The tick-lock case only asserted probe_store() returned an error, which any other
probe failure would also satisfy. Assert the error names .tick.lock so the case
proves that branch fired.
Gate r1 (PM12, quality) Low finding.
* test(cron): derive the run-claim expiry step from the TTL helper
timedelta(minutes=31) silently encoded ONESHOT_RUN_CLAIM_TTL_SECONDS = 1800; use
_oneshot_run_claim_ttl_seconds() + 60 as tests/cron/test_jobs.py already does, so
a TTL change cannot leave the step inside the live-claim window.
Gate r1 (PM12, quality) Low finding.
* fix(cron): keep store state another still-ticked profile shares through a symlink
Two profiles whose cron/ resolves to one store share a store key; when one left, its departure dropped the degraded record the remaining profile still needed (gauge flipped to writable, a second outage notice, and the earlier outage start lost for one-shot cover). Departed stores now exclude the ones still ticked. The notices test pins it (red without the set difference).
* refactor(cron): name forget_stores for what it takes; fix the clear_run_claim early-return comment
* test(cron): pass encoding= to every text read/write in the store-outage tests
* feat(telemetry): updates that stop before applying say why
Every pre-apply exit of `hermes update` records one closed token on the
receipt at the exit itself (update_receipt.record_stop_reason ->
stop_class), and shared_metrics_update.update_failure_class maps it to
the run's failure_class. aborted_before_apply stays only as the fallback
for an exit with no recorded reason.
Exits that fire before the receipt opens (update-lock refusal, Git
operation in progress, managed install) write no receipt, as before, but
now record a metrics-only row with the same closed class.
The parked receipt copy keeps stop_class, exit_code, the stop reason's
leading label and the restart/user-action flags, so a parked run
classifies the same as the in-process one. A run that committed and is
owed only an interrupt or the user's parked changes reads outcome
partial (failure_class interrupted / local_changes_parked), never failed.
* fixup: keep main.py and update_failure_class under their health caps
* test(telemetry): pre-apply exit classes, parked parity, receipt-less refusals; docs + smoke
- test_shared_metrics_update_stop_reasons: every stop token reads as its own class in-process
and through the parked copy (no free text kept), unknown tokens and post-apply tokens fall
back, a committed run that only owes parked changes is partial; the lock and Git-operation
refusals each record one row and leave the holder's latest.json byte-identical.
- test_shared_metrics_install_failures: a PermissionError stop reason now reads its own class
permission_denied (was os_error); ENOSPC reads disk_full; other OSErrors keep os_error.
- relay-shared-metrics.md: the new classes, partial outcome and parked fields.
- relay smoke: the failed update receipt names its fetch failure and the row asserts fetch_failed.
* fix(telemetry): review findings on pre-apply stop reasons
- The parked copy keeps a stop reason's label only when it is an exception type name (plus its
errno token) or one of Hermes' two fixed phrases; any other text before a colon reads '-'
(a free phrase like 'my project name:' used to survive).
- _exit_after_failed_branch_switch's branch_missing/checkout_move_failed probe cannot raise, so
the exit it names is unchanged even when git refuses.
- hermes update --plan/--check/--list-venv-holders on a managed install records no update run.
- Docs/contract say 'before the apply stage mark' and name the tokens that fire after git moved
and restored the tree; tests drive git_error_stop_class and _zip_stop_class directly.
* fix(telemetry): pre-receipt exits keep their initiator; index.lock permission errors; bare-interpreter updates park
- record_stop_without_receipt reads the update marker raw and tags initiator=desktop only when the
marker names this run's hand-off (delegate line = us, or HERMES_UPDATE_HANDOFF_PID / our parent),
so a Desktop-started early refusal reads kind=desktop and a CLI run refused by someone else's
update stays cli.
- _move_checkout_to classifies its git error with the shared classifier (git_output_stop_class):
index.lock 'File exists' -> git_index_locked, 'Permission denied'/EACCES -> permission_denied
(now a stop class), else checkout_move_failed.
- _publish_shared_metrics: an interpreter that cannot import the config reader (the -I -S bootstrap
that finalizes a dependency-preparation failure) never emits; it parks the bounded receipt in
pending_updates unless it can tell collection is off (loaded config says off, or no config.yaml),
and the next normal start applies the existing gate (report_pending_updates / begin_process purge).
* docs(telemetry): how update dashboards count a partial run
* test(desktop-update): a hand-off log read racing Add-Content retries, not fails
test_script_killed_before_publishing_the_delegate_runs_no_update failed on
the Windows arm64 lane with PermissionError reading
logs/desktop-update-handoff.log: windows.ps1 appends each line with
Add-Content, and a poll that lands while that write holds the file is a
Windows sharing violation, not a missing line. The poll helper now
retries the read (5 s budget) instead of failing the test on it.
Same test is green on main's latest arm64 run; nothing in this PR
touches the desktop-update scripts.
* test(e2e/desktop): give the build-fail updater its manual-outcome grace before asserting it exited
The build-fail spec polls 30 s for posix.sh to exit after the relaunched
app reports the outcome. That outcome is "manual" (the Desktop build is
owed), and finish() then runs launch_app (1.5 s acceptance) and
stop_ui leave-window, which sleeps the 15 s shim grace before tearing the
UI down: ~13 s of margin, and a slow Linux runner overran it (posix.sh
still alive at the 30 s mark, run 37619760207). The same code passed this
spec on bb177627828 and c4c2d9d8b54; the head that failed differs only by
a Windows test helper. 90 s keeps "no updater is left running" a real
check without racing the grace window.
* fix(cron): live-owner stale-claim reclaim measures silence, not run length
The #115692 reclaim released any running attempt whose claim was older than
max(3 x HERMES_CRON_TIMEOUT, script timeout, 2 h), even when the owner was
healthy and busy. Every cron job that legitimately ran longer than that was
marked `unknown` mid-run, its fire claim yanked from under it and the run
aborted with "lost its durable fire claim ownership" (two weekly jobs on one
install in a single day).
The run monitor now stamps `progress_at` on the execution row every 60 s
while the agent reports recent activity, and the sweep measures the stale
bound from that stamp (claim age for legacy rows). A wedged worker stops
stamping and is still reclaimed once the bound passes.
Shape: the stamper and the inactivity watchdog loop (which reads the same
idle signal) live in a new topical sibling cron/scheduler_liveness.py, so
cron/scheduler.py shrinks (4457 -> 4436 lines) instead of growing.
* fix(tools): normalize foreground terminal heartbeat
* fix(terminal): foreground heartbeat is dropped, not refused; notify intent still is
Follow-up to the cherry-picked #119201 (@KoNit-K): keep main's relocated
test file, flip the one invariant that pinned the refusal (a foreground
call with heartbeat>0 now runs with heartbeat=0 while notify=true stays
refused), reword the refusal so it no longer names heartbeat, and say in
the schema that heartbeat is ignored on foreground commands.
Why: the schema fix (4317ed0e71) only helps providers that materialize the
schema default. Models that copy a background call shape still send
heartbeat=60 on ordinary commands; this week 46 subagent sessions hit the
refusal 4 times each (173 refusals), 38 of them never ran a single shell
command, and the agents burned $316 / 1,498 model calls on retries.
* fix(desktop): setup cards stop redrawing in a loop
useSetupRows built a new row array on every render for the app-owned
lists (accent, layout, theme). That recreated the card's stage callback,
whose effect wrote $setupChooseStages, which re-rendered the card. The
rows are memoized on their inputs now, and stageSetupChoose skips the
atom write when a patch changes nothing.
* fix(onboarding): the first-task handoff follows a renamed owner profile
The setup marker saves the profile setup was made from. After that
profile was renamed, the saved name pointed at a missing directory, so
start_chat and reset targeted nothing. The owner now resolves through the
rename history (profile.yaml previous_names) that rename already records.
* fix(free-tier): a short free-tier 429 no longer blocks for a whole quota window
welcome_refusal_from_headers took the hourly bucket's reset as the wait
even when that bucket had requests left, so Retry-After: 4 with a
healthy bucket became a 2,000 s rate_limited refusal and tripped the
shared breaker. The wait now comes from an empty bucket's reset, else
Retry-After, else the body's wait.
* fix(desktop): a sign-in offer from the previous backend never opens
requestGateway keeps one identity across connection and profile
switches, so a free_tier.claim_nudge reply or status read that landed
after a switch could open the offer, or overwrite $freeTierStatus, for
the new backend. The offer timer, the claim and the finished-turn
re-read are tied to gatewayActivationEpoch() and drop a reply once the
route changes.
* fix(auth): persist profile single-use OAuth credentials when root store is empty
Fixes #103694: When a named profile runs auth add for a single-use
refresh provider (e.g. Anthropic, Codex, xAI) and the root store has
no existing rows for that provider, load_pool() sets _borrowed_root_ids = set().
Previously, add_entry() checked if borrowed_ids:, which evaluated to
False on an empty set. It fell back to _persist() -> persist_pool_entries(),
which routed to the update-only _update_root_pool_rows() on root. Because
root had no rows to update, the credential was dropped with no error and
never written to the profile's auth.json.
Checking if borrowed_ids is not None: ensures that profile-scoped pools
persist their new credential directly to the profile store via write_credential_pool(),
transitioning the profile to owning rows for that provider.
(cherry picked from commit b7467a2999832749f540d86530cf463ca0945b04)
(cherry picked from commit 9c604999e233cf6d74a83e4d7c141785b341b0d6)
* fix(auth): persist first profile OAuth row with empty root
(cherry picked from commit 5293489825af98e3e3f0e6cb485ffffa72d14833)
(cherry picked from commit 4144e14bfa7c021a92db5b0f66a75e2e0ac767d1)
* fix(auth): save a profile's first single-use login, and never print Added for an unsaved row
- Decide the profile claim when the row is written (the check from #101827)
and drop the None sentinel from #103712. On current main load_pool() leaves
a profile that borrows from an empty root at the default, so the sentinel
alone never reached the claim path (its test stays red).
- add_entry() re-reads the store after writing. A missing row raises
CredentialNotSavedError; `hermes auth add` exits with that message instead
of printing "Added", and the dashboard pool endpoint returns it as a 400.
- Tests: the two contributor tests become one invariant over every
SINGLE_USE_REFRESH_POOL_PROVIDERS provider (the row is readable, root
auth.json is byte-identical), plus the CLI success-line contract.
- Docs: profiles page, EN and zh-Hans.
Fixes #103694
(cherry picked from commit 0e250a005a3353564642ce70f100d89174a583f2)
* fix(auth): the forked-grant heal keeps a profile's own Anthropic login
`hermes -p <profile> auth add anthropic --type oauth` saves a
`manual:hermes_pkce` row in the profile. On the next load_pool(), the
forked-grant heal took every row whose source ends in `hermes_pkce` for a
copy of root's .anthropic_oauth.json whenever root's auth.json held no
Anthropic OAuth row. It moved the profile's token pair into root's file
and deleted the row from the profile: the CLI printed "Added", the login
was gone on the next turn, and root's file now held the profile's
account.
Only a `hermes_pkce` row comes from that file. `manual:hermes_pkce` is
owned by the pool and never written to the singleton
(_commit_anthropic_rotation matches the exact source for the same
reason), so the heal now matches the exact source too. The heal's own
docstring already promises to leave an independent `auth add` grant
alone.
This covers the empty-root case the previous commit enables, and main's
existing variant where root holds only an Anthropic API-key row next to
its .anthropic_oauth.json.
Co-authored-by: Enough <10966420+Enough1122@users.noreply.github.com>
* fix(auth): a profile's auth add never copies root's login into the profile
Root's login for nous, openai-codex and xai-oauth can live only in its
auth.json `providers.<id>` block, with no pool rows yet. In a profile
that borrows from root, load_pool() seeds it as a `device_code` entry
through the root fallback. The write-time claim in add_entry() kept
every entry except root's pool rows, so the profile's first `auth add`
wrote root's single-use refresh token into the profile's auth.json: a
forked grant (#100339) that the heal cannot match, because root has no
pool row for it.
- add_entry() writes only the added row when a profile claims a
single-use provider. Rows the profile seeds from its own sources are
seeded again on the next load.
- _seed_tokens_singleton() (openai-codex, xai-oauth) no longer seeds
root's providers block into a profile that owns its own rows, the
rule _seed_nous_singleton already applies. On current main such a
profile copied root's refresh token into its own pool on every load.
* fix(telemetry): update stage rows count the stage a failed run died in
hermes.update.run blamed 971 failed CLI runs in Sep 28 - Oct 5 on the apply
stage while hermes.update.stage held zero failed apply rows. Stage marks are
END marks: every pre-apply exit (fetch, channel, branch, merge, HEAD checks)
leaves no mark for the stage it died in, so the run row inferred the stage and
no stage row said it failed. A failed run whose failed_stage names a stage it
never marked now gets one failed row for it.
Rebuilt on the updater overhaul's followups model (#132386): a committed run
that owes follow-ups is a run-level success and its failed stage already has
its own mark, so only failed runs get the row; refusals never do.
* test: detached-writer custody tests read a pid from a beat that can never be empty
Mechanism: the detached build writer in the two "keeps custody until a
detached writer is gone" tests rewrote its heartbeat file in place with
pathlib.write_text, which opens with O_TRUNC and then writes. Custody
SIGKILLs that writer at an arbitrary instant; when the kill lands between
the truncating open and the write, `beat` is left empty and the teardown's
`beat.read_text().split()[0]` raises IndexError (suppress only covered
ProcessLookupError/ValueError). CI hit it on
test_a_ctrl_c_keeps_custody_until_a_detached_writer_is_gone[verbose-slow-cleanup]
(job 112764854735).
Repro: scratch copy of the writer with a 0.1 s sleep between open() and
write() -> both ctrl-c cases and the group-kill sibling fail with
IndexError at the teardown read on every run (5/5).
Fix: one shared _heartbeat_writer helper writes each beat to `<beat>.tmp`
and os.replace()s it into place, so `beat` always holds a complete
"<pid> <n>" once it exists. With the same injection moved onto the temp
write, all three cases pass 5/5. The ValueError suppress that papered over
a partial read is dropped.
* fix(tests): runner applies passthrough --ignore-glob; windows marker test stops sitting out 150s
Windows-only tests job 112764655375 (run 37612501107, a YAML-only catalog PR)
went red with "667 passed, 0 failed": test_desktop_update_windows_marker.py
hit the runner's 300s per-file cap mid-file ("(300s exceeded; process tree
SIGKILL'd)"), so its 19 tests were never counted and it was listed under
"no tests ran". Run 37612584210 died the same way. Clean runs of this file
take 220-300s (arm64 hit 296.5s and 299.8s), so it is at the cap on every run.
Two defects:
1. The desktop_updater lane gate never worked. tests-os.yml passes
--ignore-glob='*test_desktop_update_windows_*.py' after `--`, and
run_tests_parallel.py forwards it to `pytest <file>`. pytest applies
--ignore/--ignore-glob only while recursing directories, never to an
explicit file argument, so the flag did nothing. These hand-off tests ran
on every PR, including ones that never touched the updater, even though
the job printed "desktop_updater lane off: skipping ...". The runner now
applies --ignore/--ignore-glob to its own file list (pytest's fnmatch on
the absolute path, relative patterns anchored at the repo root, which is
the per-file pytest's cwd) and says how many files it excluded.
2. test_desktop_that_never_exits_is_not_relaunched_over waited out the
production 150s Desktop-exit ceiling. That one wait is half the file's
budget. The script already honours HERMES_UPDATE_DESKTOP_EXIT_SECONDS
("so the self-tests need not sit it out"), and the posix twin sets it to
2. Set it to 5 and assert that the script's refusal names 5s, so the
override is proven to reach the wait. The verdict under test (refuse
with 4, never relaunch) is the same at any ceiling.
Repro (Linux, CI-shape args):
python scripts/run_tests_parallel.py --files "<windows_marker>:<windows_cwd>:<other>" \
-- --ignore-glob='*test_desktop_update_windows_*.py' -m "platforms and not integration"
base: "Running 3 test files", both gated files run
fixed: "note: --ignore/--ignore-glob excluded 2 test files", "Running 1 test files"
New test_passthrough_ignore_drops_files_the_runner_hands_pytest_explicitly
(--files and discovery x glob/glob-spaced/path) fails 6/6 on base and
passes 6/6 fixed.
* fix(desktop): show a queued cron run row immediately after trigger
Run History only rendered sessions the backend had already materialized,
so an accepted Trigger now was invisible for the tens of seconds the
scheduler took to create the run session (#70826). The queued row is
painted from the click, settles when Run History observes a run that
started after it (any load — a fast 1s poll while queued, the cron.changed
broadcast on event-capable backends, or the visibility poll), and drops
at a bounded 90s timeout so a never-materializing run cannot wedge the
button. A failed request drops the row immediately. Pause deliberately
does not settle it: the pause response only persists enabled=false and
never cancels an already-claimed execution.
Queued design (queued-run row + polling + settle semantics) originates in
#70840 by @nv-cho; this ports it onto current main's
createCronTriggerController wiring.
Fixes #70826
* refactor(desktop-i18n): move billingBlock copy into en_billing sibling
Offset the cron queuedRun string growth in the over-cap en.ts/types.ts
facades per the code health ratchet: moved code keeps its cap.
* chore(desktop): refresh locales/_keys.desktop.json for cron.queuedRun
* fix(update): sweep stale git locks at the start of the run, not only before the fetch
The stale-lock / aborted-pack sweep ran only in the apply path, after the
snapshot. A run that died in that window left `.git/index.lock` behind with
nothing in the next run to remove it, so the following update failed with
"File exists" until an operator cleaned it up by hand. The sweep moves to the
top of `_cmd_update_impl`, ahead of the plan and the backup;
`clear_git_debris` already wraps locks and aborted-transfer pack temps (and
the partial-clone maintenance keys), and `_sweep_stale` skips everything
while a git process holds it, so the earlier position is safe.
(Salvaged from #132089 work; the receipt stop-reason half of the original
commit is now owned by the overhaul's receipt API and is not carried here.)
* fix(update): a killed git's index.lock no longer blocks the next update
`.git/index.lock` left by a killed git refused every later merge with
"File exists": the stale-lock sweep keeps any lock younger than 10 minutes,
so a `hermes update` killed mid-run (Windows crash cell `mid_fetch`, run
37573293632) or a version probe killed by its own 3 s timeout (#132089)
failed the next update at "Pulling updates...".
- `gitlock.release_dead_index_lock`: drop the lock as soon as the launch
repair's ownership proof (`_early_recovery._release_dead_index_lock`:
/proc fds + lock-keeping gits on Linux, lsof on macOS, the unlink itself on
Windows) shows no live git holds it. An interrupted tree move keeps its own
lock judgement. `_cmd_update_impl` calls it first, under the update lock.
- `version_info._git_version_info`: the read-only status probe runs as
`git --no-optional-locks status`, so a timed-out probe strands no lock.
Part of #132089 (the stale index.lock half; naming why a pre-apply exit fired is separate work)
* fix(update): on Windows a killed git's index.lock waits only for gits working in this checkout
Windows has no /proc to name a lock's holder, and its unlink-as-probe cannot see a
lock-keeping git with its fd closed (a `git commit` in the editor), so the reclaim keeps the
lock while a git.exe has its cwd or a path argument inside the checkout, and whenever the
scan cannot tell (no psutil, an uninspectable git). Any other git on the machine no longer
blocks it, unlike the age-floor sweep's machine-wide check.
* fix(update): only a killed update's index.lock is reclaimed at once
A lock nobody holds open is not proof its git is dead: a git that just created it may not have
opened it yet. The start-of-run reclaim now takes .git/index.lock only when the latest update run
died unfinished (receipt running/interrupted, owner gone) and the lock is newer than that run's
start. Every other lock is left to the age-floor sweep, and the update refuses on it truthfully
(e2e-upgrade test_live_index_lock_is_refused_truthfully went red on fd2214b0ad6).
* fix(update): the start-of-update index.lock reclaim never takes a live git's lock
Review round 2 on #133077. A killed update's receipt plus a younger lock let
release_dead_index_lock delete the lock of a live `/usr/lib/git-core/git-commit -a`
waiting in its editor: the lock fd is closed there, and the Linux holder scan only
knew `comm == git` with an argv subcommand, so the dashed form read as "no holder".
A competing `git add` then got in and the user's commit died "unable to write new
index file".
- The receipt gate stays necessary, never sufficient. The reclaim now also needs
proof that no git of ANY form works in this checkout (`_held_open(any_git=True)`):
cwd, path arguments (relative ones against the git's cwd) or GIT_DIR/GIT_WORK_TREE/
GIT_INDEX_FILE inside the checkout or its git dir, compared by path components.
A git we cannot place (unreadable cwd/environ, hidepid) keeps the lock.
- Git forms: `git`, a dashed `git-<sub>` from git-core, `git.exe`/`git-<sub>.exe`,
`-C`/`-c`/`--git-dir` globals, aliases (an unknown subcommand counts). The
launch-time repair's own scan inverts its list: only known readers are exempt.
- macOS: an empty lsof is no proof any more; `ps` lists every git (comm, so dashed
forms too) and any lock-keeping one keeps the lock.
- Windows: the psutil scan matches dashed and .exe gits, reads environ, and compares
by path components (`hermes-backup` was read as inside `hermes`).
- A failed retry no longer erases the evidence: `_killed_update_owns` reads the durable
per-run records (read-only) and asks about the last run that STARTED before the lock
was written, so a later failed run cannot discharge the killed one.
- The FIFO test helper kills its blocked git when its own setup assertion fails.
* fix(update): on macOS only a git working in this checkout keeps a dead index.lock
The lsof branch's ps scan counted every lock-keeping git on the machine,
because ps cannot say where a process works. A git commit open in any
other repository (or a sibling test worker's git on CI) therefore kept a
dead index.lock and blocked the launch-time repair after a killed update:
the macOS-only lane failed
test_a_merge_killed_writing_hermes_constants_is_repaired_by_the_next_launch
with "a running git holds the index".
Each candidate git's cwd now comes from `lsof -d cwd`, compared to the
checkout by path components (as on Linux); a candidate whose cwd cannot
be read while it still runs makes the answer unknowable, so the lock is
kept. Probe with real lsof/ps and a dashed git-commit waiting in its
editor: in another repo, 4b3c28c816c named it a holder and the fix does
not; in the checkout, both keep the lock.
* fix(update): the macOS lock scan finds ps when the launcher's PATH has none
The launch-time repair runs with whatever PATH the launcher got; the
macOS-only lane runs it with a PATH holding no git (as on the Windows
install that motivated the repair), so the bare "ps" call failed, the
scan answered "unknowable" and the dead index.lock of a killed update
was kept: test_a_merge_killed_writing_hermes_constants_is_repaired_by_the_next_launch
stayed red. ps is now resolved like lsof already is (which, then /bin/ps,
/usr/bin/ps). Probe: dead lock, PATH with neither git nor ps, at
ba2965261a0 the scan returned None (lock kept); now False (reclaimed).
* fix(update): review the macOS lock scan's lsof/ps lookups; keep it stdlib-only
The resolution-allowlist ratchet flagged the lsof lookup (moved from
_held_open to _held_open_lsof) and the new ps lookup in _ps_git_holder:
both are the stdlib-only launch-time repair asking the OS's own tools,
with an absent tool meaning "unknown" and the lock kept. Rows renamed
and added with that justification.
_still_running also imported psutil inside the launch-time repair, which
runs from hermes_bootstrap before app dependencies can import; it is now
a POSIX os.kill(pid, 0) probe (the lsof/ps branch never runs on Windows).
* fix(desktop): a setup card's typed-pick callback keeps a stable identity
The name card crashed the setup chat on a real Windows bundle (React #185,
maximum update depth): the card wrote its stage callback into
$setupChooseStages, which it also reads, and the callback depended on
rowLabels, a new object every render. 40512011b6 stopped the rows from
changing identity; this removes the dependency itself, so the callback
changes only with kind, mode, multi-select or the request. The labels
effect already stages the row names the callback used to rewrite.
The new test renders the real card: red on 17646902e1 (the crashing
build), green here and with either guard alone.
* test(ci): the CI replay names a silent step's exit code
workflow_steps.py is a helper module, not a test module, so pytest does not
rewrite its asserts: a replayed bash step that exits non-zero with empty
stdout/stderr fails as a bare 'AssertionError' with no exit code. Two
Windows arm64 os-tests runs (jobs 112635777217, 112764629268) failed this way
on unmodified baseline replays, and the NTSTATUS needed to tell a crashed
child (0xC0000142 / 0xC0000005) from a bash failure was lost. Put rc (decimal
and hex), the bash used, and both streams in the message.
* test(ci): the CI replay does not run Git Bash under x64 emulation on Windows arm64
Mechanism: Git for Windows on arm64 ships native git.exe but an x86-64 MSYS
usr/bin/bash.exe, which runs under the WoA x64 emulator. With the os-tests
lane's 16 parallel workers, a replayed step's bash dies before running a line
with 0xC000026F (STATUS_WX86_INTERNAL_ERROR) or 0xC0000005 and empty
stdout/stderr, so an unmodified baseline replay fails as a bare
AssertionError (jobs 112635777217 and 112764629268, two different nodeids).
Repro (throwaway workflows on the real runners, previous commit's diagnostics):
- windows-latest-32-arm-core, 16 copies of the file's replay tests at once,
4 rounds: 21/64 copies red, every failure 0xC000026F.
- Same load, windows-latest-32-core (x64): 0/64.
- Passing SYSTEMROOT/WINDIR or the full Windows env to the step: 12/48 and
17/48 vs 15/48 base, so the stripped env is not the cause.
- One process, 32 threads, 600 replays: 0 crashes; it takes concurrent
processes, i.e. host-wide emulator load.
Seam: _NATIVE_WINDOWS_TOO keeps platforms("linux", "windows") and adds a
skipif for win32 + native_arch() == arm64. The x64 Windows row still runs the
replay on the Windows layout (python.exe-only venv, Git Bash), Linux arm64
keeps it (native bash), and the routing guard that pins the marker passes.
* fix(free-tier): a setup chat turn does not start the sign-in offer clock
The setup chat's last turn (the fork answer and the handoff) finished clean
on the free tier, so it was recorded as the first task. The offer then came
due three minutes after setup ended, which lands in the middle of the first
real task when that task runs longer than three minutes. Seen live on a
Windows dev build: first_task_at was the setup chat's end, not the task's.
Turns in the setup profile's sessions are no longer tasks, as the offer
module already intended ("normally the task setup hands off to").
* i18n(desktop): translate cron.queuedRun in every overlay
Main added the Run History 'Queued run' label (44c1ad7f84) in English only;
this branch's overlay-completeness ratchet fails on a new untranslated key.
* Fix reasoning markup rendering in dashboard
* fix(web): scope structured reasoning UI to assistant messages
Only assistant content with known reasoning tags uses StructuredReasoning.
User and tool messages with literal <action>/<result> tags stay on Markdown.
* test(web): behaviour coverage for structured reasoning transcript rendering
RED on main: the expanded session transcript renders assistant
reasoning with literal <thinking>/<action>/<result> wrapper tags.
Keeps the user-role invariant (literal tags stay on the Markdown path).
* refactor(web): move session source config to SessionsPage_sources sibling; satisfy code-health ratchet
* fix(desktop): hide unknown cloud agent status
Omit the gateway status description when cloud discovery cannot provide a meaningful state, while preserving valid status labels. Add renderer coverage for both unknown and known states.
* test(desktop): cover empty cloud agent status
* fix(pm): reclaim dependency state of deleted checkouts (pm gc + startup worktree prune)
Every checkout that boots Hermes commits its own dependency state under
<home>/installs/<sha256(path)[:16]>/ (venv, test venv, PM runtime; ~200 MB
each) and nothing removed it when the checkout went away. On a host that
runs `hermes -w` campaigns the directory held 499 entries / 92 GB, of which
402 (80 GB) belonged to worktrees and scratch clones that no longer exist.
The state dir already records its checkout in inputs/.project-root. A new
pm/install_states.py lists entries whose recorded checkout is gone and
removes them unless something could still be reading them (install lock
taken, or a generation lease held). `pm gc` runs it and reports the count;
the startup worktree pruner runs it after its own reaping so a removed
worktree's state follows the tree instead of outliving it.
* fix(pm): orphan reclaim runs on every prune pass and keeps checkouts it cannot see
Follow-up to the review of this PR:
- The startup pruner returned early (no .worktrees/, or no tree past 24h) before the
reclaim ran, so a tree removed by hand kept its state until some other tree aged out.
_prune_stale_worktrees now runs the reclaim in a finally around the prune phases.
- A missing recorded path is not proof of deletion: a container sharing the data root
(-v ~/.hermes:/opt/data) cannot see the host's checkout, and gc there deleted the
host's live dependency environment. A state is an orphan only when its checkout sat
under the data root that owns installs/, or directly in a .worktrees/ dir that still
exists.
- PM runtime generation leases (pm-runtime/generations/*/.leases) now count as held,
so a worker still running from a deleted checkout's runtime keeps it.
* fix(pm): an interrupted orphan reclaim leaves a dir the next pass still finds
The startup reclaim runs on a daemon thread, so a hermes exit can cut a removal
short, and Windows refuses to unlink open files. shutil.rmtree takes children in
scandir order, so inputs/.project-root (the only record naming the checkout)
often went first and the remainder was invisible to every later pass.
Everything else is removed first; the record goes only once nothing else is left.
* fix(kanban): only sweep a parent workspace once the parent itself is terminal
The deferred parent sweep added for #33774 reaps a parent's scratch or
worktree workspace as soon as it has no active children left, without
checking the parent's own status (#133501). Archiving or finishing the
last active child of a parent that is still running/blocked/review
deletes the parent's live workspace from under its worker; the shared
workspace guard does not catch it because _OTHER_LIVE_PATHS_SQL excludes
the task being cleaned, i.e. the parent itself.
Gate the sweep on the parent being terminal (the same status set as
_ACTIVE_CHILDREN_SQL), so a live parent keeps its dir and reaps it at
its own terminal transition. #33774 semantics are unchanged: a parent
that finished while children still needed its handoff files is still
swept when the last child drains.
(cherry picked from commit 81f6588bbc5346097ab17e3660636c8a3be7319a)
* fix(mcp): reject fabricated DCR for providers without RFC 7591 registration (#78190)
Google's hosted Gmail/Drive MCP servers advertise no registration_endpoint
in their ASM, so the mcp SDK's create_client_registration_request falls
back to guessing POST {server_origin}/register. Those servers answer with
an opaque 404 (OAuthRegistrationError: Registration failed: 404) and the
gateway burns the reconnect ladder on an unrecoverable failure while the
CLI looks healthy because tools/list is unauthenticated on Google servers.
- Guard: HermesMCPOAuthProvider intercepts the fabricated registration
POST in the bidirectional auth_flow bridge and raises an actionable
OAuthRegistrationError (create an OAuth client, add oauth.client_id/
client_secret under mcp_servers.<name>.oauth in config.yaml) instead of
sending the request. Fires only when the URL is the exact SDK fallback
AND the ASM lacks a registration_endpoint, so servers advertising a
real RFC 7591 endpoint pass through untouched.
- Surfacing: needs_reauth tool errors now include the registration
guidance, and the generic 'MCP call failed' path unwraps TaskGroup
exception groups so the root cause reaches the model instead of
'unhandled errors in a TaskGroup (1 sub-exception)'.
- humanize_oauth_registration_error: 404 registration failures get the
same pre-registered-client guidance as 403 (CLI login path).
Tests: full discovery dance (401 -> PRM -> ASM without registration_endpoint
-> fabricated POST) drives the bridge and asserts the guard raises before
any network send; humanize 404 + passthrough cases; _auth_error_detail and
group unwrap surfacing.
Rebuilt on current main, where mcp_tool.py was decomposed and mcp 2.0
rides on httpx2: the surfacing helpers live in tools/mcp_tool_errors.py
and feed the shared _dispatch / _NEEDS_REAUTH_MSG paths in
tools/mcp_tool_handlers.py; the guard compares with the request's own URL
type and only fires once ASM was actually discovered (an undiscovered
ASM keeps main's discovery-context error, #113771), and closes the inner
SDK generator before raising so context.lock is released immediately.
* Report bounded npm probe and terminal GUI installer failures
* fix(ts): npm run fix
* fix(free-tier): HERMES_PREVIEW_FULL_CONNECTORS=1 turns the preview cohort on
The preview flag only read exactly true/false, so a bundle built with
=1 (HERMES_GUEST_ONBOARDING's spelling) silently sent {} and every guest
landed in the external cohort. 1/true now send true and 0/false send
false; anything else still omits the field.
* fix(desktop): explain missing remote profiles
* fix(desktop): lint missing-profile error and move its tests to a sibling file
Keeps remote-lifecycle.test.ts under its code-health line cap and clears
the curly/no-useless-escape errors from check:lint.
* fix(models): TokenHub's saved endpoint is not a relay, so /model keeps the curated Hy list
#134320 gave tencent-tokenhub a profile whose base_url is empty (its auth registry row owns
the endpoint). _configured_relay_base_url decides "relay or canonical" from the profile's
base_url only, so a model.base_url that `hermes model` saved as TokenHub's normal endpoint
read as a relay: the picke…
codeo1io
added a commit
to codeo1io/hermes-agent
that referenced
this pull request
Oct 11, 2026
…flicts resolved per fork doctrine) (#501) * plugin-catalog: hermes-field-notes 1.2.0 (patch recipes) * plugin-catalog: hermes-field-notes 1.2.1 (review fixes) * plugin-catalog: hermes-field-notes — README author section * chore(catalog): review disclosure for hermes-field-notes * chore: map alrcatraz for salvage of #132751 * fix(agent): let an explicit tool_reason label a soft interrupt interrupt() derived the published tool-interrupt reason from whether a message was attached, so on the soft path a system producer had no way to label itself: passing a message booked the stop as "user sent a new message", passing none booked it as "user interrupt". Both live in USER_INTERRUPT_REASONS, so interrupt_issuer() returned None and the turn exit reason became interrupted_by_user — a system abort recorded as a human stop. An explicit tool_reason now wins on the soft path as it already did for hard_cancel, keeping the message heuristic only as the fallback. (cherry picked from commit 495f1a84a15a4bf88295af1e06f225e25d57715c) * fix(agent): attribute the terminal batch-timeout abort to the guard The sequential-batch timeout guard stopped the turn with a bare interrupt(message), which the soft-path heuristic read as "user sent a new message". The abort was therefore recorded as interrupted_by_user and the skipped calls told the user they had sent a message they never wrote. Pass the guard's own tool_reason so the exit reason names the real issuer and the skipped-call notice can describe it. (cherry picked from commit a756f781b2630ea3059c8e56df132c5d8ba35844) * fix(agent): render skipped-call notices from the recorded reason Both sequential skip sites hardcoded a user-initiated cause ("User sent a new message" / "skipped due to user interrupt"), so any system abort that reached them asserted an action the user never took. Render the notice from the published reason instead: a system issuer describes itself, and only the genuinely user-initiated reasons keep the user-facing wording. (cherry picked from commit 8efe429e2a485a183c7ad46522d9a5c99cef6a9c) * fix(agent): hide the synthetic interrupt close from the transcript close_interrupted_tool_sequence() appends a synthetic assistant turn when an interrupted tail is a raw tool result, so the next user message does not land as tool -> user. The row carried a user-visible "Operation interrupted." string and was persisted as an ordinary assistant message, so background preemption — session reaping, watchdogs, batch guards — surfaced in the transcript as an interruption the user was expected to act on. Store it in the same hidden shape turn_api_call.py already uses for interrupt placeholders: content="" plus display_kind="hidden" keeps it out of rendered transcripts, while the api_content sidecar carries the LLM-visible text and is substituted at API-build time. The alternation guarantee is unchanged. (cherry picked from commit 22502b1c40c26a08eefc3d35cee067bd1d889bfc) * test(agent): cover soft-interrupt attribution and skipped-call wording The exit reason and the skipped-call notice must both name the real issuer: a system producer that stops the turn softly labels itself via tool_reason, while a message-carrying soft interrupt with no reason stays a human stop. The wording derives from the recorded reason, so a system abort never tells the user they did something they did not. (cherry picked from commit ec65b95cfb312ebb7a276a26b163df0eaad65b55) * fix(agent): render concurrent skip notices from the recorded reason The sequential path was converted to interrupt_skip_wording() but the concurrent batch kept three hardcoded user-blaming strings: the pre-batch skip (execute_tool_calls_concurrent), the unfilled-slot notice (_unfinished_tool_result) and emit_cancelled's hook message. A system producer that stops a parallel batch — watchdog, lease loss, SSE disconnect, or any segment routed through execute_tool_calls_segmented — still told the model "skipped due to user interrupt" while interrupt_issuer() booked the same turn as interrupted_by_system. Render all three from the recorded reason like the sequential siblings; the observability enums (error_type/hook_error_type) stay untouched. Also harden the interpolation the wording rides on: _append_skipped_tool_results substitutes {name} with str.replace instead of str.format, and interrupt_skip_wording escapes braces in caller-supplied reasons, so a free-form tool_reason can never raise KeyError/IndexError mid-turn. (cherry picked from commit b785bccbca6fd997921411198d277c39d3d93261) * fix(agent): hide only the placeholder close, keep caller banners visible close_interrupted_tool_sequence hid every synthetic closing turn behind display_kind="hidden", but four of its five callers pass a real user-facing notice — the truncation terminal ("Response truncated — …"), the context- overflow partial text, tool-validation partials and abort_turn_on_interrupt's "Operation interrupted: …" lines. Hiding those removed the turn's only explanation from every transcript surface. Narrow the hidden shape to what it was for: an empty final_response or the bare interrupt placeholder (which previously rendered as the "Operation interrupted." bubble users never asked for). Any other text is rendered verbatim as before. Alternation safety is unchanged in both branches. (cherry picked from commit a637af2ef44c1f0213c3138c458d6f6c48650988) * fix(agent): stop the interrupt placeholder echoing back as a reply A hidden assistant row is hidden from the transcript, not from the request: the projection substitutes its api_content sidecar into content on every call. The value chosen for that slot in #88955 was "[response interrupted]" - a short natural-language phrase sitting in the model's own prior-turn position, which models reproduce verbatim: an ordinary instruction answered with clean finish_reason=stop and nothing but the placeholder (#132949). The same hazard was already recognised twice in this codebase - the redirect scaffold is barred from assistant text because "the model echoes it" (#81841), and a runaway repetition partial is replaced by a self-describing label rather than replayed (#112764) - but the placeholder was never given the same treatment. Change the placeholder to a structural label the model has no reason to say, and retire rows already persisted under the pre-change spellings: the prep filter that dropped scaffold ghosts now also drops hidden rows carrying the legacy placeholder or the scaffold marker. Dropping is not always safe. A hidden close appended after a tool result is the only thing separating that tool message from the next user turn; removing it recreates the tool -> user alternation violation that makes strict providers ignore prior context (#48879) - the exact failure the close exists to prevent, and one repair_message_sequence deliberately leaves alone. When the neighbours would form that pair, rewrite the row's text instead of dropping it. The rewrite targets the current placeholder, so a neutralised row never re-triggers the filter. (cherry picked from commit 227e9a1ed64e0d75594d031423ecd2f5c74cd0b9) * test(agent): cover replay-echo retirement, and pin behaviour not wording The retirement filter gets five cases: a legacy placeholder row between two user turns is dropped; the same row after a tool result is kept with its text rewritten (dropping it would hand the provider the tool -> user pair that close_interrupted_tool_sequence exists to prevent); a row carrying the current placeholder is left untouched, which is what keeps #88955's re-heal guard intact and proves the rewrite cannot re-trigger itself; a scaffold ghost still drops as before; and a visible caller banner is never treated as a ghost. Verified discriminating: reverting the filter to the scaffold-only version fails all five, restoring it passes. The existing #88955 assertions compared against the literal "[response interrupted]" in four places. That couples the tests to one chosen spelling - any wording change, including this one, breaks them for the wrong reason. They now compare against the constant, asserting what the contract actually says: the durable row stays empty and hidden, the wire copy carries the neutral payload the writer stamped, and neither is ever the interrupt scaffold (#81841). (cherry picked from commit e4ebf0138b43547dbf1db36eaa84beb5c80bbfb7) * fix(agent): drop the diagnostic message from the batch-guard interrupt The terminal batch-timeout guard still passed "terminal batch tool did not complete" as the interrupt message. That text lands in _interrupt_message, which the gateway (_run_agent_drain_pending) and the CLI (_chat_resolve_interrupt) re-queue as the user's next turn, so a guard abort injected a fake user message. Call interrupt(tool_reason=...) with no message, and use a slug-safe reason so the issuer reads interrupted_by_system(terminal_batch_timeout) rather than terminal_batch_aborted:_call_timed_out. Message-less guard shape from #130209. Co-authored-by: Yuan Li <dskwelmcy@163.com> * test(agent): align skip-notice fixtures with the recorded-reason wording Skipped-call notices now render from _tool_interrupt_reason instead of a hardcoded "skipped due to user interrupt". The concurrent pre-flight stub never recorded a reason, so it fell to the "Turn interrupted" fallback and the stale assertion failed; record the new-message reason the real soft path publishes and assert its wording. The ACP fixture is cosmetic but should show the shape the executor actually emits. * fix(agent): retire replay-echo ghosts without mutating the transcript _neutralise_replay_echo_ghosts rewrote the caller's list in place (seq[:] = out) and overwrote content/api_content on durable row dicts. Those dicts are the live session rows the SessionDB flush cursor and _DB_PERSISTED_MARKER track, so an in-place edit silently diverges the in-memory row from what state.db holds. BASE built a new list here; keep that contract: return a filtered list and neutralise a shallow copy of the row, which only shapes this request's replay. * test(agent): trim the soft-interrupt salvage tests to two invariants Keep one attribution test that goes red on BASE (a tool_reason soft interrupt is booked to the system issuer with no requeued message) and one replay-echo test (a legacy placeholder row next to a tool tail is neutralised on a copy, never dropped into tool -> user, and the durable row is left untouched). The remaining wording/brace/close-row cases pin implementation detail already covered by the updated finalizer, steer and concurrent-interrupt tests. * fix(agent): render skip wording verbatim instead of brace-escaping it Fold finding 3 (B02 gate): interrupt_skip_wording doubled braces on the assumption the wording reaches str.format, but every consumer substitutes {name} with str.replace, so a reason like "a{b}" rendered as "a{{b}}". Drop the escape and its docstring claim, and look user-stop wording up directly in _USER_STOP_WORDING. * fix(agent): attribute Ctrl-C tool cancellation to a user interrupt Fold finding 2 (B02 gate, sibling of #130207): both KeyboardInterrupt handlers in tool_executor call agent.interrupt("keyboard interrupt"). With a message and no tool_reason, interrupt() records "user sent a new message", so the skipped-call notices for a Ctrl-C claimed the user had sent a new message. Pass tool_reason=_REASON_USER_INTERRUPT so the recorded reason (and the rendered wording) is "User interrupt"; the issuer stays None, so attribution to the human is unchanged. * fix(agent): neutralise legacy interrupt placeholders instead of dropping them Fold finding 1 (B02 gate, regression): the replay-echo filter now matches the legacy "[response interrupted]" placeholder and DROPPED the hidden row unless its neighbours were tool -> user. The common persisted redirect shape user -> hidden(placeholder) -> user(correction with checkpoint api_content) then became user -> user, and _merge_consecutive_users merged the pair: the correction's checkpoint sidecar was discarded and the first stored user row was rewritten in place. Always neutralise a copy ({content: "", api_content: placeholder}) and never drop: role-safe in every neighbour shape, and the echoable text is still gone. The kept replay test now also covers the redirect shape (3 rows survive, checkpoint intact, stored dicts untouched). * refactor(agent): tidy the interrupt-placeholder fold-ups Fold finding 4 (B02 gate, low): behaviour-neutral cleanups. - close_interrupted_tool_sequence: the hidden branch only runs when the stripped text is empty or already the placeholder, so `stripped or _INTERRUPTED_PLACEHOLDER` is always the placeholder; say so. - tool_executor: compute interrupt_skip_wording once for the cancelled outcome instead of twice. - move _neutralise_replay_echo_ghosts out of prepare_iteration to module level (lazy imports kept inside to avoid the conversation_loop cycle). - the finalizer test asserts the exact placeholder, not just non-empty. * test(agent): follow the renamed interrupt placeholder in the send-time pad tests Gate r2 finding 1 (High, regression): a4c687126c renamed the non-final empty-turn placeholder from "[response interrupted]" to "[interrupt: no assistant output for this turn]", but two send-time pad tests in test_partial_stream_finish_reason.py still asserted the old literal and went red on the stack head. Compare against agent.agent_runtime_helpers._INTERRUPTED_PLACEHOLDER so the tests track the constant, not a spelling. A full grep of tests/ and prod for the old wordings also found a healed prefill fixture in test_thinking_only_sanitizer.py (still green, but it models a stub the repair can no longer produce) and a stale docstring in test_steer.py; both now use the current placeholder. * test(agent): drive the real batch-timeout guard in the soft tool_reason test Gate r2 finding 2 (toothless): 482a748431 stopped the terminal batch-timeout guard from passing a diagnostic message to interrupt(), because _interrupt_message is what the gateway drain and the CLI re-queue as the user's next turn. The existing test only called agent.interrupt() directly, so reverting the guard hunk left it green. Route the existing test through _run_sequential_tool_execution_middleware with a prepared terminal call whose worker never settles, so the guard itself publishes the interrupt; the issuer and `_interrupt_message is None` asserts now go red when the guard hunk is reverted. * fix(agent): publish the Ctrl-C reason before the sequential cancel hook Gate r2 finding 3: in _run_sequential_call's KeyboardInterrupt handler, ref.emit_cancelled() ran before agent.interrupt(..., tool_reason=user interrupt), so the post_tool_call hook rendered its wording from an unset reason and reported "Tool execution cancelled. Turn interrupted" while the concurrent handler (and 47db3231e2's intent) report "User interrupt". Publish the interrupt first, matching the concurrent handler. The existing cancelled-hook test now pins the hook's error_message. * test(agent): pin the visible-banner branch of close_interrupted_tool_sequence Gate r2 finding 4 (toothless): f8aea0e81e keeps a caller-supplied banner ("Response truncated — …", partial-delivery text) visible and only hides the bare placeholder close, but nothing exercised the banner branch, so reverting it to the always-hidden shape stayed green. Extend the existing alternation test: a banner close must land as plain visible content with no display_kind. Red with f8aea0e81e's hunk reverted. * refactor(agent): match the replay-echo test docstring to the neutralise-only filter Gate r2 finding 5: d482ce366a made the prep filter neutralise legacy hidden rows on a copy instead of dropping them, but the module docstring still described retiring rows except where removal would form tool -> user. State the actual contract (never drop: removal could form tool -> user or a user -> user pair that repair merges) and drop _hidden_row's content / api_content kwargs that no caller passes. * refactor(agent): move interrupt placeholders into agent_runtime_helpers_placeholders to keep files under the size cap Code-health ratchet: agent/agent_runtime_helpers.py was 3787 > 3776 and tests/agent/test_run_agent.py 7014 > 7012 (both over the 2,000 target, so they may only shrink vs main). - _INTERRUPTED_PLACEHOLDER / _LEGACY_INTERRUPTED_PLACEHOLDER (and their rationale comment) move byte-identically into the leaf sibling agent/agent_runtime_helpers_placeholders.py. Every importer (runtime helpers via late import, conversation_loop, message_sanitization, turn_api_call, turn_iteration_prep and five test files) now reads the sibling; no shim is left in the facade. Nothing patches the old path. - test_keyboard_interrupt_emits_cancelled_post_tool_hook moves unchanged into tests/agent/test_run_agent_interrupt_hook.py, reusing the facade's agent fixture. * fix(agent): only relabel a prepared-batch abort as a batch timeout on timeout The batch-guard tail after _poll_sequential_future is shared by the "timeout" and "interrupted" outcomes. On a user stop (Ctrl-C, /stop, a new message) whose prepared terminal worker had not settled after the 3s grace, the unconditional agent.interrupt(tool_reason="terminal batch timeout") republished the interrupt: the turn was booked interrupted_by_system(terminal_batch_timeout), _interrupt_message (the user's queued next message) and _pending_redirect were nulled, and the wrong reason was re-fanned to workers. Keep batch.close() unconditional, but only call interrupt() on the timeout branch; on the interrupted branch the stop is already published. Public review blocker on #133271 (@ahrazzle, executed repro; @Enough1122). * refactor(agent): correct the stale placeholder-consistency comment The moved comment on _INTERRUPTED_PLACEHOLDER claimed it was "kept identical to the stub placeholder in chat_completion_helpers"; that module has no such literal (the claim was already stale on main and was moved verbatim). State what the constant actually is instead. Trivial note from the public review on #133271 (@ahrazzle). * fix(agent): stop terminal batch preparation from overwriting a user stop terminal_approval_batch caught both _CancelledPreparation and TimeoutError and called agent.interrupt(str(exc)). _CancelledPreparation is raised because a user interrupt is already pending, so the user's queued message and redirect were replaced by "Terminal approval preparation cancelled…", which the gateway then re-queued as a fake user turn. The TimeoutError branch booked a system timeout as a user stop with a fake requeue — the #130207 bug class itself, at the sibling site the batch guard fix missed. Now: close the batch always; only on timeout publish a message-less interrupt with tool_reason="terminal batch preparation timeout"; do nothing on cancellation (the stop is already published). Gate rP finding 1 (2c). * test(agent): pin system attribution in sequential skip notices The two _execute_tool_calls_sequential skip notices (stop before a call, and "remaining tool call(s)" after one) were rewritten to render the recorded interrupt reason (6eee1fb6b0), but reverting either back to the hardcoded "due to user interrupt" / "User sent a new message" left every kept test green. Extend the system-attribution test to drive both sites after a tool_reason stop; each reverted site now fails the test. Gate rP finding 2 (2ab). * test(agent): pin system attribution in unfinished concurrent slot results _unfinished_tool_result renders the recorded interrupt reason for a slot no worker filled (ec51b7d021), but hardcoding the old user wording back left every kept test green. Extend the system-attribution test to call it after a tool_reason stop and assert both the row and the post_tool_call error_message say "Turn aborted — …". Gate rP finding 3 (2ab). * test(agent): pin the Ctrl-C attribution on the concurrent tool path fed76c1df2 made both KeyboardInterrupt sites publish tool_reason="user interrupt", but only the sequential one was covered: reverting the concurrent worker's tool_reason left the suite green (the hook read "User sent a new message"). Parametrize the existing cancelled-hook test over the concurrent path with two calls. Gate rP finding 4 (2ab). * refactor(agent): derive user-stop skip wording from USER_INTERRUPT_REASONS _USER_STOP_WORDING restated the three USER_INTERRUPT_REASONS keys with their values capitalised, so a new human-stop reason had to be added in two places or it would render as "Turn aborted — …". Drop the mirror and capitalise the recorded reason; output is identical for all three keys (probe-B02-5.py). Gate rP finding 5 (reuse). * refactor(agent): share the hidden interrupt placeholder row message_sanitization.close_interrupted_tool_sequence and the turn_api_call repetition-loop exit each spelled the same hidden assistant row literal (content "", display_kind hidden, api_content _INTERRUPTED_PLACEHOLDER), and the conversation loop's redirect built it field by field. Add hidden_interrupt_placeholder_row() to the leaf placeholders module and use it at all three sites so the shape cannot drift between them. Gate rP finding 6 (reuse). * refactor(agent): import the interrupt placeholder at module level agent_runtime_helpers (two sites) and message_sanitization imported _INTERRUPTED_PLACEHOLDER inside the function body. The placeholders module is a leaf with no imports, so there is no cycle to dodge; import it once at module level (each module still imports cleanly in a fresh interpreter). conversation_loop keeps its local import. Gate rP finding 7 (quality). * docs(agent): say skipped-result content uses str.replace for {name} The _append_skipped_tool_results docstring still said content "is formatted with {name}", but the body substitutes with str.replace so a recorded interrupt reason containing braces renders verbatim. State that, so nobody "simplifies" it back to str.format. Gate rP finding 8 (quality). * test(agent): move the brace-rendering check after the attribution scenarios The verbatim "{x}" wording check sat between the batch-guard timeout and the user-stop scenario, mutating _tool_interrupt_reason mid-story. Move it to the end, alongside the other wording assertions. Gate rP finding 9 (quality). * refactor(agent): hoist the hidden placeholder row import and say why a timeout carries no message Gate rQ quality Lows: the placeholders module imports nothing, so conversation_loop can import it at module level like turn_api_call; the terminal-approval comment now says callers re-queue _interrupt_message, so a system stop must not set one. * fix(cron): fire an outage-skipped one-shot whose fire claim never landed Follow-up to #133283 (greptile review). The due scan stamps a one-shot's run_claim; when the later fire-claim save fails the run never starts. The next saved scan skips it on that live claim and ends the recovery window, so once the claim expires _retire_expired_oneshot saw a stale claim past grace and skipped it forever: neither fired nor retired. The scan now marks a run claim it stamps past grace (only an outage lets a one-shot through there) with outage=true; such a claim without a fire claim keeps the job covered, so it fires once the claim expires. A fire claim still means the run may have started, so at-most-once is unchanged. * fix(cron): report an unwritable tick lock in cron status and doctor Follow-up to #133283 (greptile review). When only cron/.tick.lock is unwritable (e.g. root-owned after a sudo run, in a writable cron dir) the tick degrades and returns 0, so the ticker records a success; probe_store checked only jobs.json, so `hermes cron status` said jobs would fire and `hermes doctor` reported a writable store while nothing ran. probe_store now treats an existing tick lock it cannot write the same way as a read-only jobs.json target: both status and doctor surface it with the usual fix hint. * fix(gateway): send cron store notices for a symlinked cron/ directory Follow-up to #133283 (greptile review). Store records are keyed by the resolved store path, but the notice sender rebuilt the owning profile from that path's parent. A profile whose cron/ is a symlink to another disk has a foreign parent, so neither its outage nor recovery notice reached any home channel; forget_homes had the same mismatch and never dropped such a departed profile's record. Both now match by resolving each served home's own cron/ path. * fix(cron): keep the outage marker when a skipped one-shot's dispatch fails clear_run_claim (the dispatch-failure path: interpreter shutdown, execution creation failure, pool.submit failure) set run_claim to None. The claiming scan had already ended the recovery window, and a restart drops the in-memory outage records, so the next scan saw a past-grace one-shot with no claim and retired it as missed: the job the outage skipped never fired. Keep {"outage": True} instead. With no "at" it is a stale claim, so the job is re-dispatched on the next tick, and _retire_expired_oneshot keeps it. At-most-once is unchanged: the fire_claim CAS still fences every re-dispatch. Gate r1 (PM12, 2c) Low finding. * fix(cron): drop a departed profile's store state by its resolved store path forget_homes rebuilt each store path from a departed home KEY (Path(key) / "cron"). cron/AGENTS.md forbids rebuilding a home from a key (hermes_home_key normcases), and the realpath only follows a symlinked cron/ while the home still exists: a profile whose home was deleted kept its degraded record, holding the host-wide hermes.cron.store.writable gauge at 0. register_ticked_homes now records each home's resolved store (new public store_health.store_key) while the home exists and hands the departed stores to forget_homes. gateway/cron_store_notices.py uses the same store_key against record.store (already resolved), so one normaliser decides "is this the served home's store" everywhere. Gate r1 (PM12, 2c + quality + 2ab toothless) Low finding. * test(cron): pin that the tick-lock probe names the lock file The tick-lock case only asserted probe_store() returned an error, which any other probe failure would also satisfy. Assert the error names .tick.lock so the case proves that branch fired. Gate r1 (PM12, quality) Low finding. * test(cron): derive the run-claim expiry step from the TTL helper timedelta(minutes=31) silently encoded ONESHOT_RUN_CLAIM_TTL_SECONDS = 1800; use _oneshot_run_claim_ttl_seconds() + 60 as tests/cron/test_jobs.py already does, so a TTL change cannot leave the step inside the live-claim window. Gate r1 (PM12, quality) Low finding. * fix(cron): keep store state another still-ticked profile shares through a symlink Two profiles whose cron/ resolves to one store share a store key; when one left, its departure dropped the degraded record the remaining profile still needed (gauge flipped to writable, a second outage notice, and the earlier outage start lost for one-shot cover). Departed stores now exclude the ones still ticked. The notices test pins it (red without the set difference). * refactor(cron): name forget_stores for what it takes; fix the clear_run_claim early-return comment * test(cron): pass encoding= to every text read/write in the store-outage tests * feat(telemetry): updates that stop before applying say why Every pre-apply exit of `hermes update` records one closed token on the receipt at the exit itself (update_receipt.record_stop_reason -> stop_class), and shared_metrics_update.update_failure_class maps it to the run's failure_class. aborted_before_apply stays only as the fallback for an exit with no recorded reason. Exits that fire before the receipt opens (update-lock refusal, Git operation in progress, managed install) write no receipt, as before, but now record a metrics-only row with the same closed class. The parked receipt copy keeps stop_class, exit_code, the stop reason's leading label and the restart/user-action flags, so a parked run classifies the same as the in-process one. A run that committed and is owed only an interrupt or the user's parked changes reads outcome partial (failure_class interrupted / local_changes_parked), never failed. * fixup: keep main.py and update_failure_class under their health caps * test(telemetry): pre-apply exit classes, parked parity, receipt-less refusals; docs + smoke - test_shared_metrics_update_stop_reasons: every stop token reads as its own class in-process and through the parked copy (no free text kept), unknown tokens and post-apply tokens fall back, a committed run that only owes parked changes is partial; the lock and Git-operation refusals each record one row and leave the holder's latest.json byte-identical. - test_shared_metrics_install_failures: a PermissionError stop reason now reads its own class permission_denied (was os_error); ENOSPC reads disk_full; other OSErrors keep os_error. - relay-shared-metrics.md: the new classes, partial outcome and parked fields. - relay smoke: the failed update receipt names its fetch failure and the row asserts fetch_failed. * fix(telemetry): review findings on pre-apply stop reasons - The parked copy keeps a stop reason's label only when it is an exception type name (plus its errno token) or one of Hermes' two fixed phrases; any other text before a colon reads '-' (a free phrase like 'my project name:' used to survive). - _exit_after_failed_branch_switch's branch_missing/checkout_move_failed probe cannot raise, so the exit it names is unchanged even when git refuses. - hermes update --plan/--check/--list-venv-holders on a managed install records no update run. - Docs/contract say 'before the apply stage mark' and name the tokens that fire after git moved and restored the tree; tests drive git_error_stop_class and _zip_stop_class directly. * fix(telemetry): pre-receipt exits keep their initiator; index.lock permission errors; bare-interpreter updates park - record_stop_without_receipt reads the update marker raw and tags initiator=desktop only when the marker names this run's hand-off (delegate line = us, or HERMES_UPDATE_HANDOFF_PID / our parent), so a Desktop-started early refusal reads kind=desktop and a CLI run refused by someone else's update stays cli. - _move_checkout_to classifies its git error with the shared classifier (git_output_stop_class): index.lock 'File exists' -> git_index_locked, 'Permission denied'/EACCES -> permission_denied (now a stop class), else checkout_move_failed. - _publish_shared_metrics: an interpreter that cannot import the config reader (the -I -S bootstrap that finalizes a dependency-preparation failure) never emits; it parks the bounded receipt in pending_updates unless it can tell collection is off (loaded config says off, or no config.yaml), and the next normal start applies the existing gate (report_pending_updates / begin_process purge). * docs(telemetry): how update dashboards count a partial run * test(desktop-update): a hand-off log read racing Add-Content retries, not fails test_script_killed_before_publishing_the_delegate_runs_no_update failed on the Windows arm64 lane with PermissionError reading logs/desktop-update-handoff.log: windows.ps1 appends each line with Add-Content, and a poll that lands while that write holds the file is a Windows sharing violation, not a missing line. The poll helper now retries the read (5 s budget) instead of failing the test on it. Same test is green on main's latest arm64 run; nothing in this PR touches the desktop-update scripts. * test(e2e/desktop): give the build-fail updater its manual-outcome grace before asserting it exited The build-fail spec polls 30 s for posix.sh to exit after the relaunched app reports the outcome. That outcome is "manual" (the Desktop build is owed), and finish() then runs launch_app (1.5 s acceptance) and stop_ui leave-window, which sleeps the 15 s shim grace before tearing the UI down: ~13 s of margin, and a slow Linux runner overran it (posix.sh still alive at the 30 s mark, run 37619760207). The same code passed this spec on bb177627828 and c4c2d9d8b54; the head that failed differs only by a Windows test helper. 90 s keeps "no updater is left running" a real check without racing the grace window. * fix(cron): live-owner stale-claim reclaim measures silence, not run length The #115692 reclaim released any running attempt whose claim was older than max(3 x HERMES_CRON_TIMEOUT, script timeout, 2 h), even when the owner was healthy and busy. Every cron job that legitimately ran longer than that was marked `unknown` mid-run, its fire claim yanked from under it and the run aborted with "lost its durable fire claim ownership" (two weekly jobs on one install in a single day). The run monitor now stamps `progress_at` on the execution row every 60 s while the agent reports recent activity, and the sweep measures the stale bound from that stamp (claim age for legacy rows). A wedged worker stops stamping and is still reclaimed once the bound passes. Shape: the stamper and the inactivity watchdog loop (which reads the same idle signal) live in a new topical sibling cron/scheduler_liveness.py, so cron/scheduler.py shrinks (4457 -> 4436 lines) instead of growing. * fix(tools): normalize foreground terminal heartbeat * fix(terminal): foreground heartbeat is dropped, not refused; notify intent still is Follow-up to the cherry-picked #119201 (@KoNit-K): keep main's relocated test file, flip the one invariant that pinned the refusal (a foreground call with heartbeat>0 now runs with heartbeat=0 while notify=true stays refused), reword the refusal so it no longer names heartbeat, and say in the schema that heartbeat is ignored on foreground commands. Why: the schema fix (4317ed0e71) only helps providers that materialize the schema default. Models that copy a background call shape still send heartbeat=60 on ordinary commands; this week 46 subagent sessions hit the refusal 4 times each (173 refusals), 38 of them never ran a single shell command, and the agents burned $316 / 1,498 model calls on retries. * fix(desktop): setup cards stop redrawing in a loop useSetupRows built a new row array on every render for the app-owned lists (accent, layout, theme). That recreated the card's stage callback, whose effect wrote $setupChooseStages, which re-rendered the card. The rows are memoized on their inputs now, and stageSetupChoose skips the atom write when a patch changes nothing. * fix(onboarding): the first-task handoff follows a renamed owner profile The setup marker saves the profile setup was made from. After that profile was renamed, the saved name pointed at a missing directory, so start_chat and reset targeted nothing. The owner now resolves through the rename history (profile.yaml previous_names) that rename already records. * fix(free-tier): a short free-tier 429 no longer blocks for a whole quota window welcome_refusal_from_headers took the hourly bucket's reset as the wait even when that bucket had requests left, so Retry-After: 4 with a healthy bucket became a 2,000 s rate_limited refusal and tripped the shared breaker. The wait now comes from an empty bucket's reset, else Retry-After, else the body's wait. * fix(desktop): a sign-in offer from the previous backend never opens requestGateway keeps one identity across connection and profile switches, so a free_tier.claim_nudge reply or status read that landed after a switch could open the offer, or overwrite $freeTierStatus, for the new backend. The offer timer, the claim and the finished-turn re-read are tied to gatewayActivationEpoch() and drop a reply once the route changes. * fix(auth): persist profile single-use OAuth credentials when root store is empty Fixes #103694: When a named profile runs auth add for a single-use refresh provider (e.g. Anthropic, Codex, xAI) and the root store has no existing rows for that provider, load_pool() sets _borrowed_root_ids = set(). Previously, add_entry() checked if borrowed_ids:, which evaluated to False on an empty set. It fell back to _persist() -> persist_pool_entries(), which routed to the update-only _update_root_pool_rows() on root. Because root had no rows to update, the credential was dropped with no error and never written to the profile's auth.json. Checking if borrowed_ids is not None: ensures that profile-scoped pools persist their new credential directly to the profile store via write_credential_pool(), transitioning the profile to owning rows for that provider. (cherry picked from commit b7467a2999832749f540d86530cf463ca0945b04) (cherry picked from commit 9c604999e233cf6d74a83e4d7c141785b341b0d6) * fix(auth): persist first profile OAuth row with empty root (cherry picked from commit 5293489825af98e3e3f0e6cb485ffffa72d14833) (cherry picked from commit 4144e14bfa7c021a92db5b0f66a75e2e0ac767d1) * fix(auth): save a profile's first single-use login, and never print Added for an unsaved row - Decide the profile claim when the row is written (the check from #101827) and drop the None sentinel from #103712. On current main load_pool() leaves a profile that borrows from an empty root at the default, so the sentinel alone never reached the claim path (its test stays red). - add_entry() re-reads the store after writing. A missing row raises CredentialNotSavedError; `hermes auth add` exits with that message instead of printing "Added", and the dashboard pool endpoint returns it as a 400. - Tests: the two contributor tests become one invariant over every SINGLE_USE_REFRESH_POOL_PROVIDERS provider (the row is readable, root auth.json is byte-identical), plus the CLI success-line contract. - Docs: profiles page, EN and zh-Hans. Fixes #103694 (cherry picked from commit 0e250a005a3353564642ce70f100d89174a583f2) * fix(auth): the forked-grant heal keeps a profile's own Anthropic login `hermes -p <profile> auth add anthropic --type oauth` saves a `manual:hermes_pkce` row in the profile. On the next load_pool(), the forked-grant heal took every row whose source ends in `hermes_pkce` for a copy of root's .anthropic_oauth.json whenever root's auth.json held no Anthropic OAuth row. It moved the profile's token pair into root's file and deleted the row from the profile: the CLI printed "Added", the login was gone on the next turn, and root's file now held the profile's account. Only a `hermes_pkce` row comes from that file. `manual:hermes_pkce` is owned by the pool and never written to the singleton (_commit_anthropic_rotation matches the exact source for the same reason), so the heal now matches the exact source too. The heal's own docstring already promises to leave an independent `auth add` grant alone. This covers the empty-root case the previous commit enables, and main's existing variant where root holds only an Anthropic API-key row next to its .anthropic_oauth.json. Co-authored-by: Enough <10966420+Enough1122@users.noreply.github.com> * fix(auth): a profile's auth add never copies root's login into the profile Root's login for nous, openai-codex and xai-oauth can live only in its auth.json `providers.<id>` block, with no pool rows yet. In a profile that borrows from root, load_pool() seeds it as a `device_code` entry through the root fallback. The write-time claim in add_entry() kept every entry except root's pool rows, so the profile's first `auth add` wrote root's single-use refresh token into the profile's auth.json: a forked grant (#100339) that the heal cannot match, because root has no pool row for it. - add_entry() writes only the added row when a profile claims a single-use provider. Rows the profile seeds from its own sources are seeded again on the next load. - _seed_tokens_singleton() (openai-codex, xai-oauth) no longer seeds root's providers block into a profile that owns its own rows, the rule _seed_nous_singleton already applies. On current main such a profile copied root's refresh token into its own pool on every load. * fix(telemetry): update stage rows count the stage a failed run died in hermes.update.run blamed 971 failed CLI runs in Sep 28 - Oct 5 on the apply stage while hermes.update.stage held zero failed apply rows. Stage marks are END marks: every pre-apply exit (fetch, channel, branch, merge, HEAD checks) leaves no mark for the stage it died in, so the run row inferred the stage and no stage row said it failed. A failed run whose failed_stage names a stage it never marked now gets one failed row for it. Rebuilt on the updater overhaul's followups model (#132386): a committed run that owes follow-ups is a run-level success and its failed stage already has its own mark, so only failed runs get the row; refusals never do. * test: detached-writer custody tests read a pid from a beat that can never be empty Mechanism: the detached build writer in the two "keeps custody until a detached writer is gone" tests rewrote its heartbeat file in place with pathlib.write_text, which opens with O_TRUNC and then writes. Custody SIGKILLs that writer at an arbitrary instant; when the kill lands between the truncating open and the write, `beat` is left empty and the teardown's `beat.read_text().split()[0]` raises IndexError (suppress only covered ProcessLookupError/ValueError). CI hit it on test_a_ctrl_c_keeps_custody_until_a_detached_writer_is_gone[verbose-slow-cleanup] (job 112764854735). Repro: scratch copy of the writer with a 0.1 s sleep between open() and write() -> both ctrl-c cases and the group-kill sibling fail with IndexError at the teardown read on every run (5/5). Fix: one shared _heartbeat_writer helper writes each beat to `<beat>.tmp` and os.replace()s it into place, so `beat` always holds a complete "<pid> <n>" once it exists. With the same injection moved onto the temp write, all three cases pass 5/5. The ValueError suppress that papered over a partial read is dropped. * fix(tests): runner applies passthrough --ignore-glob; windows marker test stops sitting out 150s Windows-only tests job 112764655375 (run 37612501107, a YAML-only catalog PR) went red with "667 passed, 0 failed": test_desktop_update_windows_marker.py hit the runner's 300s per-file cap mid-file ("(300s exceeded; process tree SIGKILL'd)"), so its 19 tests were never counted and it was listed under "no tests ran". Run 37612584210 died the same way. Clean runs of this file take 220-300s (arm64 hit 296.5s and 299.8s), so it is at the cap on every run. Two defects: 1. The desktop_updater lane gate never worked. tests-os.yml passes --ignore-glob='*test_desktop_update_windows_*.py' after `--`, and run_tests_parallel.py forwards it to `pytest <file>`. pytest applies --ignore/--ignore-glob only while recursing directories, never to an explicit file argument, so the flag did nothing. These hand-off tests ran on every PR, including ones that never touched the updater, even though the job printed "desktop_updater lane off: skipping ...". The runner now applies --ignore/--ignore-glob to its own file list (pytest's fnmatch on the absolute path, relative patterns anchored at the repo root, which is the per-file pytest's cwd) and says how many files it excluded. 2. test_desktop_that_never_exits_is_not_relaunched_over waited out the production 150s Desktop-exit ceiling. That one wait is half the file's budget. The script already honours HERMES_UPDATE_DESKTOP_EXIT_SECONDS ("so the self-tests need not sit it out"), and the posix twin sets it to 2. Set it to 5 and assert that the script's refusal names 5s, so the override is proven to reach the wait. The verdict under test (refuse with 4, never relaunch) is the same at any ceiling. Repro (Linux, CI-shape args): python scripts/run_tests_parallel.py --files "<windows_marker>:<windows_cwd>:<other>" \ -- --ignore-glob='*test_desktop_update_windows_*.py' -m "platforms and not integration" base: "Running 3 test files", both gated files run fixed: "note: --ignore/--ignore-glob excluded 2 test files", "Running 1 test files" New test_passthrough_ignore_drops_files_the_runner_hands_pytest_explicitly (--files and discovery x glob/glob-spaced/path) fails 6/6 on base and passes 6/6 fixed. * fix(desktop): show a queued cron run row immediately after trigger Run History only rendered sessions the backend had already materialized, so an accepted Trigger now was invisible for the tens of seconds the scheduler took to create the run session (#70826). The queued row is painted from the click, settles when Run History observes a run that started after it (any load — a fast 1s poll while queued, the cron.changed broadcast on event-capable backends, or the visibility poll), and drops at a bounded 90s timeout so a never-materializing run cannot wedge the button. A failed request drops the row immediately. Pause deliberately does not settle it: the pause response only persists enabled=false and never cancels an already-claimed execution. Queued design (queued-run row + polling + settle semantics) originates in #70840 by @nv-cho; this ports it onto current main's createCronTriggerController wiring. Fixes #70826 * refactor(desktop-i18n): move billingBlock copy into en_billing sibling Offset the cron queuedRun string growth in the over-cap en.ts/types.ts facades per the code health ratchet: moved code keeps its cap. * chore(desktop): refresh locales/_keys.desktop.json for cron.queuedRun * fix(update): sweep stale git locks at the start of the run, not only before the fetch The stale-lock / aborted-pack sweep ran only in the apply path, after the snapshot. A run that died in that window left `.git/index.lock` behind with nothing in the next run to remove it, so the following update failed with "File exists" until an operator cleaned it up by hand. The sweep moves to the top of `_cmd_update_impl`, ahead of the plan and the backup; `clear_git_debris` already wraps locks and aborted-transfer pack temps (and the partial-clone maintenance keys), and `_sweep_stale` skips everything while a git process holds it, so the earlier position is safe. (Salvaged from #132089 work; the receipt stop-reason half of the original commit is now owned by the overhaul's receipt API and is not carried here.) * fix(update): a killed git's index.lock no longer blocks the next update `.git/index.lock` left by a killed git refused every later merge with "File exists": the stale-lock sweep keeps any lock younger than 10 minutes, so a `hermes update` killed mid-run (Windows crash cell `mid_fetch`, run 37573293632) or a version probe killed by its own 3 s timeout (#132089) failed the next update at "Pulling updates...". - `gitlock.release_dead_index_lock`: drop the lock as soon as the launch repair's ownership proof (`_early_recovery._release_dead_index_lock`: /proc fds + lock-keeping gits on Linux, lsof on macOS, the unlink itself on Windows) shows no live git holds it. An interrupted tree move keeps its own lock judgement. `_cmd_update_impl` calls it first, under the update lock. - `version_info._git_version_info`: the read-only status probe runs as `git --no-optional-locks status`, so a timed-out probe strands no lock. Part of #132089 (the stale index.lock half; naming why a pre-apply exit fired is separate work) * fix(update): on Windows a killed git's index.lock waits only for gits working in this checkout Windows has no /proc to name a lock's holder, and its unlink-as-probe cannot see a lock-keeping git with its fd closed (a `git commit` in the editor), so the reclaim keeps the lock while a git.exe has its cwd or a path argument inside the checkout, and whenever the scan cannot tell (no psutil, an uninspectable git). Any other git on the machine no longer blocks it, unlike the age-floor sweep's machine-wide check. * fix(update): only a killed update's index.lock is reclaimed at once A lock nobody holds open is not proof its git is dead: a git that just created it may not have opened it yet. The start-of-run reclaim now takes .git/index.lock only when the latest update run died unfinished (receipt running/interrupted, owner gone) and the lock is newer than that run's start. Every other lock is left to the age-floor sweep, and the update refuses on it truthfully (e2e-upgrade test_live_index_lock_is_refused_truthfully went red on fd2214b0ad6). * fix(update): the start-of-update index.lock reclaim never takes a live git's lock Review round 2 on #133077. A killed update's receipt plus a younger lock let release_dead_index_lock delete the lock of a live `/usr/lib/git-core/git-commit -a` waiting in its editor: the lock fd is closed there, and the Linux holder scan only knew `comm == git` with an argv subcommand, so the dashed form read as "no holder". A competing `git add` then got in and the user's commit died "unable to write new index file". - The receipt gate stays necessary, never sufficient. The reclaim now also needs proof that no git of ANY form works in this checkout (`_held_open(any_git=True)`): cwd, path arguments (relative ones against the git's cwd) or GIT_DIR/GIT_WORK_TREE/ GIT_INDEX_FILE inside the checkout or its git dir, compared by path components. A git we cannot place (unreadable cwd/environ, hidepid) keeps the lock. - Git forms: `git`, a dashed `git-<sub>` from git-core, `git.exe`/`git-<sub>.exe`, `-C`/`-c`/`--git-dir` globals, aliases (an unknown subcommand counts). The launch-time repair's own scan inverts its list: only known readers are exempt. - macOS: an empty lsof is no proof any more; `ps` lists every git (comm, so dashed forms too) and any lock-keeping one keeps the lock. - Windows: the psutil scan matches dashed and .exe gits, reads environ, and compares by path components (`hermes-backup` was read as inside `hermes`). - A failed retry no longer erases the evidence: `_killed_update_owns` reads the durable per-run records (read-only) and asks about the last run that STARTED before the lock was written, so a later failed run cannot discharge the killed one. - The FIFO test helper kills its blocked git when its own setup assertion fails. * fix(update): on macOS only a git working in this checkout keeps a dead index.lock The lsof branch's ps scan counted every lock-keeping git on the machine, because ps cannot say where a process works. A git commit open in any other repository (or a sibling test worker's git on CI) therefore kept a dead index.lock and blocked the launch-time repair after a killed update: the macOS-only lane failed test_a_merge_killed_writing_hermes_constants_is_repaired_by_the_next_launch with "a running git holds the index". Each candidate git's cwd now comes from `lsof -d cwd`, compared to the checkout by path components (as on Linux); a candidate whose cwd cannot be read while it still runs makes the answer unknowable, so the lock is kept. Probe with real lsof/ps and a dashed git-commit waiting in its editor: in another repo, 4b3c28c816c named it a holder and the fix does not; in the checkout, both keep the lock. * fix(update): the macOS lock scan finds ps when the launcher's PATH has none The launch-time repair runs with whatever PATH the launcher got; the macOS-only lane runs it with a PATH holding no git (as on the Windows install that motivated the repair), so the bare "ps" call failed, the scan answered "unknowable" and the dead index.lock of a killed update was kept: test_a_merge_killed_writing_hermes_constants_is_repaired_by_the_next_launch stayed red. ps is now resolved like lsof already is (which, then /bin/ps, /usr/bin/ps). Probe: dead lock, PATH with neither git nor ps, at ba2965261a0 the scan returned None (lock kept); now False (reclaimed). * fix(update): review the macOS lock scan's lsof/ps lookups; keep it stdlib-only The resolution-allowlist ratchet flagged the lsof lookup (moved from _held_open to _held_open_lsof) and the new ps lookup in _ps_git_holder: both are the stdlib-only launch-time repair asking the OS's own tools, with an absent tool meaning "unknown" and the lock kept. Rows renamed and added with that justification. _still_running also imported psutil inside the launch-time repair, which runs from hermes_bootstrap before app dependencies can import; it is now a POSIX os.kill(pid, 0) probe (the lsof/ps branch never runs on Windows). * fix(desktop): a setup card's typed-pick callback keeps a stable identity The name card crashed the setup chat on a real Windows bundle (React #185, maximum update depth): the card wrote its stage callback into $setupChooseStages, which it also reads, and the callback depended on rowLabels, a new object every render. 40512011b6 stopped the rows from changing identity; this removes the dependency itself, so the callback changes only with kind, mode, multi-select or the request. The labels effect already stages the row names the callback used to rewrite. The new test renders the real card: red on 17646902e1 (the crashing build), green here and with either guard alone. * test(ci): the CI replay names a silent step's exit code workflow_steps.py is a helper module, not a test module, so pytest does not rewrite its asserts: a replayed bash step that exits non-zero with empty stdout/stderr fails as a bare 'AssertionError' with no exit code. Two Windows arm64 os-tests runs (jobs 112635777217, 112764629268) failed this way on unmodified baseline replays, and the NTSTATUS needed to tell a crashed child (0xC0000142 / 0xC0000005) from a bash failure was lost. Put rc (decimal and hex), the bash used, and both streams in the message. * test(ci): the CI replay does not run Git Bash under x64 emulation on Windows arm64 Mechanism: Git for Windows on arm64 ships native git.exe but an x86-64 MSYS usr/bin/bash.exe, which runs under the WoA x64 emulator. With the os-tests lane's 16 parallel workers, a replayed step's bash dies before running a line with 0xC000026F (STATUS_WX86_INTERNAL_ERROR) or 0xC0000005 and empty stdout/stderr, so an unmodified baseline replay fails as a bare AssertionError (jobs 112635777217 and 112764629268, two different nodeids). Repro (throwaway workflows on the real runners, previous commit's diagnostics): - windows-latest-32-arm-core, 16 copies of the file's replay tests at once, 4 rounds: 21/64 copies red, every failure 0xC000026F. - Same load, windows-latest-32-core (x64): 0/64. - Passing SYSTEMROOT/WINDIR or the full Windows env to the step: 12/48 and 17/48 vs 15/48 base, so the stripped env is not the cause. - One process, 32 threads, 600 replays: 0 crashes; it takes concurrent processes, i.e. host-wide emulator load. Seam: _NATIVE_WINDOWS_TOO keeps platforms("linux", "windows") and adds a skipif for win32 + native_arch() == arm64. The x64 Windows row still runs the replay on the Windows layout (python.exe-only venv, Git Bash), Linux arm64 keeps it (native bash), and the routing guard that pins the marker passes. * fix(free-tier): a setup chat turn does not start the sign-in offer clock The setup chat's last turn (the fork answer and the handoff) finished clean on the free tier, so it was recorded as the first task. The offer then came due three minutes after setup ended, which lands in the middle of the first real task when that task runs longer than three minutes. Seen live on a Windows dev build: first_task_at was the setup chat's end, not the task's. Turns in the setup profile's sessions are no longer tasks, as the offer module already intended ("normally the task setup hands off to"). * i18n(desktop): translate cron.queuedRun in every overlay Main added the Run History 'Queued run' label (44c1ad7f84) in English only; this branch's overlay-completeness ratchet fails on a new untranslated key. * Fix reasoning markup rendering in dashboard * fix(web): scope structured reasoning UI to assistant messages Only assistant content with known reasoning tags uses StructuredReasoning. User and tool messages with literal <action>/<result> tags stay on Markdown. * test(web): behaviour coverage for structured reasoning transcript rendering RED on main: the expanded session transcript renders assistant reasoning with literal <thinking>/<action>/<result> wrapper tags. Keeps the user-role invariant (literal tags stay on the Markdown path). * refactor(web): move session source config to SessionsPage_sources sibling; satisfy code-health ratchet * fix(desktop): hide unknown cloud agent status Omit the gateway status description when cloud discovery cannot provide a meaningful state, while preserving valid status labels. Add renderer coverage for both unknown and known states. * test(desktop): cover empty cloud agent status * fix(pm): reclaim dependency state of deleted checkouts (pm gc + startup worktree prune) Every checkout that boots Hermes commits its own dependency state under <home>/installs/<sha256(path)[:16]>/ (venv, test venv, PM runtime; ~200 MB each) and nothing removed it when the checkout went away. On a host that runs `hermes -w` campaigns the directory held 499 entries / 92 GB, of which 402 (80 GB) belonged to worktrees and scratch clones that no longer exist. The state dir already records its checkout in inputs/.project-root. A new pm/install_states.py lists entries whose recorded checkout is gone and removes them unless something could still be reading them (install lock taken, or a generation lease held). `pm gc` runs it and reports the count; the startup worktree pruner runs it after its own reaping so a removed worktree's state follows the tree instead of outliving it. * fix(pm): orphan reclaim runs on every prune pass and keeps checkouts it cannot see Follow-up to the review of this PR: - The startup pruner returned early (no .worktrees/, or no tree past 24h) before the reclaim ran, so a tree removed by hand kept its state until some other tree aged out. _prune_stale_worktrees now runs the reclaim in a finally around the prune phases. - A missing recorded path is not proof of deletion: a container sharing the data root (-v ~/.hermes:/opt/data) cannot see the host's checkout, and gc there deleted the host's live dependency environment. A state is an orphan only when its checkout sat under the data root that owns installs/, or directly in a .worktrees/ dir that still exists. - PM runtime generation leases (pm-runtime/generations/*/.leases) now count as held, so a worker still running from a deleted checkout's runtime keeps it. * fix(pm): an interrupted orphan reclaim leaves a dir the next pass still finds The startup reclaim runs on a daemon thread, so a hermes exit can cut a removal short, and Windows refuses to unlink open files. shutil.rmtree takes children in scandir order, so inputs/.project-root (the only record naming the checkout) often went first and the remainder was invisible to every later pass. Everything else is removed first; the record goes only once nothing else is left. * fix(kanban): only sweep a parent workspace once the parent itself is terminal The deferred parent sweep added for #33774 reaps a parent's scratch or worktree workspace as soon as it has no active children left, without checking the parent's own status (#133501). Archiving or finishing the last active child of a parent that is still running/blocked/review deletes the parent's live workspace from under its worker; the shared workspace guard does not catch it because _OTHER_LIVE_PATHS_SQL excludes the task being cleaned, i.e. the parent itself. Gate the sweep on the parent being terminal (the same status set as _ACTIVE_CHILDREN_SQL), so a live parent keeps its dir and reaps it at its own terminal transition. #33774 semantics are unchanged: a parent that finished while children still needed its handoff files is still swept when the last child drains. (cherry picked from commit 81f6588bbc5346097ab17e3660636c8a3be7319a) * fix(mcp): reject fabricated DCR for providers without RFC 7591 registration (#78190) Google's hosted Gmail/Drive MCP servers advertise no registration_endpoint in their ASM, so the mcp SDK's create_client_registration_request falls back to guessing POST {server_origin}/register. Those servers answer with an opaque 404 (OAuthRegistrationError: Registration failed: 404) and the gateway burns the reconnect ladder on an unrecoverable failure while the CLI looks healthy because tools/list is unauthenticated on Google servers. - Guard: HermesMCPOAuthProvider intercepts the fabricated registration POST in the bidirectional auth_flow bridge and raises an actionable OAuthRegistrationError (create an OAuth client, add oauth.client_id/ client_secret under mcp_servers.<name>.oauth in config.yaml) instead of sending the request. Fires only when the URL is the exact SDK fallback AND the ASM lacks a registration_endpoint, so servers advertising a real RFC 7591 endpoint pass through untouched. - Surfacing: needs_reauth tool errors now include the registration guidance, and the generic 'MCP call failed' path unwraps TaskGroup exception groups so the root cause reaches the model instead of 'unhandled errors in a TaskGroup (1 sub-exception)'. - humanize_oauth_registration_error: 404 registration failures get the same pre-registered-client guidance as 403 (CLI login path). Tests: full discovery dance (401 -> PRM -> ASM without registration_endpoint -> fabricated POST) drives the bridge and asserts the guard raises before any network send; humanize 404 + passthrough cases; _auth_error_detail and group unwrap surfacing. Rebuilt on current main, where mcp_tool.py was decomposed and mcp 2.0 rides on httpx2: the surfacing helpers live in tools/mcp_tool_errors.py and feed the shared _dispatch / _NEEDS_REAUTH_MSG paths in tools/mcp_tool_handlers.py; the guard compares with the request's own URL type and only fires once ASM was actually discovered (an undiscovered ASM keeps main's discovery-context error, #113771), and closes the inner SDK generator before raising so context.lock is released immediately. * fix(ts): npm run fix * fix(free-tier): HERMES_PREVIEW_FULL_CONNECTORS=1 turns the preview cohort on The preview flag only read exactly true/false, so a bundle built with =1 (HERMES_GUEST_ONBOARDING's spelling) silently sent {} and every guest landed in the external cohort. 1/true now send true and 0/false send false; anything else still omits the field. * fix(desktop): explain missing remote profiles * fix(desktop): lint missing-profile error and move its tests to a sibling file Keeps remote-lifecycle.test.ts under its code-health line cap and clears the curly/no-useless-escape errors from check:lint. * fix(models): TokenHub's saved endpoint is not a relay, so /model keeps the curated Hy list #134320 gave tencent-tokenhub a profile whose base_url is empty (its auth registry row owns the endpoint). _configured_relay_base_url decides "relay or canonical" from the profile's base_url only, so a model.base_url that `hermes model` saved as TokenHub's normal endpoint read as a relay: the picker probed {endpoint}/models and a successful reply replaced the curated Hy list, dropping curated previews the live list lacks. Before #134320 the profile lookup failed and the curated list stood. The canonical endpoint is now the profile's OR the registry row's inference_base_url, both through normalize_route_base_url. A different path on the same host is still a relay. Reported by the review bot on #134320. * test: move the profile-hook keyword compat test into its own module test_chat_completions.py grew past the 1,000-line test-file guideline with #134320's TestProfileHookKeywordCompat (1,006 lines). The test pins the transport's hook-call contract, not response handling, so it moves verbatim to test_profile_hook_keyword_compat.py (985 lines left). Reported by the review bot on #134320. * refactor(models): name the two canonical-endpoint sources; comment says what the code does Review follow-up on the TokenHub relay fix. The comment claimed the registry endpoint counts only "when the profile leaves it empty"; the code (correctly) counts both, and no provider has two different non-empty URLs. The set is now built from two named values instead of an inline hanging comprehension. Considered and not taken: reusing agent_init._provider_default_routes. It also counts the other Moonshot region and Gemini's OpenAI-compat path as canonical, which would turn a kimi-coding user pinned to the other region's host from a relay probe into native discovery. That is a behaviour change for a separate PR. * fix(tui): drop the unused route lookup in _fallback_session_info The lookup ran a profile-scoped runtime resolution on every lazy session.info and discarded it; _lazy_info_route resolves the same route on the next line. * refactor(plugins): one per-home enable lock, not two _set_plugin_enabled now takes its own per-home _selection_lock for every caller (CLI, dashboard, install card). The install card's older _enable_lock wrapped the same call with a second, identical per-home lock; drop it. * refactor(setup): setup_choose reads the facts module directly; drop the dead fork skip - The fork card has had no `skip` row since its rows became app-owned (mind, machine, figure plus two first tasks), and typed text matching no row comes back as `typed`, so `picked == "skip"` could never be true. A skipped fork ar…
3 tasks done
alt-glitch
added
os/windows
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Contract C3 of the updater campaign: after the commit point nothing fails
hermes update.Once the tree has moved,
hermes updateexits 0 unless the code itself was rolled back. A failedpost-commit step prints
⚠, lands in the receipt asoutcome: "success"withfollowups: [{step, reason}], and keeps its own obligation armed so the next launch orhermes update(including the "Already up to date" path) retries it. Exit 2 (refused /concurrent) and exit 1 (nothing committed / rolled back) keep their meaning. The owed-fleet-restart
CLI startup warning stays — it is the "mixed fleet is never silently healthy" guarantee that
replaces exit 1.
Changes
source_completion._complete_locked,source_build.build_update_products,update_cmd_maint._run_post_update_maintenance): launchers, product builds (TUI / web / desktopeach attempted even when another failed), config migration and maintenance each run under
their own guard; profile sync is best-effort (see Round 9). A failed web build no longer skips config migration or the gateway restart.
Tail failures keep
source-completion-pendingarmed; the install stamp is written only when thetail is complete.
update_cmd_fleet._verify_fleet_after_update): a stale / down / unaccounted gateway isa
gateway_restartfollow-up with the fleet obligation kept armed — nosys.exit(1), receiptsuccess. A gateway stopped with no successor (unmapped SIGTERM, a stale manual survivor) nowkeeps the restart owed and is named in the follow-up (codemap §6 V7/V12) instead of discharging the
obligation. A host stamped "restarted" whose fleet is still off the checkout restarts again instead
of dead-ending in exit 1 (§6 V4); the skip path no longer writes
failed(§6 V5). The multiplexauto-migration can no longer fail a committed update via
SystemExit(§6 V9).on the host obligation (re-arming it if a pre-restart probe had settled an inventory-less record),
so a gateway that died at boot keeps the startup warning until it serves the checkout instead of
being settled by the gateway-less discharge; a later verify with nothing live to restart keeps it
owed rather than clearing it.
sqlite_runtimefollow-up, "run the installer again") but nolonger fails the update or keeps the tail pending forever (§8 V1).
sweep leaves the fingerprint unstamped so the launch-time sweep retries it (§8 V7, V9; §7 V5).
update_receipt):begin_update_receiptwrites the run's ownarchive file +
latest.jsonasoutcome: "running"; every stage boundary refreshes it; finalizerewrites the same file. The next run marks a
runningrecord whose processes are goneinterrupted(naming its last stage) and says so (§8 V5, V4 double receipt).update_receipts/and theupdate.logtee resolve tohermes_constants.get_default_hermes_root(), never a sticky profile's home (readers too, round 2).hermes_cli/AGENTS.md,website/docs/developer-guide/source-update-completion.md,website/docs/getting-started/updating.md,website/docs/reference/cli-commands.md.Rewritten tests (old behaviour pinned)
test_update_completion_process.py::test_failed_build_preserves_exit_status_without_maintenance→
test_failed_build_after_commit_is_a_followup_and_later_steps_still_run: exit 23 → 0, receiptfailed→success+buildfollow-up, maintenance/restart now run, pending marker stillowed.test_update_fleet_check_fail_closed.py::test_empty_probe_settles_and_fails_only_when_rows_expected:SystemExit(1)/partial→ returns,success+gateway_restartfollow-up, marker not cleared.test_unmapped_stops_are_not_expected_rowskept (row prediction unchanged) + newtest_unmapped_stop_keeps_the_restart_owed.SystemExit(1)/partial→ returns,success+gateway_restartfollow-up, obligation armed:test_manual_serve_deferral.py(non-alive conditions),test_update_fleet_completion.py,test_fleet_matrix_self_restart_pending.py,test_update_external_profile_reconciliation.py,test_update_stale_gateway_yield.py,test_update_fleet_restart_pending.py::test_clean_update_escalates_surviving_serve_as_unaccounted.Each keeps its original verdict assertions (warning text, runtime outcomes, incomplete flag).
test_update_zip_completion.py: unsafe-SQLite / stale-fleet verdicts return True withsqlite_runtime/gateway_restartfollow-ups (was exit 1 /partial); the post-swap"preparation" case moved to
test_zip_build_failure_after_swap_is_a_followup; pre-swap failuresmay leave this run's own
runningrecord (neversuccess) instead of nolatest.json.test_source_build.py::test_update_failure_raises_without_retries_or_replacing_live_app: raisesProductBuildErrornaming the one failure and attempts every product (wasorder[:index+1]);the live-app / stamp checks apply to the desktop failure.
test_source_completion_stamp.py: a build failure withholds the stamp (tail owed); an unsafeSQLite verdict still reports failure but no longer withholds the stamp of a finished tail.
test_update_hangup_protection.py: patchesget_default_hermes_root(the log now lives at the root).test_selected_sqlite_controls_completion_and_action_receiptunchanged:_print_verified_update_completionstill withholds the completion line; only its caller's consequence changed.
tests/hermes_cli/test_update_post_commit_followups.py(running receipt, interruptedreconciliation, root-home receipts, build independence, config-migration follow-up).
Round 2 (review r2-commit-post: MAJOR-4 / amendment A6, m1, m7, m9, m11)
update_completion._bootstrap/_settle_after_commit): a failure in the completion bootstrap'sensure_tools_for_sync/pm.sync_venvused to end in exit 1 with no receipt (Desktop: "still on the previous version"while the tree was new). Now: exit 0,
⚠ Update follow-up 'dependencies' did not finish: <reason> (dependencies not installed yet — the next launch retries), receiptsuccess+followups: [{step: "dependencies"}],source-completion-pendingarmed,.update_exit_code0 ingateway mode. A prepared child that dies without a result settles the same way as a
completionfollow-up (hydrated from its own on-disk
runningrecord, so its stages survive). The parentresumes paused Windows gateways itself when the bootstrap could not.
TAIL_FOLLOWUPSgainsdependencies.KeyboardInterrupthitthe
cmd_updateboundary → receiptfailed/ exit 1 with the tree new. Nowupdate_receipt.finalize_interrupted_update_receiptcloses the runinterrupted(exit 130 — anon-zero exit for an interrupt), keeps a terminal record the child already wrote, and prints
⚠ Interrupted after the code was updated: the new code is in place; its remaining steps are owed and the next launch orhermes updatefinishes them.hermes logs update/handoff(logs.log_file_path),hermes logs list(lists the root's update logs under a profile), the debug bundle,the dashboard's durable
update.logcompletion marker,update_handoff.write_handoff, andpm/receipt._receipt_dir(pm's sync receipts share the dir).(
retain_receipt_manual_serves,_receipt_owed_gateways) reads the running record'scarried_manual_serves;begin_update_receiptcarries a killed run's rows forward.update_cmd_fleet.py2230 → 1902 lines; the post-restart verify phasemoved to
update_cmd_fleet_verify.py(restart-phase helpers are read throughupdate_cmd_fleetso their patches still apply). Tests retarget patches to where production reads.
updating.md(Ctrl-C, stale "fleet check exits non-zero"),source-update-completion.md,hermes_cli/AGENTS.md.Tests changed in round 2 (old behaviour pinned)
test_update_zip_completion.py: restored the dropped "preparation" case astest_zip_dependency_sync_failure_after_swap_is_a_followup[direct|git-failure](realupdate_completion._bootstrapin-process,pm.sync_venvraising after the swap) — red on theprevious head (
✗ Source update preparation failed→ exit 1 →CalledProcessErrorout of_cmd_update_impl). Newtest_ctrl_c_after_the_swap_reports_the_new_code_not_a_failure.test_update_completion_process.py:test_prepare_failure_preserves_correlated_pm_receipt(exit 1 → 0 +
dependenciesfollow-up, pm refusal still returned);test_killed_selected_python_returns_signal_exit_status(137 → 0 +completionfollow-up);test_interrupt_after_child_success_demotes_gateway_marker_at_boundary(KeyboardInterrupt /failed→ SystemExit 130 /interrupted, cleanup error still chained, gateway marker still1).test_update_fleet_restart_pending.py::test_interrupt_between_pull_and_restart_leaves_marker:KeyboardInterrupt → SystemExit 130 (obligation still armed).
test_logs.py::test_update_logs_are_read_from_the_root_home_under_a_profile,test_debug_share_update_logs.py::…root_home_update_logs_under_a_profile,test_web_server.py::test_update_status_recovers_completed_result_after_dashboard_restart(now undera profile home),
test_manual_serve_deferral.py::test_unsaved_manual_restart_warning_survives_a_running_and_a_killed_update— all 4 red with the previous readers.
Live proof
tests/e2e/core/upgrade/git/test_hostile_post.py— realhermes updatefrom a real git checkoutfixture in a bwrap sandbox, disposable homes:
be3fd671d70; round 2 cells: previous head12ca0d5cd96)e5be982b783scripts/build/web.mjs)failed, no follow-ups, config migration skipped⚠line, receiptsuccess+buildfollow-up,source-completion-pendingarmed, config still migrated; the nexthermes updatebuildsweb_dist, clears the obligation, no follow-upspartial⚠,success+gateway_restartfollow-up;hermes statuswarns the restart is owed; a retry with nothing live keeps owing it; once a gateway serves HEAD the warning clearslatest.jsonis pm'ssyncreceipt, no record of the killed updatelatest.jsonis the killed run'srunningrecord; the next update reports itinterruptedhermes update+hermes logs updateprofiles/<name>/logs/update_receipts(base); pm's sync receipts still there (previous head)update.login the root home;hermes logs updateunder the profile shows the root log. (Round 3: pm's own sync receipts are back where pm writes them — see below.)pm.client.ensure_tools_for_syncraises)failed,✗ Source update preparation failed: …⚠ … 'dependencies' … dependencies not installed yet — the next launch retries,success+dependenciesfollow-up, obligation armed; the next update completes with no follow-ups and disarms itfailed(reads "still on the previous version"), no message⚠ Interrupted after the code was updated: the new code is in place…, receiptinterrupted, obligation armed; the next update completes itprevious head: (d)(e)(f) 3 failed (
r2/e2e-prev.log); head:6 passed in 394.54s.What the user sees on head (re-run of (e)/(f) with the sandbox kept):
receipts: (e)
success, exit 0, followups[dependencies]→ next runsuccess, no followups;(f)
interrupted, exit 130 → next runsuccess.Unit (same selection,
-n 6, head vs previous head): no new failures beyond the knownxdist-flaky set (flips both ways). Serial run of the 16 touched files: previous head 824 passed;
head 829 passed + 1 failure that was the intended
KeyboardInterrupt → SystemExit 130change(
test_interrupt_between_pull_and_restart_leaves_marker, updated; its file then 42/42).Round 3 (CI reds on
e5be982b783)Commits
9331a09facd,5605b522a0a,6a556910baf,fb7af08ff35.pm/receipt.pyreverted toorigin/main(pm is not ours).hermes_cliupdate receiptsand
update.log/handoff log stay in the root home; pm's sync receipts stay where pm writes them(
get_hermes_home()), sohermes pm statuskeeps finding them — identical dir for the defaultprofile (the pm e2e below reads the updater's receipt through
hermes pm status).hermes logs/the debug bundle never listed receipts, only logs, so nothing there changes.
Live cell (d) in
test_hostile_post.pynow pins exactly that: noupdate_*receipt under thesticky profile,
hermes pm statusthere shows pm's ownsyncreceipt,hermes logs updateshows theroot log (local
6 passed; on6a55691CI it failed only because it still forbade pm's file).e2e-upgrade (git)test_conflicting_patch_is_recoverable_and_never_reported_complete): exit-0-after-commit appliedto a stash whose restore conflicted, so the user's patch sat parked while the run exited 0 with a
successreceipt. A follow-up is retried by the next launch; a parked stash is not — only the usercan re-apply it. New
update_receipt.record_user_action():_complete_selectedrecords theunsettled-autostash notice as
user_action, the receipt finalizespartial(neversuccess) andhermes updateexits 1; the gateway exit-code marker gets 1; the same holds when a post-commitdependency/build follow-up already settled the run (
_settle_after_commitprints the notice thenever-run completion child would have). The spurious
sqlite_runtimefollow-up that the withheld✓ used to trigger is gone. Real
hermes updatein the git e2e world: base (5605b52) rc=0,receipt
success— FIRES; head 4/4 pass (rc≠0, no "Update complete", stash kept).e2e-upgrade (pm)test_failed_dependency_update_is_reported_as_failed→ rewritten to A6as
test_failed_dependency_update_is_an_owed_followup: uv rejects the release'suv.lockafterthe tree moved, so the code is committed; the test now pins rc=0, HEAD = the release, the loud
⚠ Update follow-up 'dependencies' … dependencies not installed yetline, receiptsuccessnamingthe
dependenciesfollow-up, the tail obligation armed, and the previous generation still selected.The old "must exit non-zero / receipt failed" was the pre-A6 semantic (CI log showed exactly the A6
behaviour). Local run of both e2e files:
10 passed.apps/desktop/e2e/update/build-fail.spec.ts): under C3 the failed Desktopbuild is a
buildfollow-up, soposix.shsaw exit 0 and publishedok:true, manual:false—the relaunched app logged "finished OK" with an old build and no word to the user (CI). It must not
say FAILED / "still on the previous version" (false: the checkout moved).
posix.shnow detects theUpdate follow-up 'build' … desktop app build|Node dependenciesline, skips the (stale) mac bundleswap, relaunches the kept app on Linux too, and publishes the action-required result: "Hermes was
updated, but the Desktop app could not be rebuilt, so it still runs its old build. Run hermes
desktop --force-build in a terminal to rebuild it; the update log has the build error." main.ts
already renders
manualresults as the "needs one more step" dialog — no Desktop copy change. Thespec now expects
detached update finished with manual action+ that text, forbidsFAILED/previous version/finished OK, and asserts HEAD = the release. Realposix.shrun with theCI job's follow-up line: base →
{"ok":true,"manual":false,"message":"Update complete."}; head →{"ok":true,"manual":true,"message":"Hermes was updated, but the Desktop app could not be rebuilt…"},kept app relaunched. Real Electron cell:
Desktop install/update E2E (Linux, build-fail)passed on6a55691(run 37156595451).test_restart_plan_reconciliation::test_live_gateway_pids_from_fleet_…andtest_old_updater_takeover(historical payload schema ×4) patch the fleet-verify module the lookupmoved to and the root-home hand-off;
tests/fixtures/resolution_allowlist.jsonfollows the movedlookup (no stale rows).
test_update_completion_process's fakeupdate_receiptgains the newrecord_user_actionsurface.Not covered (round 3):
windows.ps1still runsdesktop_update_verifyafter exit 0, so a Windowsbuild follow-up surfaces as its verify failure (exit 8) rather than the manual note — the handoff-script
lane owns that file.
Round 5 (integration review R8, R14)
Rebased on current origin/main. Every fix reproduced RED first with the reviewer's real-process probes (bwrap, no network, disposable homes), then GREEN with the same probe.
_complete_selectedkeeps the completion's runtime-safety result separate from exit status, and fleet verification gets the real resultTrue, migration entered → verify getsFalse, migration not entered; still exit 0 +sqlite_runtimefollow-up. Healthy 3.53.1 control: migration enteredtest_unsafe_sqlite_runtime_keeps_exit_zero_but_vetoes_topology_migration(unsafe + healthy)config_migrationowed stepconfig_migrationfollow-up, pending kept, no stamp (exit 0)test_config_format_write_failure_is_owed_not_reported_complete;test_migration_policy[migration-error]now expects the raisebegin_update_receipttakes the carried-debt snapshot wheneverpreviouslackscarried_manual_serves, before the running record replaces latesttest_old_schema_handoff_receipt_keeps_the_previous_runs_manual_serve_debtupdate_finish.py, which old releases reach via--post-swap→_update_takeover) returned exit 1 on a post-commit build failurebuildfollow-up; maintenance and the fleet restart still run; exit 0; the tail stays pending and no install stamp is writtenbuildfollow-up, config migrated, pending kepttest_real_compiler_failure_is_owed_build_and_completion_continues(replaces the old exit-1 test)Unchanged: parked local changes (conflicted autostash) still exit 1. Neither
pm/norscripts/desktop-update/posix.shwas touched.Not covered: an owed tail also postpones auto-migration, which is conservative and matches pre-PR main. Sibling-profile and unreadable-config migration failures are still only reported, not recorded as owed. The R8 CI test uses a shim interpreter that reports SQLite 3.46.1; the real 3.46.1 interpreter was run only in the probe.
Not covered
dependenciesfollow-up boots the newtree whose dependencies are still owed; its own launch sync runs then (no wine2e cell here).
the Desktop passes as
HERMES_HOME).Round 6 (re-review D13 / N2: build stage)
Head
e095f5b4649.Fix. Both completion paths (
update_completion._complete_selectedand the historicalupdate_finish.finish_update) marked the receipt'sbuildstagefailedwhenever any tail follow-up was owed. A run that owed onlyconfig_migrationtherefore reported a failed build. The newupdate_receipt.record_build_stage(followups)marks itfailedonly forbuild,launchers(a product the build stage publishes) orcompletion(the tail raised before it could show those succeeded). An owedconfig_migration,maintenanceordependenciesstep stays its own follow-up and does not touch the build stage.outcomeand exit status are unchanged (success, 0).Proof (real processes).
test_config_format_write_failure_is_owed_not_reported_completeruns a real completion in which the profile home refuses the config write. It now also asserts the build stage. Pre-fix result:{'deps': 'success', 'build': 'failed', 'restart': 'skipped'}(RED). This head:build: success.test_real_compiler_failure_is_owed_build_and_completion_continues(a real TypeScript error) now asserts that the build stage isfailed: a genuine build follow-up still fails the stage.update_receipt.pyintest_update_completion_process.pygainedrecord_build_stage.Consumers of the stage field. No consumer relied on the old, coarse meaning:
observability/shared_metrics_update.pyderiveshermes.update.stagefrom the stages. It used to over-count build failures and now reports them correctly.handoff-result.tsreads.hermes-update-result.json(ok,exit_code,message).sync-status.tsreadsoutcome,pm_sync_outcome,stepsandvenv_rebuild.posix.shandwindows.ps1do not parse receipt stages, anddoctordoes not read receipts.Round 8 (review r8: P2 post-commit paths, design D5)
Head
344497d6192. Commits:1fa0ca961e8(reviewer's6bba7332a3a, cherry-picked unchanged),b138ed572e3(fix + tests),344497d6192(docs).Ported: receipt identity and durable terminal facts (
6bba7332a3a). A parent's stalerunningsnapshot no longer overwrites a terminal record that the child already wrote. The receipt also records the writer's process creation time, so a recycled PID cannot keep a dead owner looking alive.record_user_actionnow persists immediately. Its four process tests (test_update_receipt_durability.py) pass here.D5: closes the three post-commit paths that could still report failure.
logs/update_receiptsreplaced by an ordinary file)_finishprinted✓ Update complete!and⚠ Update receipt not written, then returned 1._settle_after_commitreturned 1update_receipt.finalized_receipt(update_id)keeps the correlated terminal record finalized in memory, and_read_terminal_receiptfalls back to it, sorun_completion's unchanged correlation gate is satisfied. The⚠warning stays. A user action (local changes left in the stash) still returns 1test_receipt_store_failure_after_commit_keeps_exit_zero_and_a_correlated_receipt[finish|settle].update_exit_codewas overwritten with 1, so the gateway/updatereader announced a failed update while the run exited 0 withsuccess0. The restart debt is reported by thegateway_restartfollow-up, the host obligation and the⚠lines in the forwarded output. All three marker writers were removed (update_cmd_fleet×2,update_cmd_windows×1), not only the Windows one. A resume that raised could still fail the run until Round 9test_restart_failure_is_owed_and_never_reported_as_a_failed_update[failed-unit|abort|windows]. A barrier reads the marker at the start of verification, where a gateway reader could already consume it, and again at the endprofile_syncfollow-upTAIL_FOLLOWUPS, so an owed profile sync clearedsource-completion-pendingand the install stamp was writtenSystemExit), the tail stays owed and no stamp is written. A per-profile error that_sync_profiles_after_updatecatches is printed and is not owed (narrowed in Round 9)test_profile_sync_followup_keeps_the_tail_owedProof. The production code at
1fa0ca961e8fails all 6 new cells (RED). The same 6 cells pass at this head (GREEN):test_update_post_commit_safety.py10/10. Each RED failure is the asserted one:code 1 == 0×2, marker'1' == '0'×3, andpending False×1. The only replaced functions are host boundaries: the systemd unit restart, the process table and the fresh-child recovery, the Windows SCM resume, and the gateway service recorders the harness already used. Forprofile_sync, aSystemExitis raised at the step itself. The restart phase, abort-recovery verdict, verification, receipt and marker code all run unmodified. A selection of 110 update/gateway test files run locally with the canonical runner produced identical failure sets at the old and new heads: 208 failures each, all from the local real-home I/O guard and this host's environment. No test started failing.CI at
344497d6192: 45 pass, 0 fail, 23 skipped. This includes all sixe2e-upgradelanes, the four Linux Desktop update E2E cells, Windows-only (x64 and arm64), the Windows install + update E2E, and Install & Update E2E on linux, windows and macos.Not covered (round 8):
windows_resumefollow-up amended after finalization lives only in that process's printed output when the store is down. Nothing retries publishing a receipt the store refused._sync_profiles_after_updatestill swallowsExceptioninternally (review F1), soprofile_syncis owed only when the step escapes, for example withSystemExit. This round fixes the tail classification, not the helper's error protocol._resume_windows_gateways_and_merge_outcome, with a refusing SCM resume, on Linux.scripts/desktop-update/posix.shwith Desktop update hand-off scripts hold the update marker as a lock and report committed updates as committed #132354, andtests/hermes_cli/test_update_zip_completion.pywith fix(update): make the git/ZIP swap a single crash-safe commit point #132361 (that one already existed ate095f5b4649). A merge with fix(update): a killed Windows updater no longer strands paused gateways, and never resumes them onto a torn tree #132338 (WIN) is clean.Round 9 (re-review R8: M4, m6)
Head
b822fae6ba4. Commits:18ae9dd8fb5(fix + tests),b822fae6ba4(docs).M4: a refused Windows resume after the commit point no longer fails the update, on any of the three paths.
finally(and theCalledProcessError→ ZIP fallbackfinally) in_cmd_update_implrun_completioncopied the child'sresume_needed=Trueback into the parent's token. Thefinallyran the resume again, and itsRuntimeErrorreachedmain.py'sBaseExceptionhandler: exit 1 and.update_exit_code=1 ("/update failed")_complete_source_updatemarks the attempt settled. Thefinallyresumes only what is still owed, and a failure there is thewindows_resumefollow-up instead of an exceptionupdate_finish.main(historical takeover)code = 1when the resume raisedwindows_resumefollow-up (amended onto the terminal receipt when the run already finalized). The exit status is unchanged, andresume_handledreports the attempt so the historical parent does not replay itTests (real processes; the only replaced function is the Windows SCM resume, which refuses):
test_update_post_commit_safety.py::test_zip_update_never_replays_a_refused_resume_after_the_commit_pointdrives the realmain.cmd_update→_cmd_update_implZIP route →_complete_source_update→run_completion, with a real completion child that answers like_finishafter a refused resume. Host inventory, backup, hangup handling and the ZIP download/swap are the other boundaries. It asserts exit 0, marker0, onewindows_resumefollow-up, and no parent resume call (neitherfinallynor atexit).test_update_finish.py::test_refused_gateway_resume_after_commit_is_a_followup_not_exit_1runs the realupdate_finish.pychild (the existing takeover fixture). It asserts exit 0,outcome: success, awindows_resumefollow-up, marker0andresume_handled: true.RED with this head's tests on the
344497d6192production code: the ZIP cell exits 1 with thefinally'sRuntimeErrorand anException ignored in atexit callbackreplay; the takeover cell exits 1 (Gateway recovery failed). GREEN at this head. On a local trial merge with WIN (13017670871),test_update_post_commit_safety.py,test_update_finish.pyandtest_update_pause_record.pyall pass (41 passed, 0 failed). A selection of 116 update/gateway files was run with the canonical runner. Its failure set matches the old head's local environment failures (real-home I/O guard, this host's.worktrees/manifest.json), and each extra one also fails on344497d6192. No test started failing.CI at
b822fae6ba4: 46 pass, 0 fail, 23 skipped, including the Windows install + update E2E.m6: claim narrowed.
_sync_profiles_after_updatestill swallowsException, soprofile_syncis owed only when the step escapes (for exampleSystemExit). The docs (source-update-completion.md) and this body now say exactly that. The helper's error protocol is unchanged.Not covered (round 9):
CalledProcessError→ ZIP fallbackfinally(Windows-only: it requires_is_windows()) shares the helper with the ZIP route but has no separate cell.finally/atexit run it once and report a failure as a follow-up. That attempt is not skipped.scripts/desktop-update/posix.shwith Desktop update hand-off scripts hold the update marker as a lock and report committed updates as committed #132354, andtests/hermes_cli/test_update_zip_completion.pywith fix(update): make the git/ZIP swap a single crash-safe commit point #132361. Both already exist at the base. A merge with WIN fix(update): a killed Windows updater no longer strands paused gateways, and never resumes them onto a torn tree #132338 is clean.Infographic