fix(update): refresh installer bootstrap-cache scripts on every update (stale hermes-setup mitigation) - #82229
Merged
Merged
Conversation
… update Pre-#67193 hermes-setup binaries (June 2026 and earlier, including the newest published build) resolve bootstrap-cache/install-<branch>.ps1 by "exists -> reuse forever": a branch-ref cache entry written at install time is never re-downloaded, so every GUI update/repair executes a months-stale install script. The binary has no self-update path, so no amount of `hermes update` fixes it. Live incident (2026-08-09, ryanc): install-main.ps1 cached June 4 lacked the #81327 venv process-tree sweep; the bootstrap venv stage died with "Cannot remove item venv\Scripts\python.exe: Access denied" on a straggler backend pair, twice, despite every relevant fix already being merged on main - the installer simply never ran that code. Fix: `_refresh_bootstrap_cache_scripts()` runs at the end of every update path (git, zip, already-up-to-date repair), overwriting mutable branch-ref cache entries with the freshly pulled scripts/install.ps1 / install.sh. The stale binary's unconditional reuse becomes a feature: it "reuses" a file the update keeps permanently current. Post-#67193 installers re-download on every run anyway, so this is a harmless pre-seed of identical bytes for them. Scope guards: 40-hex commit-SHA entries are immutable pins and are never touched; .ps1 gets the UTF-8 BOM to match the installer's cache format (#67193); best-effort - a failed refresh never fails the update. E2E on the incident machine: poisoned the real bootstrap-cache/install-main.ps1 with a stub, ran the real function - healed byte-exact to the checkout's script (BOM intact, #81327 tree-kill sweep present).
Contributor
૮ >ﻌ< ა ci reviewran on 7423ffa
|
1 task
teknium1
added a commit
that referenced
this pull request
Aug 9, 2026
…match installer pin rules Two cache-key correctness follow-ups to #82229 (review feedback): 1. Abbreviated commit pins are immutable too. The installer's is_valid_commit() accepts 7-40 hex chars, but the Python refresh exempted only exactly-40-hex names — an abbreviated pin like install-4ce1994.ps1 could be overwritten with a branch script. The predicate now mirrors the Rust rule (7-40 hex = immutable, never rewritten), applied to the sanitized target ref. 2. Refresh only the update-target ref's cache key. The helper rewrote EVERY mutable-ref entry with the active checkout's script: with install-main.ps1 and install-bb_gui.ps1 coexisting, updating main replaced both with main's script — cross-branch cache poisoning in the other direction. It now computes the single cache key for the branch being updated, using the installer's own ref sanitization (sanitize_ref: non [A-Za-z0-9._-] -> '_', so bb/gui -> install-bb_gui.ps1), and touches nothing else. Entries the bootstrapper never wrote are not created. The branch is threaded from the existing `branch = _resolve_update_branch(args)` in both _cmd_update_impl call sites and _update_via_zip (main-only by its own guard). Regression tests lock down both invariants: abbreviated-SHA pin untouched (including when passed as the branch), coexisting mutable refs (main refresh leaves install-bb_gui.ps1 byte-identical), sanitize_ref parity, and uncached-ref no-op. E2E on the incident machine's real bootstrap-cache: planted a stale install-main.ps1 + sibling install-bb_gui.ps1 + abbreviated pin install-4ce1994.ps1; refresh("main") healed main byte-exact and left both others untouched; refresh("4ce1994") was a no-op. The pre-existing 40-hex pin entry in the real cache was also untouched.
kotasiddharth
pushed a commit
to kotasiddharth/hermes-agent
that referenced
this pull request
Aug 9, 2026
…match installer pin rules Two cache-key correctness follow-ups to NousResearch#82229 (review feedback): 1. Abbreviated commit pins are immutable too. The installer's is_valid_commit() accepts 7-40 hex chars, but the Python refresh exempted only exactly-40-hex names — an abbreviated pin like install-4ce1994.ps1 could be overwritten with a branch script. The predicate now mirrors the Rust rule (7-40 hex = immutable, never rewritten), applied to the sanitized target ref. 2. Refresh only the update-target ref's cache key. The helper rewrote EVERY mutable-ref entry with the active checkout's script: with install-main.ps1 and install-bb_gui.ps1 coexisting, updating main replaced both with main's script — cross-branch cache poisoning in the other direction. It now computes the single cache key for the branch being updated, using the installer's own ref sanitization (sanitize_ref: non [A-Za-z0-9._-] -> '_', so bb/gui -> install-bb_gui.ps1), and touches nothing else. Entries the bootstrapper never wrote are not created. The branch is threaded from the existing `branch = _resolve_update_branch(args)` in both _cmd_update_impl call sites and _update_via_zip (main-only by its own guard). Regression tests lock down both invariants: abbreviated-SHA pin untouched (including when passed as the branch), coexisting mutable refs (main refresh leaves install-bb_gui.ps1 byte-identical), sanitize_ref parity, and uncached-ref no-op. E2E on the incident machine's real bootstrap-cache: planted a stale install-main.ps1 + sibling install-bb_gui.ps1 + abbreviated pin install-4ce1994.ps1; refresh("main") healed main byte-exact and left both others untouched; refresh("4ce1994") was a no-op. The pre-existing 40-hex pin entry in the real cache was also untouched.
ma1138569845
pushed a commit
to ma1138569845/dechnicAuditor-agent
that referenced
this pull request
Aug 10, 2026
…match installer pin rules Two cache-key correctness follow-ups to NousResearch#82229 (review feedback): 1. Abbreviated commit pins are immutable too. The installer's is_valid_commit() accepts 7-40 hex chars, but the Python refresh exempted only exactly-40-hex names — an abbreviated pin like install-4ce1994.ps1 could be overwritten with a branch script. The predicate now mirrors the Rust rule (7-40 hex = immutable, never rewritten), applied to the sanitized target ref. 2. Refresh only the update-target ref's cache key. The helper rewrote EVERY mutable-ref entry with the active checkout's script: with install-main.ps1 and install-bb_gui.ps1 coexisting, updating main replaced both with main's script — cross-branch cache poisoning in the other direction. It now computes the single cache key for the branch being updated, using the installer's own ref sanitization (sanitize_ref: non [A-Za-z0-9._-] -> '_', so bb/gui -> install-bb_gui.ps1), and touches nothing else. Entries the bootstrapper never wrote are not created. The branch is threaded from the existing `branch = _resolve_update_branch(args)` in both _cmd_update_impl call sites and _update_via_zip (main-only by its own guard). Regression tests lock down both invariants: abbreviated-SHA pin untouched (including when passed as the branch), coexisting mutable refs (main refresh leaves install-bb_gui.ps1 byte-identical), sanitize_ref parity, and uncached-ref no-op. E2E on the incident machine's real bootstrap-cache: planted a stale install-main.ps1 + sibling install-bb_gui.ps1 + abbreviated pin install-4ce1994.ps1; refresh("main") healed main byte-exact and left both others untouched; refresh("4ce1994") was a no-op. The pre-existing 40-hex pin entry in the real cache was also untouched.
randlee
pushed a commit
to randlee/hermes-agent
that referenced
this pull request
Aug 11, 2026
…match installer pin rules Two cache-key correctness follow-ups to NousResearch#82229 (review feedback): 1. Abbreviated commit pins are immutable too. The installer's is_valid_commit() accepts 7-40 hex chars, but the Python refresh exempted only exactly-40-hex names — an abbreviated pin like install-0a62495.ps1 could be overwritten with a branch script. The predicate now mirrors the Rust rule (7-40 hex = immutable, never rewritten), applied to the sanitized target ref. 2. Refresh only the update-target ref's cache key. The helper rewrote EVERY mutable-ref entry with the active checkout's script: with install-main.ps1 and install-bb_gui.ps1 coexisting, updating main replaced both with main's script — cross-branch cache poisoning in the other direction. It now computes the single cache key for the branch being updated, using the installer's own ref sanitization (sanitize_ref: non [A-Za-z0-9._-] -> '_', so bb/gui -> install-bb_gui.ps1), and touches nothing else. Entries the bootstrapper never wrote are not created. The branch is threaded from the existing `branch = _resolve_update_branch(args)` in both _cmd_update_impl call sites and _update_via_zip (main-only by its own guard). Regression tests lock down both invariants: abbreviated-SHA pin untouched (including when passed as the branch), coexisting mutable refs (main refresh leaves install-bb_gui.ps1 byte-identical), sanitize_ref parity, and uncached-ref no-op. E2E on the incident machine's real bootstrap-cache: planted a stale install-main.ps1 + sibling install-bb_gui.ps1 + abbreviated pin install-0a62495.ps1; refresh("main") healed main byte-exact and left both others untouched; refresh("0a62495") was a no-op. The pre-existing 40-hex pin entry in the real cache was also untouched.
blut-agent
pushed a commit
to blut-agent/hermes-agent-fork
that referenced
this pull request
Aug 11, 2026
…match installer pin rules Two cache-key correctness follow-ups to NousResearch#82229 (review feedback): 1. Abbreviated commit pins are immutable too. The installer's is_valid_commit() accepts 7-40 hex chars, but the Python refresh exempted only exactly-40-hex names — an abbreviated pin like install-4ce1994.ps1 could be overwritten with a branch script. The predicate now mirrors the Rust rule (7-40 hex = immutable, never rewritten), applied to the sanitized target ref. 2. Refresh only the update-target ref's cache key. The helper rewrote EVERY mutable-ref entry with the active checkout's script: with install-main.ps1 and install-bb_gui.ps1 coexisting, updating main replaced both with main's script — cross-branch cache poisoning in the other direction. It now computes the single cache key for the branch being updated, using the installer's own ref sanitization (sanitize_ref: non [A-Za-z0-9._-] -> '_', so bb/gui -> install-bb_gui.ps1), and touches nothing else. Entries the bootstrapper never wrote are not created. The branch is threaded from the existing `branch = _resolve_update_branch(args)` in both _cmd_update_impl call sites and _update_via_zip (main-only by its own guard). Regression tests lock down both invariants: abbreviated-SHA pin untouched (including when passed as the branch), coexisting mutable refs (main refresh leaves install-bb_gui.ps1 byte-identical), sanitize_ref parity, and uncached-ref no-op. E2E on the incident machine's real bootstrap-cache: planted a stale install-main.ps1 + sibling install-bb_gui.ps1 + abbreviated pin install-4ce1994.ps1; refresh("main") healed main byte-exact and left both others untouched; refresh("4ce1994") was a no-op. The pre-existing 40-hex pin entry in the real cache was also untouched.
33hodl
pushed a commit
to 33hodl/hermes-agent
that referenced
this pull request
Aug 12, 2026
…match installer pin rules Two cache-key correctness follow-ups to NousResearch#82229 (review feedback): 1. Abbreviated commit pins are immutable too. The installer's is_valid_commit() accepts 7-40 hex chars, but the Python refresh exempted only exactly-40-hex names — an abbreviated pin like install-4ce1994.ps1 could be overwritten with a branch script. The predicate now mirrors the Rust rule (7-40 hex = immutable, never rewritten), applied to the sanitized target ref. 2. Refresh only the update-target ref's cache key. The helper rewrote EVERY mutable-ref entry with the active checkout's script: with install-main.ps1 and install-bb_gui.ps1 coexisting, updating main replaced both with main's script — cross-branch cache poisoning in the other direction. It now computes the single cache key for the branch being updated, using the installer's own ref sanitization (sanitize_ref: non [A-Za-z0-9._-] -> '_', so bb/gui -> install-bb_gui.ps1), and touches nothing else. Entries the bootstrapper never wrote are not created. The branch is threaded from the existing `branch = _resolve_update_branch(args)` in both _cmd_update_impl call sites and _update_via_zip (main-only by its own guard). Regression tests lock down both invariants: abbreviated-SHA pin untouched (including when passed as the branch), coexisting mutable refs (main refresh leaves install-bb_gui.ps1 byte-identical), sanitize_ref parity, and uncached-ref no-op. E2E on the incident machine's real bootstrap-cache: planted a stale install-main.ps1 + sibling install-bb_gui.ps1 + abbreviated pin install-4ce1994.ps1; refresh("main") healed main byte-exact and left both others untouched; refresh("4ce1994") was a no-op. The pre-existing 40-hex pin entry in the real cache was also untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Closes the last layer of the 2026-08-09 Windows update-failure incident:
hermes updatenow refreshes the Desktop installer'sbootstrap-cachescripts from the fresh checkout on every run, so stale hermes-setup binaries stop executing months-old install code.The problem
Pre-#67193
hermes-setup.exebuilds (June 2026 and earlier — including the newest published binary, June 5) resolvebootstrap-cache/install-<branch>.ps1as "exists → reuse forever". A branch-ref script cached at install time is never re-downloaded, and the binary has no self-update path, so every GUI update/repair runs frozen-in-time install code no matter how current the repo is.Live incident:
install-main.ps1cached June 4 lacked #81327's venv process-tree sweep → the bootstrap venv stage died twice withCannot remove item venv\Scripts\python.exe: Access deniedon a straggler backend pair — hours after every relevant fix was already merged on main. The fixes existed; the installer never executed them.The fix
_refresh_bootstrap_cache_scripts(), called at the end of all three update paths (git, zip fallback, already-up-to-date repair): overwrite mutable branch-ref cache entries with the freshly pulledscripts/install.ps1/install.sh. The stale binary's unconditional-reuse bug becomes a feature — it "reuses" a file the update keeps permanently current. Post-#67193 installers re-download on every run anyway, so for them this is a harmless pre-seed of identical bytes.Scope guards:
.ps1gets the UTF-8 BOM to match the installer's cache format ([Bug]: GUI installer fails on non-English Windows — install.ps1 parse error from encoding, plus stderr decoded as UTF-8 hides the real message #67193 encoding fix)Type of Change
How to Test
pytest tests/hermes_cli/test_update_bootstrap_cache_refresh.py -o "addopts=--timeout-method=thread"— 8 passed (overwrite ps1+BOM / sh no-BOM / BOM-not-doubled / SHA-pin untouched / current-entry silent / missing dir/sources no-op / never-raises).test_update_venv_health.py,test_update_orphan_backend_reap.py,test_update_modified_notice.py— 22 passed.E2E verification (the actual incident machine)
Poisoned the real
%LOCALAPPDATA%\hermes\bootstrap-cache\install-main.ps1with a stub, ran the real function: healed byte-exact to the checkout's script — BOM intact, #81327 tree-kill sweep present (taskkill /F /T /PID $treePid). Output:✓ Refreshed installer bootstrap-cache script(s): install-main.ps1.Incident fix-stack context
#82158 (scan truncation) → #81327 (installer tree-kill) → #82179 (orphan reap) → #82191 (Desktop teardown + tree-aware classifier) → this (make sure stale installers actually run all of the above).
Checklist
.ps1and.sh; pure-Python, no OS gating needed; silent no-op whenbootstrap-cache/doesn't exist (CLI-only installs)