Skip to content

fix: Windows update no longer hangs and Bot Mode chats survive it (dead-pin adopt + ownership preserve + hand-off reap) - #90738

Closed
teknium1 wants to merge 4 commits into
mainfrom
fix/bot-mode-update-recovery-incident
Closed

teknium1 wants to merge 4 commits into
mainfrom
fix/bot-mode-update-recovery-incident

Conversation

@teknium1

Copy link
Copy Markdown
Collaborator

Summary

Three fixes for the Windows Desktop update → Bot Mode breakage incident reported live on 2026-08-20 (Teknium's production machine), bundled into one PR at his request so another agent can compare/merge against a parallel PR. Each is independently revertable (three commits, disjoint subsystems).

The incident, in the user's words: a desktop update "hanged for 12 minutes (usually ~2-3min)"; after a force-close, "all the bot profiles chats were empty, some wouldn't connect and failed at 'Waking up '"; and after reinstalling, bots he'd opened while broken were stuck on a fresh "introduce yourself" session while his default and one other profile were left on new empty chats with the originals seemingly lost.

No user data was ever deleted. Every symptom is a surfacing/lifecycle failure — messages sat intact (but hidden + unpinned) on disk the whole time. This PR repairs the three mechanisms and the damaged profiles self-heal on next open.


Root-cause analysis (evidence-backed)

Logs came from the user's ~/.hermes/agent.zip (Windows). Reproduced locally against a real profile DB driving a real Electron instance over CDP + the gateway RPC.

Fault 1 — 12-minute update hang (hermes_cli/update_cmd.py)

update.log / desktop-update-handoff.log showed the GUI updater running update --yes --gateway --force and then stalling, with a swarm of leaked per-profile serve backends holding cryptography\_rust.pyd:

✗ Other Hermes processes are running from this install's venv:
  PID …  python.exe … -m hermes_cli.main --profile mr-tester serve   ← Desktop backend
  PID …  … --profile probe-inherit se  / turqoise / clippy / maroon / …  (9+ profiles)
  On Windows these keep native extension files (.pyd) locked …

Windows can't replace a mapped .pyd held by a running process, so the dependency sync dead-ended. The existing reaper (_orphaned_desktop_backend_pids) only reaps backends whose supervising parent is provably dead; it returns None (keep refusing) the instant any holder still has a live parent. During a hand-off several backends still had a lingering parent (the tearing-down Electron process, or the two-hop venv python.exe launcher→uv-managed worker chain mid-exit), so the whole set was disqualified and the update hung.

Fault 2 — corrupt backend-ownership.json erases live-backend records (apps/desktop/electron/backend-ownership.ts, #89298)

parseBackendOwnership returned [] on any JSON parse error, and reapOrphans then unconditionally rewrote the file — so one corrupt read permanently orphaned every backend it described (the reporter's "28 leaked serves"). This amplifies Fault 1 (more leaked backends → more .pyd locks) and leaves backends the desktop can no longer track.

Fault 3 — bots reintroduce themselves on a dead chat pin (apps/desktop/src/plugins/hermes-bots/plugin.js)

A bot's canonical-chat pin (profile.yamlui_meta.hermes-bots.chat) can go stale: it points at a session id that was never persisted or was rewritten past recovery. Live-reproduced: pin 20260819_162021_63fdca in profile.yaml, but sage's state.db actually holds 20260819_201302_9619b9 "Bot Chat" (20 messages). On a dead pin:

  1. profiles.list preferred_sessionnull (verified over RPC).
  2. The recovery branches in openBotCanonicalChat looked for an adoptable chat via last_session / preferred_session — but both are computed from a hidden-EXCLUDING query (_latest_profile_session_rowslist_sessions_rich(source=None)), and Bot Mode sessions are hidden=1 by design.
  3. So the real Bot Chat was never found → fall through to createCanonicalChatnew session + "Hey, tell me about yourself!" kickoff.

This is why: bots opened while broken nulled their pin and are stuck on the intro; bots not touched kept their pins and history; and the pin, once nulled to disk (profile.yaml), stays null across a reinstall.

(The "Waking up … / won't connect" symptom is Faults 1+2 in combination: leaked/evicted backends + the desktop's LRU cap of 3 profile backends vs. 9+ bot profiles, so most bots cold-start or fail to become ready.)


Changes

1. plugins/hermes-bots/plugin.js — adopt the existing hidden Bot Chat instead of reintroducing

  • findExistingCanonicalBotChat(name): browses the profile's hidden sessions via session.list { include_hidden: true } (the same hidden-inclusive view the Bots "Sessions" submenu already uses) and returns the existing "Bot Chat" id.
  • adoptOrCreateCanonicalChat(name): adopts that chat (open + re-pin via saveBotMeta) if present; mints a fresh canonical chat only when there genuinely is none.
  • All three mint-new branches in openBotCanonicalChat (no-pin/no-history, pin-resolves-to-non-BotChat, definitively-gone) now route through it. The bot is reintroduced ONLY when there is truly no forever-chat to return to.
  • isCanonicalBotChatHistory matches on root_title/title === 'Bot Chat' (only adopts Bot Mode plumbing — never an unrelated user conversation, preserving the Desktop: BOTS sidebar preview shows wrong session content on click #88200 identity guard).

2. electron/backend-ownership.ts (+main.ts) — preserve records on a corrupt ownership file (#89298)

  • parseBackendOwnershipDetailed distinguishes "empty/valid" from "corrupt/unreadable".
  • reapOrphans / clear skip the rewrite when the file is corrupt (quarantine it as .corrupt via the new store quarantine hook) instead of erasing live-backend records.
  • main.ts wires the store's quarantine implementation.

3. hermes_cli/update_cmd.py (+main.py lazy-export) — reap leaked backends during a GUI hand-off

  • _handoff_reapable_backend_pids(matches): reaps surviving Hermes serve/dashboard backends from this venv regardless of a live parent, but only when the caller confirms the hand-off context.
  • Wired as the final rung before the existing dead-end, gated on: args.gateway AND the update-incomplete marker present (_update_marker_path().exists()) AND no live hermes.exe shim (_detect_concurrent_hermes_instances). In that window nothing legitimate supervises/respawns a serve backend (Desktop tree-kills its backends and parks any relaunch behind the marker, fix(desktop/windows): in-app update cascades into backend restart loop when user relaunches Hermes mid-update #50238), so a survivor is a leak, not a race.
  • Conservative: a non-backend holder (operator REPL, stray script) disqualifies the whole set → keep refusing; psutil unavailable → keep refusing. Reaps via the existing _stop_process_trees (taskkill /T /F, matching the Desktop's forceKillProcessTree and install.ps1).

Validation

Area Result
hermes-bots plugin suite 335/335 pass
New canonical-chat-adopt.test.mjs 3/3 pass; 3/3 FAIL on reverted plugin (sabotage-checked)
backend-ownership.test.ts 17/17 pass (incl. corrupt-file preservation)
New test_update_handoff_backend_reap.py 7/7 pass; 7/7 FAIL on reverted update_cmd (sabotage-checked)
Update-guard regression suites (orphan reap, scan blockers, venv health, self-lock) 78/78 pass
tsc -p tsconfig.electron.json clean
ruff check (update_cmd.py, main.py, new test) clean
eslint (main.ts, backend-ownership.ts, plugin.js) 0 errors
npm run build (apps/desktop) ✓ dist built

Live end-to-end reproduction of Fault 3 (the money shot)

Driven over the gateway RPC against a real profile DB with a dead pin set in profile.yaml:

dead-pin preferred_session: null            ← the trigger that used to reintroduce
ADOPT TARGET: 20260819_201302_9619b9  title: Bot Chat   ← findExistingCanonicalBotChat via session.list include_hidden
RESUMED history message count: 4            ← session.resume on the adopted id returns REAL history, not a new intro

Before the fix: dead pin → null → mint intro session (the user's bug). After: dead pin → adopt the real hidden Bot Chat → full history restored.


Recovery for the already-damaged machine

The user's damaged profiles self-heal: their old Bot Chats are hidden + unpinned on disk, and Fault-3 fix re-adopts them by title on next open (re-pinning profile.yaml). No manual restore or data recovery needed. If desired, a one-time offline pin-repair pass could be added, but it is not required — opening each bot repairs its own pin.


Notes for the comparing agent / reviewer

  • Independence: commits are plugin.js (renderer) / backend-ownership.ts+main.ts (electron main) / update_cmd.py+main.py (CLI). No shared hunks — any one can be dropped or cherry-picked.
  • Fault 2 overlaps existing PR fix(desktop): corrupt backend-ownership.json no longer erases records of live backends (#89298) #90497 (same commit, 010d4e059) — if fix(desktop): corrupt backend-ownership.json no longer erases records of live backends (#89298) #90497 lands first, drop that commit here on rebase.
  • Environment caveat on my live testing: reproduced on Linux (XWayland-rootless); the CDP/DOM + gateway-RPC paths are faithful, but a "Waking up" swap-overlay hang I observed is a rootless-XWayland WS-reconnect artifact of my box, NOT part of the bug — which is why Fault 3 was verified at the deterministic RPC layer, not through the wedged overlay. The update-path fix (Fault 1) is Windows-only and is covered by unit tests with a fake psutil (cannot be exercised on Linux at runtime).
  • What this PR does NOT change: the LRU cap of 3 profile backends (a separate UX question — bots beyond the cap still cold-start as "Waking up"); the Bot Mode hidden=1-by-design storage; the kickoff/intro copy. Those are candidates for follow-ups if the maintainer wants the roster to keep more bots warm.

Infographic

Bot Mode + Update Recovery

…hat instead of reintroducing the bot on a new session

Symptom (reported live on Windows after an update): opening a bot showed
a fresh 'introduce yourself' session and the real forever-chat history
looked gone. Root cause: the pinned canonical-chat id can go stale (points
at a session id that was never persisted or was rewritten past recovery).
On a dead pin, profiles.list returns preferred_session=null, and the
recovery branches relied on last_session/preferred_session for an adoptable
history — but both are computed from a hidden-EXCLUDING query, and Bot Mode
sessions are hidden by design. So the real Bot Chat (intact on disk) was
never found and every open minted a new intro.

Fix: before minting, findExistingCanonicalBotChat browses the profile's
hidden sessions (session.list include_hidden:true — the same view the
Sessions submenu uses) and adopts the existing 'Bot Chat'. All three
mint-new branches route through adoptOrCreateCanonicalChat; the bot is
reintroduced ONLY when there is genuinely no forever-chat to return to.
The user's messages were never lost — only unpinned.

Verified end-to-end over gateway RPC against a real profile DB: dead pin ->
preferred_session=null -> session.list finds the hidden Bot Chat ->
session.resume returns the real history (not a new intro). 335/335 plugin
tests; new adopt tests fail on reverted plugin (sabotage-checked).
… of live backends (#89298)

parseBackendOwnership returned [] for unreadable JSON and reapOrphans
unconditionally rewrote survivors — one corrupt read replaced the roster
with [], permanently orphaning every backend it described. The sweep now
detects corruption, parks the file as .corrupt (evidence preserved), and
skips the rewrite; empty/missing files keep the legacy sweep behavior.
… of dead-ending the venv sync (Windows)

Field incident (2026-08-20): a Windows Desktop update hand-off
(update --yes --gateway --force) left a swarm of per-profile serve
backends (mr-tester, probe-inherit, turqoise, clippy, maroon, …) holding
cryptography/_rust.pyd. Some still had a live parent (the tearing-down
Electron process, or the venv launcher->worker two-hop chain mid-exit),
so the strict orphan-only reap (_orphaned_desktop_backend_pids, which
bails the instant ANY holder has a live parent) disqualified the whole
set and the venv-holder guard dead-ended. The user saw a ~12-minute hang,
force-closed, and the half-done state stranded bot sessions.

New rung: _handoff_reapable_backend_pids reaps surviving Hermes
serve/dashboard backends from this venv — live parent or not — but ONLY
in the hand-off context the caller gates on: args.gateway AND the
update-incomplete marker present AND no live hermes.exe shim. In that
window nothing legitimate supervises or respawns a serve backend (the
Desktop tree-kills its backends and parks any relaunch behind the marker,
#50238), so a surviving backend is a leak, not a race. A non-backend
holder (operator REPL, stray script) still disqualifies the whole set;
psutil-unavailable returns None (keep refusing). Wired as the final rung
before the existing dead-end, after the orphan-only reap.
@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

running on 05d41ab — chore: retrigger CI (zero-job dispatch failure, auto-heal)


Still running 7 jobs: JS & TS checks / apps/desktop / check:test:ui:shard-2of3, Python tests / Run tests slice 1/12, Python tests / Run tests slice 3/12, Python tests / Run tests slice 4/12, Python tests / Run tests slice 6/12, Python tests / Run tests slice 8/12, Python tests / Run tests slice 9/12

⚠️ Warnings

OSV vulnerability scan · View job

7 known vulnerabilities found in pinned dependencies.

How to fix:

Review the findings in the Security tab. Update the affected dependencies if a patched version is available.

@teknium1

Copy link
Copy Markdown
Collaborator Author

Split up per the incident's ownership. The Bot Mode fix here is superseded by #90732 (teknium1's canonical-chat adopt-before-mint — deeper root cause: the UNIQUE(title) squat + silent set_session_title drop + auto-titler rename loop, plus it keeps a history-bearing pin instead of adopting-by-title, and removes the Sessions browser). The ownership-file fix is #90497. The remaining updater-only piece (GUI-handoff backend reap — the 12-min hang) is now standalone in #90746, which touches no plugin.js/ownership code and can land independently.

@teknium1 teknium1 closed this Aug 20, 2026
@alt-glitch alt-glitch added type/bug Something isn't working comp/desktop Electron desktop app (apps/desktop/*) comp/cli CLI entry point, hermes_cli/, setup wizard area/install-update Installer, updater, packaging, wheels, doctor area/sessions Session lifecycle, resume, persistence, history P2 Medium — degraded but workaround exists sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades labels Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/install-update Installer, updater, packaging, wheels, doctor area/sessions Session lifecycle, resume, persistence, history comp/cli CLI entry point, hermes_cli/, setup wizard comp/desktop Electron desktop app (apps/desktop/*) P2 Medium — degraded but workaround exists sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-platform-windows Sweeper risk: may break or behave differently on native Windows sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants