From 4aa8f71380568683a5d246f6cf9ef3c5c2d199f8 Mon Sep 17 00:00:00 2001 From: AlchemistChaos Date: Fri, 5 Jun 2026 22:01:57 +0100 Subject: [PATCH 1/2] =?UTF-8?q?docs(desktop):=20document=20sleep/wake=20"S?= =?UTF-8?q?tarting=20Hermes=E2=80=A6"=20composer=20lockout?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Detailed pre-fix investigation of a bug where, after the Mac sleeps and wakes, the desktop chat composer stays disabled on the "Starting Hermes…" placeholder and never recovers — only a full app quit + reopen fixes it. Root cause: in remote / global-remote mode, startHermes() (main.cjs:4322) returns a cached connectionPromise with no liveness check, and the cache is only invalidated by the local backend child's 'exit'/'error' handlers. A remote primary spawns no child process (main.cjs:4328-4348), so the cached descriptor is never invalidated for the life of the main process. After sleep the renderer's (sound) reconnect loop keeps re-dialing the same dead remote endpoint forever; a relaunch works only because it resets the module-level connectionPromise. The renderer reconnect loop, gateway state machine, and the exact "connecting"-pinned placeholder logic are all traced with file:line evidence. Root cause confirmed by three independent analyses (0.95–0.98). The fix lands in a separate commit so the diagnosis can be reviewed on its own. Co-Authored-By: Claude Opus 4.8 (1M context) --- ...ktop-sleep-wake-reconnect-stale-backend.md | 231 ++++++++++++++++++ 1 file changed, 231 insertions(+) create mode 100644 docs/bugs/desktop-sleep-wake-reconnect-stale-backend.md diff --git a/docs/bugs/desktop-sleep-wake-reconnect-stale-backend.md b/docs/bugs/desktop-sleep-wake-reconnect-stale-backend.md new file mode 100644 index 000000000000..3adee3c14db3 --- /dev/null +++ b/docs/bugs/desktop-sleep-wake-reconnect-stale-backend.md @@ -0,0 +1,231 @@ +# Bug: Desktop composer locks on "Starting Hermes…" after sleep/wake (stale cached backend) + +- **Status:** Diagnosed — fix on branch `fix/desktop-sleep-wake-reconnect-stale-backend` +- **Severity:** High (user is fully locked out of an active session; only an app restart recovers) +- **Component:** `apps/desktop` (Electron main + renderer gateway boot/reconnect) +- **Primarily affects:** Remote / `global-remote` mode backends +- **Workaround:** Quit and reopen the desktop app + +> This document is the pre-fix write-up for review. The fix lands in a **separate +> commit** so the diagnosis can be reviewed independently of the change. + +--- + +## 1. User-visible symptom + +After the machine sleeps (lid closed / idle) and is woken later, returning to an +open chat in Hermes Desktop leaves the message composer **disabled**, showing the +placeholder **"Starting Hermes…"**. The user cannot type or send anything — they +appear "locked out" of a session that was working fine before sleep. + +The session itself is long-lived (the reporter's status bar showed +`Session 2:23:38`), and **quitting and reopening the app restores chat +immediately.** + +Reporter's words: + +> "after i leave a chat for a while in hermes desktop app… and come back to it… +> i can't seem to continue chatting? i seem to be locked out?" +> "i think i put my laptop to sleep and came back to it" +> "btw it works after i quit and re-open hermes desktop app" + +This is a **legitimate bug**, not a usage error. A session is supposed to survive +sleep/wake and reconnect transparently. + +## 2. Reproduction + +1. Launch Hermes Desktop connected to a **remote** gateway (`global-remote` mode, + or any `mode: 'remote'` connection). +2. Open/continue a chat and confirm the composer is enabled. +3. Put the Mac to sleep (or let the network drop long enough that the remote WS is + torn down), then wake it some time later. +4. Observe: the composer stays disabled on **"Starting Hermes…"** and never + recovers, no matter how long you wait. +5. Quit the app and reopen it → chat works again. + +## 3. The exact stuck state (why the placeholder text matters) + +The placeholder is a precise signal of the internal gateway state. + +`apps/desktop/src/app/chat/composer/index.tsx:207` + +```ts +const placeholder = disabled + ? gatewayState === 'closed' || gatewayState === 'error' + ? t.composer.placeholderReconnecting // "Reconnecting to Hermes…" + : t.composer.placeholderStarting // "Starting Hermes..." + : restingPlaceholder +``` + +`apps/desktop/src/app/chat/index.tsx:184,359` + +```ts +const gatewayOpen = gatewayState === 'open' +// … + +``` + +So: + +- `disabled` ⟺ `gatewayState !== 'open'`. +- The placeholder is **"Starting Hermes…"** specifically when `disabled` **and** + `gatewayState ∉ {closed, error}` — i.e. the gateway is pinned in **`connecting`** + (or `idle`), **not** cleanly closed. + +If the socket had merely closed, we'd see **"Reconnecting to Hermes…"**. We see +"Starting Hermes…", which means the renderer is perpetually mid-connect against a +backend it can never reach. + +## 4. Root cause + +The renderer's reconnect loop is **sound** — it is *not* where the lockout +originates. The lockout comes from the Electron main process handing the renderer +a **stale, cached connection descriptor** that points at a backend which is no +longer reachable, and never re-validating or rebuilding it for the life of the +process. + +### 4a. The renderer reconnect loop is correct (ruled out as the cause) + +`apps/desktop/src/app/gateway/hooks/use-gateway-boot.ts` + +- On a post-boot `closed`/`error` it schedules a reconnect with exponential + backoff (1s → 2s → 4s → 8s → 15s cap, `:166-178`, `:212-226`). +- `attemptReconnect` (`:115-164`) is serialized by a single `reconnecting` latch + that is **always** reset in a `finally` block (`:157-163`) — it cannot leak + `true` forever. +- Wake signals (`powerMonitor` resume, `online`, `visibilitychange`) nudge an + immediate reconnect (`:230-243`). + +`apps/shared/src/json-rpc-gateway.ts` + +- `connect()` carries a **15s connect timeout** (`DEFAULT_CONNECT_TIMEOUT_MS`, + `:62-65`) that forces a hung `connecting` → `error` (`:161-182`) and drops the + half-open socket, so a zombie `connecting` cannot persist on its own. + +Conclusion: the renderer keeps retrying forever and would recover the instant +`getConnection()` returned a reachable descriptor. It never does. + +### 4b. The main process caches the connection and never invalidates it (the bug) + +The reconnect path is: + +`getConnection(profile)` (preload `:4`) → IPC `hermes:connection` (`main.cjs:4607`) +→ `ensureBackend(profile)` (`main.cjs:4135`) → `startHermes()` (`main.cjs:4312`). + +`startHermes()` is a **pure cache hit with no liveness check**: + +`apps/desktop/electron/main.cjs:4322` + +```js +if (connectionPromise) return connectionPromise +``` + +`connectionPromise` is only ever cleared by: + +- the local backend child's `'error'`/`'exit'` handlers + (`main.cjs:4400-4415`, `4416-4438` → null `connectionPromise`), or +- `startHermes()`'s own boot `catch` (`:4462-4475`), or +- `resetHermesConnection()` (`:4084-4093`), which today is only called on a + **profile switch / connection-config change**, never during a normal reconnect. + +### 4c. Remote mode has no child process, so the cache is never cleared + +For a **remote** primary, `startHermes()` resolves via `resolveRemoteBackend()` +and returns **without spawning any child** — `hermesProcess` stays `null`: + +`apps/desktop/electron/main.cjs:4328-4348` + +```js +const remote = await resolveRemoteBackend(primaryProfileKey()) +if (remote) { + await waitForHermes(remote.baseUrl, remote.token) + // … returns { mode: 'remote', baseUrl, token, wsUrl, … } + return { /* remote descriptor */ } +} +``` + +Because there is no child process, the `'error'`/`'exit'` handlers that would +normally null `connectionPromise` **never exist**. The resolved remote descriptor +(`{ baseUrl, token, wsUrl }`, captured at boot) is therefore cached for the +**entire lifetime of the main process**. + +### 4d. Putting it together + +1. At boot the remote is reachable; `startHermes()` resolves and caches a good + `{ mode: 'remote', baseUrl, token, wsUrl }`. +2. Sleep tears down the live WebSocket (and, over a long sleep, the remote + endpoint the descriptor points at may move / restart / drop the session). +3. On wake the renderer's loop fires `getConnection()` → `startHermes()` → + returns the **same cached descriptor** (`:4322`). +4. The renderer re-mints a WS ticket and calls `gateway.connect()` against that + descriptor. It can't connect → `connecting` → (15s) `error` → backoff → + `connecting` → … **forever**. The composer stays disabled on "Starting Hermes…". +5. Nothing in the running app ever re-resolves or rebuilds the remote connection, + because only a child-process exit (which never happens for remote) clears the + cache. + +## 5. Why quit + reopen fixes it + +`connectionPromise` is a **module-level variable** (`main.cjs:476-477`). A full +quit tears down the entire Node/Electron process, so on relaunch +`connectionPromise === null` and `startHermes()` is forced to rebuild from +scratch — re-running `resolveRemoteBackend()` + `waitForHermes()` and producing a +fresh, reachable descriptor. This is the single state reset that an in-app +reconnect cannot trigger, and it is exactly why reopening works while waiting does +not. + +## 6. Scope of impact + +- **Remote / `global-remote` mode (primary):** broken as described — no self-heal + without an app restart. +- **Local mode:** mostly self-heals, because the local child's `'exit'` handler + nulls `connectionPromise` on a real crash, letting the next `getConnection()` + respawn. **One residual gap:** a local child that becomes *unresponsive but + never exits* (hung) would also hand back a cached-but-dead descriptor. This is + rarer and is treated as a follow-up (see §8). + +## 7. Verification + +Diagnosis was cross-checked by three independent investigations plus a fix +red-team: + +- **Remote-cache lens — confirms (0.98):** in remote mode `connectionPromise` is + never invalidated for the life of the process; `startHermes()` returns the + cached value unconditionally at `:4322`. +- **Renderer-loop lens — confirms (0.98):** the renderer reconnect loop is + provably sound (no latch leak, 15s timeout prevents stuck `connecting`, retries + indefinitely) → the lockout must be backend-side. +- **Relaunch-diff lens — confirms (0.95):** the only state a full relaunch resets + that an in-app reconnect cannot is the module-level `connectionPromise` cache. + +## 8. Planned fix (summary; lands in the next commit) + +**Revalidate-on-reconnect.** Add an opt-in liveness check so a reconnect cannot +re-dial a dead cached backend: + +- The renderer's **backoff-paced** reconnect (`use-gateway-boot.ts` only) calls + `getConnection(profile, { revalidate: true })`. +- On a cache hit with `revalidate`, `startHermes()` fast-probes the **public** + `/api/status` (token-free `fetchPublicJson`, ~2.5s). If it fails, drop the cache + via `resetHermesConnection()` and rebuild, so the renderer's existing loop gets + a fresh, reachable descriptor — no app restart required. + +Red-team-driven guard rails (to avoid regressions): + +- **Do not** add `revalidate` to `use-gateway-request.ts` (it fires on any + transient request blip and could needlessly tear down a healthy backend). +- **Only tear down `mode === 'remote'`** connections; local backends self-heal via + the child `exit` handler, so a probe miss there is treated as "WS not reattached + yet", not "backend dead". +- **Require 2 consecutive probe failures** before rebuild, so a single + captive-portal / VPN-re-establishing blip on wake doesn't trigger a respawn the + backoff loop would have ridden out. +- Steady-state and cold boot stay `revalidate`-off → **zero added latency** on the + happy paths. + +### Out of scope / follow-ups + +- Pool (non-primary, multi-profile) backends are not revalidated in this change. +- The local "alive-but-wedged, never exits" case (§6). +- An optional renderer watchdog for a zombie `connecting` state (the loop is + already proven sound, so this is belt-and-suspenders). From 54c4f15bff8297cbf98a3638c75e1a167b91ced1 Mon Sep 17 00:00:00 2001 From: AlchemistChaos Date: Fri, 5 Jun 2026 22:10:58 +0100 Subject: [PATCH 2/2] fix(desktop): recover chat after sleep/wake by revalidating the cached backend MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After the Mac slept and woke, the chat composer stayed disabled on the "Starting Hermes…" placeholder until the app was fully relaunched. Root cause (see docs/bugs/desktop-sleep-wake-reconnect-stale-backend.md): startHermes() returns a cached connectionPromise with no liveness check, and that cache is only cleared by the local backend child's 'exit'/'error' handlers. A remote / global-remote primary spawns no child process, so the cached descriptor is never invalidated for the life of the main process — the renderer's (sound) reconnect loop re-dials the same dead remote forever. Only a relaunch, which resets the module-level connectionPromise, recovered. Fix: revalidate-on-reconnect. The renderer's backoff-paced attemptReconnect now calls getConnection(profile, { revalidate: true }); on a cache hit startHermes() fast-probes the public /api/status (token-free, ~2.5s) and, if the backend is unreachable, tears the stale connection down via resetHermesConnection() and rebuilds — so the existing reconnect loop gets a fresh, reachable descriptor with no app restart. Recovery lands within a couple of backoff ticks (typically seconds; up to ~35s only if connect attempts hit their 15s timeout). Hardening (per fix red-team + code review): - revalidate is opt-in and ONLY wired into use-gateway-boot's backoff-paced reconnect — not use-gateway-request, which fires on any transient blip and could needlessly SIGTERM a healthy local child. - Only mode==='remote' connections are torn down; local backends self-heal via the child 'exit' handler, so a probe miss there is treated as "WS not reattached yet", not "backend dead". - A teardown requires 2 consecutive probe failures *within one reconnect episode* (time-windowed streak), so a single captive-portal / VPN-on-wake blip doesn't trigger a respawn, and a stale miss from an earlier, since-recovered episode can't pre-load the counter. - Concurrency: after the probe we re-check connectionPromise===cached with no intervening await (resetHermesConnection is synchronous). If a peer rebuilt we return their fresh connection; if a backend 'exit' or a rejected cache nulled it, we fall through and build fresh instead of returning null. - Steady-state and cold boot stay revalidate-off → zero added latency. - The renderer dismisses the boot-progress overlay on the post-boot 'open' transition, so an in-place rebuild (which re-drives boot progress) can't leave the overlay stuck at ~94%. Liveness/decision/episode logic is extracted into pure, unit-tested helpers in hardening.cjs (probeBackendAlive, shouldRebuildStaleConnection, isFreshRevalidateEpisode) since main.cjs can't be loaded headlessly. New tests added to hardening.test.cjs; full electron/*.test.cjs suite passes (84/84). Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/desktop/electron/hardening.cjs | 83 +++++++++++- apps/desktop/electron/hardening.test.cjs | 121 +++++++++++++++++- apps/desktop/electron/main.cjs | 105 ++++++++++++++- apps/desktop/electron/preload.cjs | 2 +- .../src/app/gateway/hooks/use-gateway-boot.ts | 16 ++- apps/desktop/src/global.d.ts | 10 +- 6 files changed, 324 insertions(+), 13 deletions(-) diff --git a/apps/desktop/electron/hardening.cjs b/apps/desktop/electron/hardening.cjs index 4ffdea051b51..a58788231e47 100644 --- a/apps/desktop/electron/hardening.cjs +++ b/apps/desktop/electron/hardening.cjs @@ -21,6 +21,81 @@ function resolveTimeoutMs(timeoutMs, fallbackMs = DEFAULT_FETCH_TIMEOUT_MS) { return fallback } +// ── Sleep/wake reconnect: cached-connection liveness revalidation ──────────── +// After sleep/wake the renderer reconnects through getConnection(), which on the +// main side hands back a cached backend descriptor. A REMOTE primary spawns no +// child process, so the 'exit'/'error' handlers that would clear a dead cache +// never fire — without a liveness check the renderer re-dials the same +// unreachable remote forever and the composer stays stuck on "Starting Hermes…". +// These pure helpers back the revalidate-on-reconnect path in main.cjs; they live +// here so they can be unit-tested without loading Electron. + +// Consecutive failed probes required before tearing down and rebuilding a cached +// remote connection. A single miss is ignored so a transient post-wake blip +// (captive portal, VPN re-establishing) doesn't trigger a full respawn that the +// renderer's own backoff would have ridden out. +const REMOTE_REVALIDATE_FAILURE_THRESHOLD = 2 +// Fast single-shot liveness probe budget. A healthy loopback or remote answers +// /api/status well under this; long enough to tolerate a momentary stall. +const REVALIDATE_PROBE_TIMEOUT_MS = 2_500 +// Window within which probe failures count as one reconnect episode. Comfortably +// larger than the renderer's 15s backoff cap (so two genuinely-consecutive misses +// always fall inside it) yet short enough that a stale miss from a since-recovered +// episode expires before the next one. +const REVALIDATE_STREAK_WINDOW_MS = 60_000 + +// Probe whether a resolved backend connection descriptor is still reachable by +// hitting its PUBLIC /api/status endpoint. Deliberately host-liveness only — it +// does NOT validate auth (an expired OAuth session still returns 200; auth is +// handled separately by the ws-ticket mint). `fetchJson` is injected (the +// token-free fetchPublicJson from main.cjs) so this stays pure and testable. +// Never throws: returns false on any failure. +async function probeBackendAlive(conn, fetchJson, { timeoutMs = REVALIDATE_PROBE_TIMEOUT_MS } = {}) { + if (!conn || typeof conn.baseUrl !== 'string' || !conn.baseUrl || typeof fetchJson !== 'function') { + return false + } + + const base = conn.baseUrl.replace(/\/+$/, '') + + try { + await fetchJson(`${base}/api/status`, { timeoutMs }) + return true + } catch { + return false + } +} + +// Decide whether a cached connection that just failed a liveness probe should be +// torn down and rebuilt. Only REMOTE connections are eligible: a local backend +// self-heals via its child 'exit' handler, so a probe miss there means "the WS +// hasn't reattached yet", not "the backend is dead" — tearing it down would +// needlessly SIGTERM a healthy child. Requires a failure streak so a single +// transient miss can't trigger a respawn. +function shouldRebuildStaleConnection({ + alive, + mode, + failureStreak, + threshold = REMOTE_REVALIDATE_FAILURE_THRESHOLD +}) { + if (alive) { + return false + } + + return mode === 'remote' && Number(failureStreak) >= threshold +} + +// True when a probe failure is far enough from the previous one (or is the first +// ever) to count as a NEW reconnect episode rather than a continuation — so the +// caller resets its consecutive-failure streak and a stale miss left over from an +// earlier, since-recovered episode can't pre-load the "2 consecutive" counter. +function isFreshRevalidateEpisode(now, lastFailureAt, windowMs = REVALIDATE_STREAK_WINDOW_MS) { + if (!lastFailureAt) { + return true + } + + return Number(now) - Number(lastFailureAt) > windowMs +} + function encryptDesktopSecret(value, safeStorageApi) { const raw = String(value || '') @@ -176,9 +251,15 @@ async function resolveReadableFileForIpc(filePath, options = {}) { module.exports = { DATA_URL_READ_MAX_BYTES, DEFAULT_FETCH_TIMEOUT_MS, + REMOTE_REVALIDATE_FAILURE_THRESHOLD, + REVALIDATE_PROBE_TIMEOUT_MS, + REVALIDATE_STREAK_WINDOW_MS, TEXT_PREVIEW_SOURCE_MAX_BYTES, encryptDesktopSecret, + isFreshRevalidateEpisode, + probeBackendAlive, resolveReadableFileForIpc, resolveTimeoutMs, - sensitiveFileBlockReason + sensitiveFileBlockReason, + shouldRebuildStaleConnection } diff --git a/apps/desktop/electron/hardening.test.cjs b/apps/desktop/electron/hardening.test.cjs index 865da8fe797e..f03037b3e9d7 100644 --- a/apps/desktop/electron/hardening.test.cjs +++ b/apps/desktop/electron/hardening.test.cjs @@ -7,10 +7,16 @@ const { pathToFileURL } = require('node:url') const { DEFAULT_FETCH_TIMEOUT_MS, + REMOTE_REVALIDATE_FAILURE_THRESHOLD, + REVALIDATE_PROBE_TIMEOUT_MS, + REVALIDATE_STREAK_WINDOW_MS, encryptDesktopSecret, + isFreshRevalidateEpisode, + probeBackendAlive, resolveReadableFileForIpc, resolveTimeoutMs, - sensitiveFileBlockReason + sensitiveFileBlockReason, + shouldRebuildStaleConnection } = require('./hardening.cjs') test('resolveTimeoutMs falls back to defaults and accepts overrides', () => { @@ -114,3 +120,116 @@ test('resolveReadableFileForIpc validates existence type size and sensitivity', }) assert.equal(envTemplate.resolvedPath, envTemplatePath) }) + +// ── Sleep/wake reconnect: cached-connection liveness revalidation ──────────── + +test('probeBackendAlive returns true when /api/status responds', async () => { + const calls = [] + const fetchJson = async (url, opts) => { + calls.push({ url, opts }) + return { ok: true } + } + + const alive = await probeBackendAlive({ baseUrl: 'https://gw.example.com' }, fetchJson, { timeoutMs: 2500 }) + + assert.equal(alive, true) + assert.equal(calls.length, 1) + // Trailing slash on baseUrl must not double up before /api/status. + assert.equal(calls[0].url, 'https://gw.example.com/api/status') + assert.equal(calls[0].opts.timeoutMs, 2500) +}) + +test('probeBackendAlive normalizes a trailing slash on baseUrl', async () => { + let seen = null + const fetchJson = async url => { + seen = url + return null + } + + await probeBackendAlive({ baseUrl: 'http://127.0.0.1:8787/' }, fetchJson) + assert.equal(seen, 'http://127.0.0.1:8787/api/status') +}) + +test('probeBackendAlive returns false (never throws) when the probe rejects', async () => { + const fetchJson = async () => { + throw new Error('ECONNREFUSED') + } + + const alive = await probeBackendAlive({ baseUrl: 'https://gw.example.com' }, fetchJson) + assert.equal(alive, false) +}) + +test('probeBackendAlive returns false for a missing baseUrl or fetcher', async () => { + const ok = async () => ({}) + assert.equal(await probeBackendAlive(null, ok), false) + assert.equal(await probeBackendAlive({ baseUrl: '' }, ok), false) + assert.equal(await probeBackendAlive({ baseUrl: 'https://gw.example.com' }, undefined), false) +}) + +test('probeBackendAlive defaults the probe timeout to REVALIDATE_PROBE_TIMEOUT_MS', async () => { + let seenTimeout + const fetchJson = async (_url, opts) => { + seenTimeout = opts.timeoutMs + return {} + } + + await probeBackendAlive({ baseUrl: 'https://gw.example.com' }, fetchJson) + assert.equal(seenTimeout, REVALIDATE_PROBE_TIMEOUT_MS) +}) + +test('shouldRebuildStaleConnection only tears down a remote past the failure threshold', () => { + // A live backend is never rebuilt, regardless of streak. + assert.equal( + shouldRebuildStaleConnection({ alive: true, mode: 'remote', failureStreak: 99 }), + false + ) + + // A local backend is never rebuilt here — it self-heals via its child 'exit' + // handler, so a probe miss means "WS not reattached yet", not "backend dead". + assert.equal( + shouldRebuildStaleConnection({ alive: false, mode: 'local', failureStreak: 99 }), + false + ) + + // A single remote miss is ignored (rides out transient post-wake blips)… + assert.equal( + shouldRebuildStaleConnection({ alive: false, mode: 'remote', failureStreak: 1 }), + false + ) + + // …but the second consecutive miss crosses the threshold and rebuilds. + assert.equal( + shouldRebuildStaleConnection({ + alive: false, + mode: 'remote', + failureStreak: REMOTE_REVALIDATE_FAILURE_THRESHOLD + }), + true + ) + + // Unknown / undefined mode (e.g. a rejected cache) is not eligible. + assert.equal( + shouldRebuildStaleConnection({ alive: false, mode: undefined, failureStreak: 5 }), + false + ) +}) + +test('REMOTE_REVALIDATE_FAILURE_THRESHOLD requires at least two misses', () => { + assert.ok(REMOTE_REVALIDATE_FAILURE_THRESHOLD >= 2) +}) + +test('isFreshRevalidateEpisode scopes the failure streak to one reconnect episode', () => { + // First-ever failure (no prior timestamp) always starts a fresh episode. + assert.equal(isFreshRevalidateEpisode(1_000, 0), true) + + // A miss within the window continues the same episode (streak accumulates). + assert.equal(isFreshRevalidateEpisode(10_000, 10_000 - (REVALIDATE_STREAK_WINDOW_MS - 1)), false) + + // A miss past the window is a new episode (streak resets) — this is what stops + // a stale miss from an earlier, since-recovered episode pre-loading the counter. + assert.equal(isFreshRevalidateEpisode(10_000, 10_000 - (REVALIDATE_STREAK_WINDOW_MS + 1)), true) + + // The window comfortably exceeds the renderer's 15s backoff cap so two + // genuinely-consecutive backoff-paced misses always fall inside it. + assert.ok(REVALIDATE_STREAK_WINDOW_MS > 15_000) +}) diff --git a/apps/desktop/electron/main.cjs b/apps/desktop/electron/main.cjs index b42913093d7c..4e0b55266365 100644 --- a/apps/desktop/electron/main.cjs +++ b/apps/desktop/electron/main.cjs @@ -47,8 +47,11 @@ const { DEFAULT_FETCH_TIMEOUT_MS, TEXT_PREVIEW_SOURCE_MAX_BYTES, encryptDesktopSecret: encryptDesktopSecretStrict, + isFreshRevalidateEpisode, + probeBackendAlive, resolveReadableFileForIpc, - resolveTimeoutMs + resolveTimeoutMs, + shouldRebuildStaleConnection } = require('./hardening.cjs') let nodePty = null @@ -475,6 +478,18 @@ function registerMediaProtocol() { let mainWindow = null let hermesProcess = null let connectionPromise = null +// Consecutive failed liveness probes of the cached PRIMARY connection during a +// revalidate-on-reconnect (sleep/wake). Crossing REMOTE_REVALIDATE_FAILURE_THRESHOLD +// tears the stale remote connection down and rebuilds; reset on any live probe, a +// fresh build, or when a new reconnect episode starts (see lastRevalidateFailureAt +// and isFreshRevalidateEpisode). See startHermes(). +let remoteRevalidateFailures = 0 +// Timestamp of the most recent failed liveness probe, used to scope the +// "consecutive failures" streak to a single reconnect episode: a miss far enough +// from the previous one resets the streak so a stale failure left over from an +// earlier, since-recovered episode can't pre-load the counter and force an early +// rebuild. +let lastRevalidateFailureAt = 0 // Additional per-profile backends, keyed by profile name. The PRIMARY backend // (the desktop's launch profile) stays managed by hermesProcess + // connectionPromise + startHermes(); this pool only holds EXTRA profile @@ -4132,11 +4147,11 @@ function primaryProfileKey() { // profile to startHermes() (the window backend: boot UI, bootstrap, remote // mode), and any OTHER profile to a lazily-spawned pool backend. An empty / // unknown profile resolves to the primary, so all legacy callers are unchanged. -async function ensureBackend(profile) { +async function ensureBackend(profile, options) { const key = profile && String(profile).trim() ? String(profile).trim() : primaryProfileKey() if (key === primaryProfileKey()) { - return startHermes() + return startHermes(options) } const existing = backendPool.get(key) @@ -4309,17 +4324,93 @@ function stopAllPoolBackends() { } } -async function startHermes() { +async function startHermes(options = {}) { // Latched-failure short-circuit: once bootstrap has failed in this // process, every subsequent startHermes() call re-throws the same error // without re-running install.ps1. This prevents the renderer's // ensureGatewayOpen retries (and any other getConnection callers) from // restarting a 5-10 minute install loop while the user is still reading - // the failure overlay. + // the failure overlay. Must stay ABOVE the revalidate block so a latched + // failure is never reinterpreted as a dead-cache rebuild. if (bootstrapFailure) { throw bootstrapFailure } - if (connectionPromise) return connectionPromise + + if (connectionPromise) { + // Steady-state and cold boot reuse the cached connection untouched — no + // added latency on the happy paths. + if (!options.revalidate) { + return connectionPromise + } + + // Reconnect-after-wake path (renderer's backoff-paced attemptReconnect): + // confirm the cached backend is actually reachable before the renderer + // re-dials it. A REMOTE primary has no child process, so the 'exit'/'error' + // handlers that would clear a dead connectionPromise never fire — without + // this probe the renderer re-dials the same unreachable remote forever and + // the composer stays stuck on "Starting Hermes…". Local backends self-heal + // via the child 'exit' handler and are deliberately never torn down here. + const cached = connectionPromise + let conn = null + + try { + conn = await cached + } catch { + // The cached boot rejected (its own catch nulls connectionPromise). Treat + // as unreachable; the fall-through below builds a fresh connection. + conn = null + } + + const alive = conn ? await probeBackendAlive(conn, fetchPublicJson) : false + + if (alive) { + remoteRevalidateFailures = 0 + + return cached + } + + // The cache vanished while we awaited/probed it — a concurrent backend 'exit' + // nulled it, or the cached boot itself rejected. There's nothing to + // revalidate: hand back a peer's fresh rebuild if one exists, otherwise fall + // through and build a new connection below. + if (connectionPromise !== cached) { + if (connectionPromise) { + return connectionPromise + } + } else { + // Cache still present but unreachable. Require consecutive misses within a + // single reconnect episode before tearing a REMOTE down (locals self-heal + // via their child 'exit' handler, so a probe miss there means "WS not + // reattached yet"). The time-window reset stops a stale miss from an + // earlier, since-recovered episode from pre-loading this episode's counter. + const now = Date.now() + + if (isFreshRevalidateEpisode(now, lastRevalidateFailureAt)) { + remoteRevalidateFailures = 0 + } + + lastRevalidateFailureAt = now + remoteRevalidateFailures += 1 + + if (!shouldRebuildStaleConnection({ alive, mode: conn?.mode, failureStreak: remoteRevalidateFailures })) { + // Not (yet) a confirmed-dead remote: leave the cache in place and let the + // renderer's backoff loop retry. The next consecutive miss crosses the + // threshold and rebuilds; a local backend never rebuilds here at all. + return cached + } + + // resetHermesConnection is synchronous (nulls connectionPromise + SIGTERMs + // any local child); we re-checked connectionPromise === cached with no + // intervening await above, so this can't kill a freshly-rebuilt backend. + rememberLog('Cached Hermes backend failed liveness revalidation; rebuilding connection.') + resetHermesConnection() + } + } + + // Any path that reaches a fresh build establishes a new connection, so the + // stale-probe streak no longer applies (covers the revalidate teardown above, + // a peer's concurrent teardown, and a rebuild from a self-nulled/rejected cache). + remoteRevalidateFailures = 0 connectionPromise = (async () => { await advanceBootProgress('backend.resolve', 'Resolving Hermes backend', 8) @@ -4604,7 +4695,7 @@ function createWindow() { }) } -ipcMain.handle('hermes:connection', async (_event, profile) => ensureBackend(profile)) +ipcMain.handle('hermes:connection', async (_event, profile, options) => ensureBackend(profile, options)) ipcMain.handle('hermes:backend:touch', async (_event, profile) => { touchPoolBackend(profile) return { ok: true } diff --git a/apps/desktop/electron/preload.cjs b/apps/desktop/electron/preload.cjs index e1479b1ba99b..38620d698572 100644 --- a/apps/desktop/electron/preload.cjs +++ b/apps/desktop/electron/preload.cjs @@ -1,7 +1,7 @@ const { contextBridge, ipcRenderer, webUtils } = require('electron') contextBridge.exposeInMainWorld('hermesDesktop', { - getConnection: profile => ipcRenderer.invoke('hermes:connection', profile), + getConnection: (profile, options) => ipcRenderer.invoke('hermes:connection', profile, options), touchBackend: profile => ipcRenderer.invoke('hermes:backend:touch', profile), getGatewayWsUrl: profile => ipcRenderer.invoke('hermes:gateway:ws-url', profile), getBootProgress: () => ipcRenderer.invoke('hermes:boot-progress:get'), diff --git a/apps/desktop/src/app/gateway/hooks/use-gateway-boot.ts b/apps/desktop/src/app/gateway/hooks/use-gateway-boot.ts index 7bd0ba34a735..595371ab7b88 100644 --- a/apps/desktop/src/app/gateway/hooks/use-gateway-boot.ts +++ b/apps/desktop/src/app/gateway/hooks/use-gateway-boot.ts @@ -120,7 +120,12 @@ export function useGatewayBoot({ reconnecting = true try { - const conn = await desktop.getConnection($activeGatewayProfile.get()) + // revalidate: the main process liveness-probes the cached backend before + // returning it and rebuilds a dead one (e.g. a remote backend that became + // unreachable across a sleep/wake — it has no child process whose 'exit' + // would otherwise clear the stale cache). Without this the renderer would + // re-dial the same dead endpoint forever and stay on "Starting Hermes…". + const conn = await desktop.getConnection($activeGatewayProfile.get(), { revalidate: true }) if (cancelled) { return @@ -218,6 +223,15 @@ export function useGatewayBoot({ reconnectAttempt = 0 reauthNotified = false clearReconnectTimer() + // A revalidate-driven reconnect can rebuild the backend in place (when + // getConnection found the cached one dead), which re-drives the boot + // progress overlay via resetHermesConnection/advanceBootProgress. Unlike + // the initial boot, nothing calls completeDesktopBoot() afterwards, so + // dismiss it here once we're open again — otherwise the overlay would + // stick at ~94%. No-op on a normal (non-rebuild) reconnect. + if (bootCompleted) { + completeDesktopBoot() + } } else if (bootCompleted && (st === 'closed' || st === 'error')) { // The socket dropped after a healthy boot (typically sleep/wake). Try // to bring it back instead of leaving the composer stuck disabled. diff --git a/apps/desktop/src/global.d.ts b/apps/desktop/src/global.d.ts index 43718037a1d1..6de2dc190840 100644 --- a/apps/desktop/src/global.d.ts +++ b/apps/desktop/src/global.d.ts @@ -5,8 +5,14 @@ declare global { hermesDesktop: { // Resolve a backend connection. Omit `profile` (or pass the primary) for // the window's backend; pass a named profile to lazily spawn/reuse that - // profile's backend from the pool. - getConnection: (profile?: string | null) => Promise + // profile's backend from the pool. Pass `{ revalidate: true }` on a + // reconnect (sleep/wake) so the main process liveness-probes the cached + // backend and rebuilds it if it has gone unreachable, instead of handing + // back a dead descriptor the renderer would re-dial forever. + getConnection: ( + profile?: string | null, + options?: { revalidate?: boolean } + ) => Promise // Keepalive: mark a pool profile backend as recently used so the idle // reaper spares it while its chat is active. touchBackend: (profile?: string | null) => Promise<{ ok: boolean }>