Skip to content

fix(desktop): raise remote liveness probe timeout from 2.5s to 10s - #49841

Open
GratefulDave wants to merge 1 commit into
NousResearch:mainfrom
GratefulDave:fix/desktop-remote-liveness-probe-timeout
Open

fix(desktop): raise remote liveness probe timeout from 2.5s to 10s#49841
GratefulDave wants to merge 1 commit into
NousResearch:mainfrom
GratefulDave:fix/desktop-remote-liveness-probe-timeout

Conversation

@GratefulDave

Copy link
Copy Markdown
Contributor

Summary

Fixes the connect → immediately-stale → reconnect loop reported in #49787 for remote backends reached via high-latency paths (Tailscale relay, cross-region VPN).

Root cause

hermes:connection:revalidate probes /api/status with a hardcoded 2_500ms timeout:

// main.cjs:5309 (before)
await fetchPublicJson(`${base}/api/status`, { timeoutMs: 2_500 })

This fires on every sleep/wake (onPowerResume), network online event, and visibilitychange. On a Tailscale-connected remote, relay latency can intermittently exceed 2.5s — causing the probe to time out and drop the cached connection as stale even when the backend is perfectly healthy. The result is an infinite loop:

Remote Hermes backend is ready
→ Cached remote Hermes backend failed liveness probe; dropping stale connection.
→ startHermes() rebuilds from scratch (boot progress overlay reappears at ~24%)
→ next wake/focus event fires revalidateConnection() again
→ same 2.5s race → drop again → loop

This is distinct from the bootstrap marker issue also reported in #49787 (setup screen on every launch), which has a separate fix.

Fix

Raise the revalidate timeout to 10_000ms, matching the timeout already used for the connection test (fetchPublicJson at line 4437):

// main.cjs:5309 (after)
await fetchPublicJson(`${base}/api/status`, { timeoutMs: 10_000 })

A genuinely dead backend still fails within 10s. A live backend over a high-latency relay no longer gets false-evicted.

Verification

  • Affected setup: Mac M5, macOS Tahoe 26.5.1, Hermes Desktop v0.17.0, remote backend on Mac mini via Tailscale tunnel
  • Tailscale round-trip: ~223ms average with occasional spikes above 2.5s
  • GET /api/status returns HTTP 200 with valid JSON when probed directly; Let's Encrypt cert, no TLS issues
  • With timeoutMs: 2_500: desktop.log shows repeated Cached remote Hermes backend failed liveness probe within seconds of each Remote Hermes backend is ready — confirmed via 8–10 loop iterations per session
  • With timeoutMs: 10_000: probe succeeds reliably; loop stops

Related

…ousResearch#49787)

The hermes:connection:revalidate IPC handler probes /api/status with a
hardcoded 2500ms timeout. This is too tight for remote backends reached
via high-latency paths (e.g. Tailscale relay): the probe races to
timeout even while the backend is healthy, causing:

  Remote Hermes backend is ready
  → Cached remote Hermes backend failed liveness probe; dropping stale connection.
  → startHermes() rebuilds the connection from scratch
  → immediately revalidates on the next wake/focus event
  → loop

The probe fires on every sleep/wake (onPowerResume), network online
event, and window visibilitychange — so on a Tailscale-connected remote
the loop runs continuously, showing the boot progress overlay repeatedly
and never settling.

Fix: raise the revalidate timeout to 10_000ms, matching the timeout
already used for the connection test at line 4437. A genuinely dead
backend still fails within 10s; a live backend over a high-latency
relay no longer gets false-evicted.

Verified: Tailscale round-trip on the affected setup is ~223ms average
with occasional spikes. 10s gives ample headroom without delaying
detection of a truly unreachable backend.

Fixes NousResearch#49787
@alt-glitch alt-glitch added type/bug Something isn't working comp/tui Terminal UI (ui-tui/ + tui_gateway/) comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists comp/desktop Electron desktop app (apps/desktop/*) and removed comp/tui Terminal UI (ui-tui/ + tui_gateway/) labels Jun 20, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for isolating the remote liveness failure mode. The current implementation still has the reported behavior: apps/desktop/electron/main.ts:7405 probes /api/status with timeoutMs: 2_500, and its catch resets the remote connection at apps/desktop/electron/main.ts:7408-7415.

Problems

  • The diff modifies apps/desktop/electron/main.cjs, but current main renamed that file to apps/desktop/electron/main.ts in 39d09453f95e8aefc0c97e5d9b30ff341cae9ed8. The live handler is therefore unchanged by this PR as submitted.
  • There is no focused regression test for hermes:connection:revalidate; repository searches find the production handler and stale-probe log, but no corresponding test.

Suggested changes

  • Apply the 10_000 timeout at apps/desktop/electron/main.ts:7405.
  • Add a focused test for a delayed-but-successful remote status probe and for a genuinely failed probe resetting the cached remote connection.

Automated hermes-sweeper review.

const base = conn.baseUrl.replace(/\/+$/, '')
try {
await fetchPublicJson(`${base}/api/status`, { timeoutMs: 2_500 })
await fetchPublicJson(`${base}/api/status`, { timeoutMs: 10_000 })

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current main renamed this source file to apps/desktop/electron/main.ts in 39d09453f95e8aefc0c97e5d9b30ff341cae9ed8; the live handler is now at main.ts:7405 and still uses 2_500. Port this change to that handler so the fix reaches current main.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/desktop Electron desktop app (apps/desktop/*) comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Desktop app: reinstall/setup prompt on every launch despite valid config

4 participants