Skip to content

fix(desktop): survive and resume turns interrupted mid-stream - #71184

Merged
OutThisLife merged 4 commits into
mainfrom
bb/desktop-stream-resume
Jul 25, 2026
Merged

fix(desktop): survive and resume turns interrupted mid-stream#71184
OutThisLife merged 4 commits into
mainfrom
bb/desktop-stream-resume

Conversation

@OutThisLife

@OutThisLife OutThisLife commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

A turn interrupted by a disconnect, a renderer crash, or a full app/machine death used to leave the desktop with nothing to show for it: the gateway cleared its in-flight state while emitting terminal frames the detached client never received, nothing on the renderer side survived a process death, and the interrupted turn never finished. This closes that hole from both ends — the turn's progress survives, a failure renders as a failure, and a fresh interruption picks itself back up.

  • The gateway keeps failed turns replayable. A turn that ends in error retains a compact inflight snapshot (user prompt, partial text, error, recoverable) until the next turn starts, on all three loss paths: returned-error results, turn exceptions (which now close with the same status:"error" message.complete frame as returned errors, instead of a bare error event), and agent-init failures.
  • The desktop journals the running turn's visible tail — tool calls included — to localStorage, throttled off the delta-flush hot path, bounded, and cleared the moment the turn settles. Session resume folds it back onto the restored transcript. When the backend also has a live text-only snapshot for the same turn, the journal overlays its richer structure onto that row (longer text wins, base row id kept so live deltas keep landing) instead of treating the turn as caught up.
  • Terminal error frames render as failures. The frame's structured error/partial fields drive the failed-bubble state instead of a text regex that missed the gateway's Error: <detail> texts. Partial output stays visible, and a retained failure replayed through resume renders as failed rather than as a healthy reply.
  • Crash-interrupted turns continue themselves on resume. A turn writes a durable marker when it starts running and retires it the instant its outcome reaches the client — success, handled error, and interrupt all retire it — so a surviving marker is positive proof the process died mid-turn. Retirement happens at the terminal frame rather than at the end of the turn thread, because post-turn work (titles, memory sync, goal hooks) keeps running for a second or more after the reply lands, and quitting inside that window would otherwise look like a crash and re-run a finished turn. session.resume re-submits a fresh interruption with the original prompt in a recovery note, streamed live to the client that just resumed and shown as a compact "resumed interrupted turn" event row. Stale interruptions just leave the recovered partial in place for a manual continue. Config lives under desktop.auto_continue (enabled, freshness_minutes 15, max_attempts 2) — the same freshness window plus crash-loop breaker the messaging gateway's restart auto-resume already uses.

Supersedes #44339 and #46837; thanks @OmarB97 and @Reza2kn, both credited via Co-authored-by. Rebuilding onto current main also fixes the review defects on each: the backend-first merge ordering that dropped locally recorded tool progress, and the returned-error path that was never routed through retention.

Test plan

  • scripts/run_tests.sh tests/tui_gateway/ tests/test_tui_gateway_server.py tests/test_tui_gateway_queue_on_busy.py tests/test_tui_gateway_ws.py tests/hermes_cli/test_config*.py — 1154 passed, including 8 new retention tests and 19 new auto-continue tests
  • apps/desktop: full vitest suite — 2873 passed, including journal persist/merge, terminal error frames, and the auto-continue timeline row
  • npx playwright test e2e/image-attachment-resume.spec.ts — the real-gateway seeding spec, which caught the marker-retirement window above
  • npm run typecheck across all three desktop tsconfigs, plus eslint and ruff clean

@github-actions

github-actions Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

૮ >ﻌ< ა ci review

ran on 082bd17

all good!

@OutThisLife OutThisLife changed the title fix(desktop): crash/disconnect turn recovery — retained failed turns + in-flight journal fix(desktop): crash/disconnect turn recovery — retention, journal, auto-continue Jul 25, 2026
@OutThisLife
OutThisLife force-pushed the bb/desktop-stream-resume branch from cca7800 to 83ca6d2 Compare July 25, 2026 04:16
@OutThisLife OutThisLife changed the title fix(desktop): crash/disconnect turn recovery — retention, journal, auto-continue fix(desktop): survive and resume turns interrupted mid-stream Jul 25, 2026
@alt-glitch alt-glitch added type/bug Something isn't working P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/desktop Electron desktop app (apps/desktop/*) comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins platform/telegram Telegram bot adapter area/sessions Session lifecycle, resume, persistence, history sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state needs-decision Awaiting maintainer decision before any implementation labels Jul 25, 2026
OutThisLife and others added 4 commits July 24, 2026 23:31
A turn that ended in error cleared inflight_turn and emitted its terminal
frame in the same breath. If the client was disconnected during that window
(the exact case for a failure like a network drop), the frame went to the
detached drop-transport and the in-memory state was already gone — the
desktop reconnected to a session with no trace of the failure.

Failed turns now retain a compact error snapshot (user prompt, partial
assistant text, error, recoverable) that session.resume's inflight payload
carries to a reconnecting client. Covers all three loss sites: the
returned-error result path, the turn exception path (which now closes with
the same status:"error" message.complete frame shape instead of a bare
error event), and agent-init failure. The snapshot lives until the next
turn starts or the session closes; _run_prompt_submit replaces a retained
error leftover instead of appending onto it.

Co-authored-by: Reza Sayar <rsayar@uvic.ca>
The renderer's session-state cache is memory-only and the backend's
inflight snapshot dies with the backend process, so nothing survived a
full app or machine death mid-turn: reopening the session showed the
transcript up to the last committed turn and silently dropped everything
the crashed turn had streamed.

While a turn runs, the visible tail (user prompt + streamed assistant
rows, tool calls included) is now journaled to localStorage — throttled
off the delta-flush hot path, bounded (24 entries / 7 days), cleared the
moment the turn settles. Session resume folds the journaled tail back
onto the restored transcript. When the backend also has a live text-only
inflight projection for the same turn, the journal overlays its richer
structure onto that row (longer text wins, base row id kept so live
deltas keep landing) instead of treating it as caught up — the ordering
defect that dropped locally recorded tool progress in the original PR.

Co-authored-by: Omar Baradei <omar@kostudios.io>
message.complete frames with status "error" were detected only by a text
regex heuristic, which misses the gateway's "Error: <detail>" texts and
partial-text failures — a failed turn rendered as a healthy reply. The
structured error/partial fields now drive the failure state: the bubble is
marked failed from the frame's error field, and a partial failure keeps
its streamed text visible instead of stripping it. session.resume's
inflight projection likewise carries a retained failure's error onto the
projected assistant row, so a failed turn recovered after a disconnect
renders as failed rather than as a healthy partial answer.

Co-authored-by: Reza Sayar <rsayar@uvic.ca>
Mid-turn progress lives only in process memory — the agent flushes to
SQLite at turn end — so an app/backend/machine death mid-turn lost the
turn entirely: reopening the session showed the recovered partial, but
the work never finished and the prompt itself survived nowhere durable.

Turns now write a durable marker (bounded per-profile sidecar) when they
start running and clear it when they conclude; success, handled error,
and interrupt all clear it, so a surviving marker is positive proof of a
process death. session.resume reads the marker: a fresh interruption
(desktop.auto_continue.freshness_minutes, default 15) is re-submitted
automatically as a continuation turn carrying the original prompt in an
interruption note, streams live to the client that just resumed, and
renders as a "resumed interrupted turn" event row. Stale markers are
cleared and the recovered partial speaks for itself; a turn that keeps
crashing stops auto-continuing after max_attempts (default 2) — the same
freshness + crash-loop-breaker posture as the messaging gateway's
restart auto-resume.
@OutThisLife
OutThisLife force-pushed the bb/desktop-stream-resume branch from 83ca6d2 to 082bd17 Compare July 25, 2026 04:32
@OutThisLife
OutThisLife enabled auto-merge July 25, 2026 04:35
@OutThisLife
OutThisLife merged commit 62e0722 into main Jul 25, 2026
47 checks passed
@OutThisLife
OutThisLife deleted the bb/desktop-stream-resume branch July 25, 2026 04:42
teknium1 added a commit that referenced this pull request Jul 25, 2026
…cing test

#71184 (stream resume) upgraded agent-build-failure delivery from a bare
'error' event to a terminal message.complete frame (status=error,
recoverable) so failed turns replay on resume. The #71140 test pinned the
old event shape and broke on main where the two merged within the hour.
Contract unchanged: build failure must reach the client visibly.
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…ream-resume

fix(desktop): survive and resume turns interrupted mid-stream
randlee pushed a commit to randlee/hermes-agent that referenced this pull request Aug 11, 2026
…ailure surfacing test

NousResearch#71184 (stream resume) upgraded agent-build-failure delivery from a bare
'error' event to a terminal message.complete frame (status=error,
recoverable) so failed turns replay on resume. The NousResearch#71140 test pinned the
old event shape and broke on main where the two merged within the hour.
Contract unchanged: build failure must reach the client visibly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/sessions Session lifecycle, resume, persistence, history comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/desktop Electron desktop app (apps/desktop/*) comp/gateway Gateway runner, session dispatch, delivery comp/plugins Plugin system and bundled plugins needs-decision Awaiting maintainer decision before any implementation P3 Low — cosmetic, nice to have platform/telegram Telegram bot adapter sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages sweeper:risk-session-state Sweeper risk: may lose/corrupt/mis-associate session or context state type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants