Live Stream: Add compression recovery action

[franksong2702]

Thinking Path

Long-running sessions can end in compression_exhausted when automatic compression cannot safely create enough model-facing room. A bare follow-up like continue usually replays the same exhausted context, so WebUI should make the terminal state durable and guide the user into a focused linked continuation instead of silently trying the same turn again.

This PR implements the first #4685 recovery slice: persist terminal recovery metadata, expose one recommended action, make that action retry-safe, and block generic continuation attempts while preserving substantive user prompts.

Refs #4685.

What Changed

• Added api/compression_recovery.py for durable compression_exhausted recovery payloads and generic continuation-intent matching.
• Stamps compression_recovery and recommended_recovery_action when streaming produces a compression-exhausted terminal error.
• Adds POST /api/session/compression-recovery/start to create a linked focused continuation session with the same workspace/model/profile/project/toolset/worktree lane and an empty model-facing transcript.
• Marks focused recovery child sessions with their source/action and reuses the existing child on repeated start requests, including after the in-memory session cache is cleared.
• Blocks generic continue / go on / 继续 prompts on exhausted sessions with a 409 recovery response, while allowing substantive prompts and clearing session-level recovery only when a new turn actually starts.
• Renders a compact recovery card in the assistant error message with a Start focused continuation action and frontend interception for generic continuation attempts.
• Surfaces a malformed successful recovery response as an error toast instead of silently re-enabling the recovery button.
• Documents the user-facing troubleshooting flow for Context compression exhausted and updates the session-SSE RFC source anchors that shifted with the new route.

Why It Matters

Users should not have to guess what to do after a long-running session exhausts compression. The exhausted session remains terminal and inspectable, while the recommended path creates a focused linked continuation that avoids replaying the oversized context tail.

Repeated clicks or multiple open tabs now converge on the same focused continuation instead of creating duplicate sibling sessions.

Contract Routing

• Contract family: Live-to-Final terminal/recovery states and WebUI run-state consistency around compression.
• Existing rule used: compression_exhausted is a specific terminal state, not normal completion, and automatic compression material should not become current user intent.
• Evidence: terminal recovery metadata is persisted in session metadata and the assistant error message; generic continuation is blocked on both client and server; focused continuation starts with empty messages/context_messages, keeps parent_session_id for lineage, and has a persisted child marker so the recovery action is idempotent.
• Contract Change: None. This implements the existing Track long-running context compression exhaustion and recovery UX #4685 follow-up direction without changing the Live-to-Final product contract.

UI Evidence

• Before: a compression-exhausted error left the user with ordinary composer behavior; typing a bare continuation could attempt to reuse the exhausted session context.
• After: the terminal error includes a .compression-recovery-card with Start focused continuation; generic continuation input scrolls/focuses that card and server-side fallback returns compression_recovery_required.
• After review hardening: a successful-but-malformed recovery response shows Compression recovery failed: Compression recovery did not return a session. instead of silently doing nothing.
• Automated UI wiring coverage: tests/test_compression_recovery_action.py verifies the card/action/intercept hooks and malformed-response guard are present, and node --check verifies the changed browser files parse.

Verification

• git diff --check
• python3 scripts/ruff_lint.py --diff origin/master — no new violations on added/modified lines
• node --check static/ui.js && node --check static/messages.js
• python3 -m py_compile api/compression_recovery.py api/models.py api/routes.py api/streaming.py api/session_recovery.py api/webui_session_db.py tests/test_compression_recovery_action.py tests/test_auto_compression_terminal_failure.py
• ./scripts/test.sh tests/test_compression_recovery_action.py -v — 9 passed
• ./scripts/test.sh tests/test_auto_compression_terminal_failure.py -v — 14 passed
• ./scripts/test.sh tests/test_issue4812_session_sse_contract_rfc.py -v — 33 passed
• ./scripts/test.sh tests/test_compression_recovery_action.py tests/test_auto_compression_terminal_failure.py tests/test_auto_compression_card.py tests/test_live_to_final_anchor_visible_order.py tests/test_turn_duration_display.py tests/test_webui_state_db_reconciliation.py tests/test_issue4812_session_sse_contract_rfc.py — 217 passed
• Claude Code CLI review, tool-less diff review, run three times. The first review found a real recovery-clear durability gap, which is fixed here. Greptile later raised idempotency and malformed-success-response gaps; both are fixed in 1e9d42c05. The final blocker-only review raised context_messages persistence, verified as already covered by Session.save() extra fields and the disk JSON test.

Risks / Follow-ups

• This does not solve every Track long-running context compression exhaustion and recovery UX #4685 roadmap item. It adds the first actionable recovery path for exhausted sessions.
• Browser screenshot/video is not attached; this state is terminal-error driven and is covered here by DOM/CSS wiring tests plus syntax checks.
• Future slices can add richer recovery options, but this PR intentionally exposes one backend-selected recommendation.

Release Note

Compression-exhausted sessions now show a focused continuation recovery action instead of encouraging bare continue retries that can replay exhausted context.

Model Used

OpenAI GPT-5 Codex for implementation and verification, plus Claude Code CLI and Greptile as independent diff reviewers.

[franksong2702]

CI triage update:

• The failed shard-0 jobs were from stale api/routes.py anchors in docs/rfcs/session-sse-contract-v1.md after the route file moved.
• The branch is now at dfc0686dbdb580f78480800ec52eb52223c43b94, which refreshes those RFC anchors.
• Local verification: ./scripts/test.sh tests/test_issue4812_session_sse_contract_rfc.py -> 33 passed.
• GitHub has started a fresh run for this head; browser-smoke and lint are already green, and the Python matrix is still in progress.

[greptile-apps]

Greptile Summary

This PR adds a durable recovery flow for compression_exhausted terminal sessions: a new api/compression_recovery.py module, a /api/session/compression-recovery/start endpoint, server-side generic-continuation blocking (409 on bare "continue" prompts), and a frontend recovery card with idempotent child-session creation. The stack is well-tested with 217 passing tests across seven test files.

• Backend: stamp_compression_exhausted_recovery persists recovery metadata when the streaming layer detects compression_exhausted; _handle_chat_start clears then optionally restores recovery around _start_run, and the new endpoint uses _COMPRESSION_RECOVERY_START_LOCK + find_compression_recovery_session to guarantee idempotent child creation even after cache eviction.
• Frontend: _activeCompressionRecoveryPayload reads session-level recovery first and only falls back to message scanning for legacy sessions (guarded by hasOwnProperty); shouldInterceptCompressionRecoveryContinuation blocks generic continuation client-side before sending, mirroring the server-side 409 guard.

Confidence Score: 5/5

Safe to merge — the recovery flow is well-scoped, idempotent, and thoroughly tested across seven test files.

The chat-start recovery-clear/restore logic handles the 501, 4xx, and exception paths correctly. The _COMPRESSION_RECOVERY_START_LOCK + find_compression_recovery_session design makes the recovery-start endpoint idempotent across double-clicks, tab duplicates, and cache evictions. The only gap found is that the disk-scan portion of find_compression_recovery_session lacks the same broad exception guard present in the memory-scan portion — a runtime error from _profiles_match would surface as a 500 rather than a graceful fallback, which is a minor hardening gap unlikely to trigger in practice.

api/models.py — disk scan in find_compression_recovery_session is the only spot that deserves a second look.

Important Files Changed

Filename	Overview
api/compression_recovery.py	New module providing recovery payload construction, generic-continuation intent matching, and session stamp/clear helpers; logic is clean and well-scoped.
api/models.py	Adds four new Session fields and find_compression_recovery_session; memory scan defensively wraps _compression_recovery_child_matches in try/except but the disk scan loop does not, leaving an unhandled exception path.
api/routes.py	Adds _handle_session_compression_recovery_start with _COMPRESSION_RECOVERY_START_LOCK for idempotency, and modifies _handle_chat_start to clear/restore recovery around _start_run with correct 501/4xx/exception restore paths.
api/streaming.py	Stamps compression recovery metadata on both inline-error and exception-path error messages when _err_type/exc_type is compression_exhausted; consistent with the two existing error paths.
static/ui.js	Adds recovery card rendering, hasOwnProperty-guarded payload lookup, generic-continuation intercept, and startCompressionRecovery with malformed-response guard; all user-controlled values passed through esc().
static/messages.js	Adds shouldInterceptCompressionRecoveryContinuation check in send() and persists _compressionRecovery from the SSE error payload onto the message object.
tests/test_compression_recovery_action.py	New test file with 9 tests covering intent matching, chat-start blocking/clearing/restoring, recovery-start idempotency, profile isolation, disk round-trip, and UI wiring assertions.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Browser
    participant routes as api/routes.py
    participant streaming as api/streaming.py
    participant recovery as api/compression_recovery.py
    participant models as api/models.py

    Browser->>routes: POST /api/session/chat (long run)
    routes->>streaming: _start_run(session)
    streaming-->>recovery: stamp_compression_exhausted_recovery(session)
    recovery-->>models: "session.compression_recovery = payload"
    streaming-->>Browser: SSE: compression_exhausted error + _compressionRecovery

    Note over Browser: Renders .compression-recovery-card

    Browser->>routes: POST /api/session/chat (bare continue)
    routes-->>Browser: 409 compression_recovery_required

    Browser->>routes: POST /api/session/compression-recovery/start
    routes->>models: find_compression_recovery_session(sid, action)
    models-->>routes: None (first call)
    routes->>models: "Session(..., context_messages=[])"
    models-->>routes: saved child session
    routes-->>Browser: "200 {session, source_session_id}"

    Browser->>routes: POST /api/session/compression-recovery/start (retry)
    routes->>models: find_compression_recovery_session(sid, action)
    models-->>routes: existing child session
    routes-->>Browser: "200 {session, message: Opened existing...}"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Browser
    participant routes as api/routes.py
    participant streaming as api/streaming.py
    participant recovery as api/compression_recovery.py
    participant models as api/models.py

    Browser->>routes: POST /api/session/chat (long run)
    routes->>streaming: _start_run(session)
    streaming-->>recovery: stamp_compression_exhausted_recovery(session)
    recovery-->>models: "session.compression_recovery = payload"
    streaming-->>Browser: SSE: compression_exhausted error + _compressionRecovery

    Note over Browser: Renders .compression-recovery-card

    Browser->>routes: POST /api/session/chat (bare continue)
    routes-->>Browser: 409 compression_recovery_required

    Browser->>routes: POST /api/session/compression-recovery/start
    routes->>models: find_compression_recovery_session(sid, action)
    models-->>routes: None (first call)
    routes->>models: "Session(..., context_messages=[])"
    models-->>routes: saved child session
    routes-->>Browser: "200 {session, source_session_id}"

    Browser->>routes: POST /api/session/compression-recovery/start (retry)
    routes->>models: find_compression_recovery_session(sid, action)
    models-->>routes: existing child session
    routes-->>Browser: "200 {session, message: Opened existing...}"

Loading

_{Reviews (26): Last reviewed commit: "Merge latest master into compression rec..." | Re-trigger Greptile}

[franksong2702]

Aftercare update for the latest Greptile findings:

• Added commit 1e9d42c05 to make POST /api/session/compression-recovery/start reuse an existing focused continuation instead of creating duplicate siblings on repeated calls.
• Added persisted recovery-child markers plus a route-level find/create lock; repeated calls also reuse the child after the in-memory session cache is cleared.
• Updated startCompressionRecovery() so a 200 response without session.session_id now surfaces an error toast instead of silently doing nothing.
• Refreshed the route anchors in docs/rfcs/session-sse-contract-v1.md after the route-file line shift.

Local verification:

• git diff --check
• python3 scripts/ruff_lint.py --diff origin/master
• node --check static/ui.js && node --check static/messages.js
• python3 -m py_compile api/compression_recovery.py api/models.py api/routes.py api/streaming.py api/session_recovery.py api/webui_session_db.py tests/test_compression_recovery_action.py tests/test_auto_compression_terminal_failure.py
• ./scripts/test.sh tests/test_compression_recovery_action.py tests/test_auto_compression_terminal_failure.py tests/test_auto_compression_card.py tests/test_live_to_final_anchor_visible_order.py tests/test_turn_duration_display.py tests/test_webui_state_db_reconciliation.py tests/test_issue4812_session_sse_contract_rfc.py — 217 passed

GitHub checks are running on the new head.

[cutter-sh]

🎬 Cutter preview — PR #5538

/s/:session_id — Compression-exhausted assistant messages now offer a one-click Start focused continuation recovery.

Attempt generic continuation on exhausted session — Compression recovery surfaces a Start focused continuation action when the run can't safely continue in place.

[franksong2702]

CI follow-up for the shard-4 branch metadata failure:

• Added commit f25571342 to keep the optional compression-recovery child markers in Session.compact() while moving them after the existing parent_session_id field.
• This restores the existing branch metadata contract test's source-order expectation without changing the compact payload behavior for recovery child sessions.

Local verification:

• ./scripts/test.sh tests/test_465_session_branching.py::test_session_compact_includes_parent -v — 1 passed
• ./scripts/test.sh tests/test_compression_recovery_action.py -v — 9 passed
• git diff --check
• python3 -m py_compile api/models.py
• node --check static/ui.js
• python3 scripts/ruff_lint.py --diff origin/master — no new violations on added/modified lines

GitHub checks restarted on f25571342ec6e499b753f44ccca09d18097374b7 and are currently pending.

[franksong2702]

Aftercare base refresh update:

• Merged latest master (b504fe305) into this PR branch with commit 8717dcc0 to clear the behind-base state.
• Incoming upstream change was fix(#5551): refresh sidebar recency after missed session events #5562/bug(session-list): reactivated older conversation not bumped to top / into Today until manual refresh #5551, touching CHANGELOG.md, static/sessions.js, tests/test_issue3916_external_refresh_poll.py, tests/test_issue4151_pwa_focus_sse.py, and tests/test_webui_external_refresh_frontend.py; the merge auto-resolved cleanly.
• Added commit 7425dc35 to address Greptile's state-DB sidecar reconstruction gap: compression_recovery_source_session_id and compression_recovery_action now round-trip through WebUIJsonSessionDB metadata rows and _state_db_row_to_sidecar().
• Local focused verification passed: env -u HERMES_WEBUI_DEFAULT_MODEL ./scripts/test.sh tests/test_compression_recovery_action.py tests/test_auto_compression_terminal_failure.py tests/test_issue3916_external_refresh_poll.py tests/test_issue4151_pwa_focus_sse.py tests/test_webui_external_refresh_frontend.py -> 56 passed.
• Local JS syntax checks passed: node --check static/sessions.js, node --check static/messages.js, and node --check static/ui.js.
• Python/static checks passed: python3 -m py_compile api/session_recovery.py api/webui_session_db.py tests/test_compression_recovery_action.py, python3 scripts/ruff_lint.py --diff origin/master, and git diff --check.
• GitHub compare is now behind_by: 0, ahead_by: 15 at head 7425dc354946543d31ab25b3c52e397cdac82dd1.
• GitHub Actions and Greptile started fresh runs on 7425dc35 and are still pending at the time of this note.

[nesquena-hermes]

🔬 Gate certification — RED ⛔ (recovery-child not lineage-isolated → shows parent transcript + wrong-session button 409; + cross-profile child reuse — 2 CORE) · concept → Nathan

Certified head: sha:7425dc35 (rebased onto current master, git apply clean) · PR: #5538 · franksong2702 (NEW), Live Stream: add compression recovery action · +878/−6, 13 files, new subsystem
Verdict: The concept is genuinely good — turning a compression-exhausted dead-end into a "start focused continuation" recovery action is a real UX win. But the implementation has two COREs in the recurring lineage-isolation / profile-scoping family: the recovery child carries parent_session_id without an isolation marker (so it displays the exhausted parent's transcript + a wrong-session recovery button that 409s), and the recovery-child lookup isn't profile-scoped (cross-profile reuse). Both need fixing; also a substantial new-subsystem + visible feature from a new author → Nathan's concept + visual sign-off.

What I ran (rebased worktree /tmp/wt-rebase-5538)

Gate	Result
Rebase onto current master	✅ git apply clean
Codex (reproduce)	SHIP-WITH-FIXES — 2 distinct CORE (lineage-leak + cross-profile reuse); Codex verified by direct invocation
Full pytest suite	✅ rc=0 green — but the two edges are NOT test-covered (Codex's direct invocation caught them, the gate's value on a new subsystem)
PR's own test	✅ 248-line test_compression_recovery_action.py passes (doesn't cover the lineage/profile edges)

Findings

⛔ CORE (Codex-verified) — recovery child shows the exhausted PARENT's transcript + a wrong-session recovery button (routes.py:19877 → _merged_webui_lineage_messages_for_display() :11709/:8126): the focused-continuation child gets parent_session_id but NO lineage-isolation marker (not session_source=="fork", no relationship_type=="child_session"), so GET /api/session merges the parent's messages into the empty child (Codex: empty child + parent returns parent messages). This surfaces the parent's _compressionRecovery card in the NEW session, and ui.js:13499 renders the button with the CHILD session id → posting the wrong session → 409. Fix: mark recovery children isolated (session_source="fork" or a dedicated recovery-child guard in _merged_webui_lineage_messages_for_display() returning primary_messages when compression_recovery_source_session_id/_action is set); render recovery buttons with recovery.source_session_id, not the current session id. (Same lineage-isolation class as #5582/#5556/#5563 — a new child carrying parent_session_id MUST be marked isolated.)

⛔ CORE (Codex-verified) — recovery-child reuse is not session/profile-scoped (routes.py:19854, find_compression_recovery_session() at models.py:3323/3340): the start handler validates the SOURCE session's profile, but the child-reuse lookup scans all cached/persisted sessions matching only compression_recovery_source_session_id + action — no child-profile check. A foreign-profile/corrupt child with that marker can be returned + serialized. Fix: make the lookup take the source profile or validate _profiles_match(child.profile, source.profile) + active-profile visibility before reuse; ignore mismatches and create a fresh child.

✅ Otherwise: server payload build (build_compression_recovery_payload/stamp_...) is structured; the concept (recovery-from-exhaustion action) is a real improvement.

Recommendation to the next agent / author

RED — gate-fail/changes-requested (2 CORE): (1) mark the recovery-continuation child as lineage-isolated so it doesn't inherit the parent's transcript/recovery-card, and render the recovery button with recovery.source_session_id; (2) profile-scope find_compression_recovery_session() (require _profiles_match + active-profile visibility on the reused child). The concept is worth shipping — a compression-exhausted recovery action is a genuine UX win — but the child-session isolation + profile-scoping must be fixed (recurring lineage class). Then, as a substantial new subsystem + visible feature from a new contributor, it's Nathan's concept + visual sign-off. concept 4/5 (good idea, two implementation COREs). Author @franksong2702 (new). crit=3. (Gate value: a green suite didn't cover the lineage-merge or cross-profile-reuse — Codex's direct route invocation exposed both; a new child that carries parent_session_id without an isolation marker is the SAME recurring bug I've now seen on #5582/#5556/#5563 — always check a new-child feature marks isolation + scopes lookups by profile.)

---

_Gate-certifier layer (warm-up → gate → release). I do not merge/tag/deploy. Rebased onto current master; Codex verified (direct invocation) the recovery child inherits the parent transcript (no isolation marker → _merged_webui_lineage_messages_for_display merges parent) + renders the recovery button with the child sid (409), and find_compression_recovery_session reuses cross-profile (no profiles_match). Suite green (edges not test-covered). Concept good; 2 lineage/profile COREs. Cert valid for sha:7425dc35.

[nesquena-hermes]

🔬 Release-manager deep review — engineering GREEN, converged; held for maintainer concept + visual sign-off

Ran the full authoritative gate on the live head (d1ed4a0b) merged onto current master (review-5538 worktree). Independently re-verified the two round-1 gate-cert COREs are genuinely fixed — not just claimed:

✅ CORE-1 (lineage isolation) — verified by code read. _merged_webui_lineage_messages_for_display() (api/routes.py:8153-8161) is double-guarded: it returns the child's own primary_messages when compression_recovery_source_session_id + compression_recovery_action are set, AND independently when session_source=="fork". The recovery child cannot inherit the exhausted parent's transcript, and the recovery button renders with recovery.source_session_id (static/ui.js:426), not the child sid — no wrong-session 409.

✅ CORE-2 (profile-scope) — verified by code read. _compression_recovery_child_matches() (api/models.py:3306-3313) rejects any child whose profile doesn't match the source via _profiles_match, and fails closed (returns False on import error) before any reuse. find_compression_recovery_session() is called with source_profile (routes.py).

Gate results

Gate	Result
ESLint runtime + ruff-forward + node --check + py-compile	✅ CLEAN
Browser smoke (/, /#settings, /#sessions)	✅ CLEAN — 0 console errors
Codex (reproduce, direct invocation)	✅ SAFE TO SHIP — 0 findings; verified compression-only stamping (won't fire on normal/auth/quota/cancel/tool-limit completion), generic-continuation block+clear, CSRF-gated POST
Full pytest suite (-p no:xdist)	✅ 12139 passed, 0 failed
Fable-UX	SHIP-WITH-UX-FIXES — concept sound, cross-device OK; 1 real minor + 2 optional (below)
#5592 _soft_partial_terminal_failure block	✅ preserved (grep=2)

Fable non-blocking UX items (fold in on approval)

1. Stale-actionable card (minor real bug): after recovery is cleared by a substantive prompt, the persisted card in the transcript still shows an enabled button → clicking yields a raw 409 toast. Fix: gate the button on session-level recovery still being active, or map that 409 to a neutral toast.
2. Attach the session snapshot to the apperror payload (streaming.py:9761, mirroring the result path at :8706) so the pre-send hint fires without a reload.
3. Optional: frontend recognizes type: "compression_recovery_required" on the chat-start 409 and routes it through showCompressionRecoveryContinuationHint().

Status

Engineering-converged and gate-clean. Held for maintainer decision because this is a new subsystem (api/compression_recovery.py) + a visible recovery card in the chat surface + a concept call on the #4685 recovery direction, from a new contributor. On approval I'll fold in Fable fix #1 (+ optionally #2/#3), re-run the gate, and ship. Thanks @franksong2702 — solid work converging the two lineage/profile COREs.

[nesquena-hermes]

Shipped in v0.51.889 (deployed live — the recovery functions are in the served ui.js). Thanks @franksong2702 — a genuine UX win turning a compression-exhausted dead-end into a one-click focused continuation. Folded in a small polish before ship (stale-card 409 → neutral toast + retired-card styling, with a finally-guard so the retired button stays inert) and re-gated to convergence: Codex SAFE, Fable SHIP-UX, full suite green, both lineage/profile isolation COREs verified fixed. 🙏