fix(webui): stage SSE-error reconnect probes so the live turn doesn't blank out

[allenliang2022]

Problem

On the source WebUI, after a turn finishes streaming the whole assistant message
could briefly blank out and then reappear (sometimes only after a refresh).
The trigger is a transient error event on the chat EventSource.

Root cause

In attachLiveStream, the EventSource.onerror handler made a single 1.5s
reconnect probe. If /api/chat/stream/status did not yet report active or
replay_available at that one moment, it fell straight through to
_handleStreamError(source), which:

• _clearOwnerInflightState()
• S.activeStreamId = null
• pushes a Connection interrupted… message
• renderMessages()

That wipes the live message DOM/state even when the backend was still
producing tokens, or when the run-journal replay file was a beat away from
becoming visible. The settled response then reappeared later from the
sidecar/replay path — producing the disappear-then-restore flicker users saw.

This was reproduced on a live source build: client sse_error events on the
chat response stream led to the live state being cleared while the backend
stream_end/done completed normally afterward — i.e. the frontend cleared
too early, the backend was fine.

Fix

Replace the single 1.5s probe with a short staged retry window
_retryDelays = [1500, 3000, 5000, 8000] ms, driven by a recursive
_probeReconnect(attempt):

• each stage re-queries /api/chat/stream/status and reconnects (active)
  or replays (replay_available) as soon as the backend is reachable;
• the live DOM and S.activeStreamId / INFLIGHT state are kept intact across
  the whole window;
• _handleStreamError(source) is only reached after every stage has failed.

The existing offline / page-hidden deferrals and the _isSessionCurrentPane
guard are preserved on every stage, so a backgrounded or switched-away session
still bails out instead of leaking a reconnect.

Verification

Reproduced and verified live on a source build of nesquena/hermes-webui at HEAD with an instrumented
EventSource (hooking the chat stream onerror, a stubbed
/api/chat/stream/status, and a 100ms sampler of S.activeStreamId + the
message DOM + the interrupted-marker text):

• Transient error, backend still alive — error injected mid-stream, status
  returns not-ready on probe Portability #1 then active on probe Hermes Web UI — Sprints 11-14: multi-provider models, settings, sessi… #2: S.activeStreamId
  stayed non-null the entire time, no Connection interrupted marker ever
  appeared, and the stream reattached at ~4.5s (Reconnecting… (2/4) →
  Reconnected). The old single-probe path would have cleared at 1.5s.
• Genuinely dead connection — status returns not-ready on all four stages:
  live state was held intact for the full window (probes at ~1.5s / 4.5s / 9.5s /
  17.6s, status text Reconnecting… → (2/4) → (3/4) → (4/4)), and only
  after the last stage failed (~17.6s) did it fall through to
  _handleStreamError (activeStreamId → null + interrupted marker). So the error
  path is preserved, just deferred until recovery is actually impossible.

Tests

• tests/test_sse_error_multi_probe_reconnect.py (new) — source-locks the staged
  shape (_retryDelays, _probeReconnect, per-stage status re-query) and the
  ordering invariant: the terminal _handleStreamError call sits after the
  next-stage scheduling guard, and neither _clearOwnerInflightState() nor
  S.activeStreamId=null appears in the reconnect block before the window is
  exhausted.
• node --check static/messages.js passes.
• Neighboring locks green: test_webui_external_refresh_frontend.py,
  test_issue3916_external_refresh_poll.py, test_tars_scroll_reset_regressions.py,
  test_issue4811_reconnect_chronology.py, test_issue3877_midstream_flicker.py,
  test_issue3103_sse_no_connection_close.py, test_inflight_stream_reuse.py
  (84 passed) + the new file (7 passed).

Single-file behavior change (static/messages.js) plus its source-lock test.

[greptile-apps]

Greptile Summary

This PR fixes a user-visible "blank-then-restore" flicker on the live assistant turn by replacing the single 1.5 s reconnect probe in attachLiveStream's onerror handler with a four-stage backoff window ([1500, 3000, 5000, 8000] ms). Live DOM state and S.activeStreamId are preserved across all stages; _handleStreamError is only reached once every stage has failed.

• static/messages.js: onerror now drives a recursive _probeReconnect(attempt) closure that re-queries /api/chat/stream/status at each stage and reconnects or replays as soon as the backend is reachable, instead of erroring out after a single unsuccessful probe.
• tests/test_sse_error_multi_probe_reconnect.py (new): source-locks the staged-retry shape, ordering invariants, and the absence of premature state-wipes inside the reconnect window using brace-matching extraction of the guarded block.
• tests/test_run_journal_frontend_static.py: anchor updated from the now-removed setComposerStatus('Reconnecting' string to _reconnectAttempted=true;, window widened to 1100 chars to cover the expanded block.

Confidence Score: 5/5

Safe to merge — the change is confined to the SSE error handler in messages.js and its tests, correctly defers the terminal error path until all probes fail, and preserves all existing offline/page-hidden/session-switch guard semantics.

The reconnect logic is straightforward: a recursive closure that re-queries stream status at each stage and only calls _handleStreamError once every delay slot is exhausted. Live DOM state and activeStreamId are never touched inside the retry window, the _isSessionCurrentPane and _terminalStateReached guards are re-evaluated at the start of every probe, and the new test suite uses brace-matched block extraction to reliably enforce the ordering invariant. No pre-existing guards were removed or weakened.

No files require special attention.

Important Files Changed

Filename	Overview
static/messages.js	Replaces single 1.5 s reconnect probe with four-stage backoff; live state correctly preserved across all stages, terminal error path properly guarded behind nextDelay falsy check.
tests/test_sse_error_multi_probe_reconnect.py	New source-lock test uses robust brace-matching _reconnect_block() instead of a fixed character window; covers probe shape, counter ordering, and the no-early-state-wipe invariant.
tests/test_run_journal_frontend_static.py	Anchor updated to stable _reconnectAttempted=true; string; the 1100-char fixed window is generous enough to cover all four assertions which appear early in the block.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant ES as EventSource
    participant OH as onerror handler
    participant PR as _probeReconnect(attempt)
    participant API as /api/chat/stream/status
    participant HSE as _handleStreamError

    ES->>OH: error event
    OH->>OH: guard checks (_terminalStateReached, _isSessionCurrentPane)
    OH->>OH: "_reconnectAttempted=true, setComposerStatus(Reconnecting 1/4)"
    OH->>PR: setTimeout(_probeReconnect(0), 1500ms)

    loop For each attempt 0..3
        PR->>PR: guard checks (_terminalStateReached, _isSessionCurrentPane)
        PR->>API: GET /api/chat/stream/status
        alt active or replay_available
            API-->>PR: active or replay_available
            PR->>ES: _wireSSE(new EventSource)
            note over PR: reconnected, live state preserved
        else not ready, more stages remain
            API-->>PR: not ready
            PR->>PR: setComposerStatus Reconnecting N+1/4
            PR->>PR: setTimeout _probeReconnect(attempt+1)
            note over PR: live DOM and S.activeStreamId intact
        else all stages exhausted
            API-->>PR: not ready on attempt 3
            PR->>HSE: _handleStreamError(source)
            note over HSE: clears INFLIGHT, shows error
        end
    end

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant ES as EventSource
    participant OH as onerror handler
    participant PR as _probeReconnect(attempt)
    participant API as /api/chat/stream/status
    participant HSE as _handleStreamError

    ES->>OH: error event
    OH->>OH: guard checks (_terminalStateReached, _isSessionCurrentPane)
    OH->>OH: "_reconnectAttempted=true, setComposerStatus(Reconnecting 1/4)"
    OH->>PR: setTimeout(_probeReconnect(0), 1500ms)

    loop For each attempt 0..3
        PR->>PR: guard checks (_terminalStateReached, _isSessionCurrentPane)
        PR->>API: GET /api/chat/stream/status
        alt active or replay_available
            API-->>PR: active or replay_available
            PR->>ES: _wireSSE(new EventSource)
            note over PR: reconnected, live state preserved
        else not ready, more stages remain
            API-->>PR: not ready
            PR->>PR: setComposerStatus Reconnecting N+1/4
            PR->>PR: setTimeout _probeReconnect(attempt+1)
            note over PR: live DOM and S.activeStreamId intact
        else all stages exhausted
            API-->>PR: not ready on attempt 3
            PR->>HSE: _handleStreamError(source)
            note over HSE: clears INFLIGHT, shows error
        end
    end

Loading

_{Reviews (3): Last reviewed commit: "fix(test): update run-journal reconnect ..." | Re-trigger Greptile}

[allenliang2022]

Thanks @greptile-apps — both P2s addressed in 265e176f:

1. Stage counter skipping (1/4) (static/messages.js) — applied your suggestion. _retryDelays is now declared before the first setComposerStatus, and the initial probe shows Reconnecting… (1/${_retryDelays.length}). The counter now progresses (1/4) → (2/4) → (3/4) → (4/4) instead of jumping from a bare Reconnecting… straight to (2/4).

2. Magic-number test window (tests/test_sse_error_multi_probe_reconnect.py) — replaced the block_start + 1600 slice in both test_handle_stream_error_only_after_retry_window_exhausted and test_live_state_not_cleared_mid_window with a _reconnect_block() helper that brace-matches from if(!_reconnectAttempted&&streamId){ to its matching close, so the ordering / no-early-clear assertions slice the actual block body and can't silently under-assert if future guard lines grow it. Also added test_stage_counter_starts_at_one to source-lock the (1/N) start.

node --check static/messages.js passes; the test file is green (8 passed) along with the neighboring reconnect/flicker locks.

[greptile-apps]

Both fixes look correct and complete.

static/messages.js: _retryDelays is now declared before the first setComposerStatus, so `Reconnecting… (1/${_retryDelays.length})` is in scope on the initial probe. The counter progresses (1/4) → (2/4) → (3/4) → (4/4) as expected.

tests/test_sse_error_multi_probe_reconnect.py: The _reconnect_block() brace-matching helper is clean — it correctly advances i to the opening brace position and depth-counts to find the matching close, so the sliced body will always cover the actual block regardless of future growth. The new test_stage_counter_starts_at_one locks both the (1/N) string and the declaration-before-use ordering. The two ordering/no-early-clear tests now operate on the full brace-matched block instead of the fixed character window.

Both P2s are fully addressed. LGTM — this is ready to merge.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.